Oct 30 12:48:58.195832 ip-10-0-137-21 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 12:48:58.195843 ip-10-0-137-21 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 12:48:58.195853 ip-10-0-137-21 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 12:48:58.196158 ip-10-0-137-21 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 12:49:09.555968 ip-10-0-137-21 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 12:49:09.555998 ip-10-0-137-21 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot eb44e9f4016b4b3c8320cea1e9014cf9 -- Oct 30 12:51:11.707899 ip-10-0-137-21 systemd[1]: Starting Kubernetes Kubelet... Oct 30 12:51:12.096550 ip-10-0-137-21 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 12:51:12.096550 ip-10-0-137-21 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 12:51:12.096550 ip-10-0-137-21 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 12:51:12.096550 ip-10-0-137-21 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 12:51:12.100421 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100354 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:51:12.100421 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100421 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:51:12.100421 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100425 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:51:12.100525 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100428 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:51:12.100525 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100431 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:51:12.100525 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100435 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:51:12.100525 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100439 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:51:12.100525 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100442 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:51:12.100525 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100445 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:51:12.100525 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100449 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:51:12.100525 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100460 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:51:12.100525 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100464 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:51:12.100525 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100466 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:51:12.100525 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100469 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:51:12.100525 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100472 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:51:12.100525 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100484 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:51:12.100525 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100487 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:51:12.100525 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100490 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:51:12.100525 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100492 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:51:12.100525 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100495 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:51:12.100525 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100497 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:51:12.100525 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100500 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:51:12.100983 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100503 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:51:12.100983 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100506 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:51:12.100983 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100508 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:51:12.100983 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100511 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:51:12.100983 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100514 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:51:12.100983 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100516 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:51:12.100983 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100519 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:51:12.100983 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100522 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:51:12.100983 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100524 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:51:12.100983 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100527 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:51:12.100983 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100531 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:51:12.100983 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100534 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:51:12.100983 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100537 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:51:12.100983 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100539 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:51:12.100983 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100542 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:51:12.100983 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100544 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:51:12.100983 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100547 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:51:12.100983 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100550 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:51:12.100983 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100552 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:51:12.101438 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100554 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:51:12.101438 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100557 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:51:12.101438 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100559 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:51:12.101438 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100562 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:51:12.101438 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100564 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:51:12.101438 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100567 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:51:12.101438 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100570 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:51:12.101438 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100572 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:51:12.101438 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100575 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:51:12.101438 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100577 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:51:12.101438 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100580 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:51:12.101438 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100582 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:51:12.101438 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100585 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:51:12.101438 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100588 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:51:12.101438 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100591 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:51:12.101438 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100594 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:51:12.101438 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100596 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:51:12.101438 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100599 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:51:12.101438 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100601 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:51:12.101438 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100604 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:51:12.101917 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100606 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:51:12.101917 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100622 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:51:12.101917 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100625 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:51:12.101917 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100628 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:51:12.101917 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100631 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:51:12.101917 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100633 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:51:12.101917 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100636 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:51:12.101917 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100639 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:51:12.101917 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100642 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:51:12.101917 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100644 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:51:12.101917 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100647 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:51:12.101917 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100649 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:51:12.101917 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100653 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:51:12.101917 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100657 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:51:12.101917 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100660 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:51:12.101917 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100663 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:51:12.101917 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100665 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:51:12.101917 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100668 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:51:12.101917 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100670 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:51:12.102382 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100673 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:51:12.102382 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100676 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:51:12.102382 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100678 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:51:12.102382 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100681 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:51:12.102382 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100684 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:51:12.102382 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100687 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:51:12.102382 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100690 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:51:12.102382 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100693 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:51:12.102382 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100695 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:51:12.102382 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100698 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:51:12.102382 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100701 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:51:12.102382 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100703 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:51:12.102382 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100706 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:51:12.102382 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100708 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:51:12.102382 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100711 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:51:12.102382 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100714 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:51:12.102382 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100716 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:51:12.102382 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100719 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:51:12.102382 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100721 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:51:12.102382 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100724 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:51:12.102864 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100726 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:51:12.102864 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100731 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:51:12.102864 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100733 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:51:12.102864 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100736 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:51:12.102864 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100738 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:51:12.102864 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.100741 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:51:12.102864 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102347 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:51:12.102864 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102356 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:51:12.102864 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102359 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:51:12.102864 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102363 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:51:12.102864 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102366 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:51:12.102864 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102368 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:51:12.102864 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102371 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:51:12.102864 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102380 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:51:12.102864 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102383 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:51:12.102864 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102386 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:51:12.102864 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102389 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:51:12.102864 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102392 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:51:12.102864 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102395 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:51:12.103340 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102397 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:51:12.103340 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102400 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:51:12.103340 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102403 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:51:12.103340 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102405 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:51:12.103340 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102408 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:51:12.103340 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102410 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:51:12.103340 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102412 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:51:12.103340 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102415 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:51:12.103340 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102417 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:51:12.103340 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102420 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:51:12.103340 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102422 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:51:12.103340 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102425 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:51:12.103340 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102427 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:51:12.103340 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102430 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:51:12.103340 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102434 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:51:12.103340 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102437 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:51:12.103340 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102439 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:51:12.103340 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102441 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:51:12.103340 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102450 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:51:12.103340 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102452 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:51:12.103847 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102455 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:51:12.103847 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102457 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:51:12.103847 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102460 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:51:12.103847 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102463 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:51:12.103847 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102465 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:51:12.103847 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102468 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:51:12.103847 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102470 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:51:12.103847 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102473 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:51:12.103847 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102476 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:51:12.103847 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102479 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:51:12.103847 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102481 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:51:12.103847 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102484 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:51:12.103847 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102486 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:51:12.103847 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102489 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:51:12.103847 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102493 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:51:12.103847 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102497 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:51:12.103847 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102500 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:51:12.103847 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102503 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:51:12.103847 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102506 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:51:12.104321 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102509 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:51:12.104321 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102512 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:51:12.104321 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102516 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:51:12.104321 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102519 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:51:12.104321 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102521 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:51:12.104321 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102524 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:51:12.104321 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102526 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:51:12.104321 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102530 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:51:12.104321 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102532 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:51:12.104321 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102535 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:51:12.104321 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102537 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:51:12.104321 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102540 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:51:12.104321 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102542 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:51:12.104321 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102545 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:51:12.104321 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102548 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:51:12.104321 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102552 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:51:12.104321 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102556 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:51:12.104321 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102559 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:51:12.104321 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102562 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:51:12.104792 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102565 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:51:12.104792 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102568 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:51:12.104792 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102571 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:51:12.104792 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102573 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:51:12.104792 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102576 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:51:12.104792 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102578 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:51:12.104792 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102581 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:51:12.104792 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102583 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:51:12.104792 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102586 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:51:12.104792 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102588 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:51:12.104792 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102591 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:51:12.104792 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102593 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:51:12.104792 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102596 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:51:12.104792 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102598 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:51:12.104792 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102601 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:51:12.104792 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102603 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:51:12.104792 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102606 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:51:12.104792 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102621 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:51:12.104792 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102624 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:51:12.104792 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102626 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:51:12.105256 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102630 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:51:12.105256 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102633 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:51:12.105256 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102636 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:51:12.105256 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102638 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:51:12.105256 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102641 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:51:12.105256 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102647 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:51:12.105256 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102650 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:51:12.105256 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102653 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:51:12.105256 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102655 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:51:12.105256 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102658 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:51:12.105256 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102660 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:51:12.105256 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102663 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:51:12.105256 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102666 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:51:12.105256 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102668 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:51:12.105256 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.102670 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:51:12.105256 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103148 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:51:12.105256 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103153 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:51:12.105256 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103156 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:51:12.105256 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103159 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:51:12.105256 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103161 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:51:12.105731 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103164 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:51:12.105731 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103166 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:51:12.105731 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103169 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:51:12.105731 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103172 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:51:12.105731 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103174 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:51:12.105731 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103177 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:51:12.105731 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103179 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:51:12.105731 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103182 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:51:12.105731 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103184 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:51:12.105731 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103187 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:51:12.105731 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103189 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:51:12.105731 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103192 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:51:12.105731 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103195 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:51:12.105731 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103199 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:51:12.105731 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103203 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:51:12.105731 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103206 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:51:12.105731 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103209 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:51:12.105731 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103211 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:51:12.105731 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103214 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:51:12.106177 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103216 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:51:12.106177 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103219 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:51:12.106177 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103221 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:51:12.106177 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103224 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:51:12.106177 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103226 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:51:12.106177 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103229 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:51:12.106177 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103232 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:51:12.106177 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103234 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:51:12.106177 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103237 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:51:12.106177 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103240 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:51:12.106177 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103242 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:51:12.106177 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103245 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:51:12.106177 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103247 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:51:12.106177 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103250 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:51:12.106177 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103253 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:51:12.106177 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103255 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:51:12.106177 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103258 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:51:12.106177 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103261 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:51:12.106177 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103264 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:51:12.106660 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103268 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:51:12.106660 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103271 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:51:12.106660 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103273 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:51:12.106660 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103276 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:51:12.106660 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103279 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:51:12.106660 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103282 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:51:12.106660 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103284 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:51:12.106660 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103287 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:51:12.106660 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103289 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:51:12.106660 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103292 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:51:12.106660 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103295 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:51:12.106660 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103298 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:51:12.106660 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103300 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:51:12.106660 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103303 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:51:12.106660 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103305 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:51:12.106660 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103307 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:51:12.106660 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103310 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:51:12.106660 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103312 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:51:12.106660 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103316 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:51:12.106660 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103318 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:51:12.107207 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103321 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:51:12.107207 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103323 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:51:12.107207 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103326 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:51:12.107207 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103329 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:51:12.107207 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103331 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:51:12.107207 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103334 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:51:12.107207 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103336 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:51:12.107207 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103339 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:51:12.107207 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103342 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:51:12.107207 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103345 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:51:12.107207 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103347 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:51:12.107207 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103350 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:51:12.107207 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103353 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:51:12.107207 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103355 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:51:12.107207 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103358 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:51:12.107207 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103361 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:51:12.107207 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103363 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:51:12.107207 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103366 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:51:12.107207 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103369 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:51:12.107207 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103371 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:51:12.107689 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103374 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:51:12.107689 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103377 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:51:12.107689 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103379 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:51:12.107689 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103382 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:51:12.107689 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103384 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:51:12.107689 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103387 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:51:12.107689 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103389 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:51:12.107689 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103392 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:51:12.107689 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103394 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:51:12.107689 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103397 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:51:12.107689 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103399 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:51:12.107689 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103401 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:51:12.107689 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103404 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:51:12.107689 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103408 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:51:12.107689 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103410 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:51:12.107689 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103413 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:51:12.107689 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103416 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:51:12.107689 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103418 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:51:12.107689 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103421 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:51:12.108138 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103424 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:51:12.108138 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103427 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:51:12.108138 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103429 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:51:12.108138 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.103432 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:51:12.108138 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104352 2582 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 12:51:12.108138 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104363 2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 12:51:12.108138 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104369 2582 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 12:51:12.108138 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104374 2582 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 12:51:12.108138 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104378 2582 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 12:51:12.108138 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104381 2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 12:51:12.108138 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104386 2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 12:51:12.108138 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104390 2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 12:51:12.108138 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104394 2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 12:51:12.108138 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104397 2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 12:51:12.108138 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104401 2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 12:51:12.108138 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104404 2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 12:51:12.108138 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104408 2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 12:51:12.108138 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104411 2582 flags.go:64] FLAG: --cgroup-root="" Oct 30 12:51:12.108138 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104414 2582 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 12:51:12.108138 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104417 2582 flags.go:64] FLAG: --client-ca-file="" Oct 30 12:51:12.108138 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104419 2582 flags.go:64] FLAG: --cloud-provider="external" Oct 30 12:51:12.108138 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104422 2582 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 12:51:12.108138 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104428 2582 flags.go:64] FLAG: --cluster-domain="" Oct 30 12:51:12.108740 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104431 2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 12:51:12.108740 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104434 2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 12:51:12.108740 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104437 2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 12:51:12.108740 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104442 2582 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 12:51:12.108740 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104446 2582 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 12:51:12.108740 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104449 2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 12:51:12.108740 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104461 2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 12:51:12.108740 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104465 2582 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 12:51:12.108740 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104468 2582 flags.go:64] FLAG: --contention-profiling="false" Oct 30 12:51:12.108740 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104471 2582 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 12:51:12.108740 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104474 2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 12:51:12.108740 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104478 2582 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 12:51:12.108740 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104481 2582 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 12:51:12.108740 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104485 2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 12:51:12.108740 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104488 2582 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 12:51:12.108740 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104491 2582 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 12:51:12.108740 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104494 2582 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 12:51:12.108740 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104497 2582 flags.go:64] FLAG: --enable-server="true" Oct 30 12:51:12.108740 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104500 2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 12:51:12.108740 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104504 2582 flags.go:64] FLAG: --event-burst="100" Oct 30 12:51:12.108740 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104507 2582 flags.go:64] FLAG: --event-qps="50" Oct 30 12:51:12.108740 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104510 2582 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 12:51:12.108740 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104513 2582 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 12:51:12.108740 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104516 2582 flags.go:64] FLAG: --eviction-hard="" Oct 30 12:51:12.109367 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104520 2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 12:51:12.109367 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104523 2582 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 12:51:12.109367 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104526 2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 12:51:12.109367 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104529 2582 flags.go:64] FLAG: --eviction-soft="" Oct 30 12:51:12.109367 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104532 2582 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 12:51:12.109367 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104535 2582 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 12:51:12.109367 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104538 2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 12:51:12.109367 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104541 2582 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 12:51:12.109367 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104544 2582 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 12:51:12.109367 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104546 2582 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 12:51:12.109367 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104549 2582 flags.go:64] FLAG: --feature-gates="" Oct 30 12:51:12.109367 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104554 2582 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 12:51:12.109367 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104557 2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 12:51:12.109367 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104560 2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 12:51:12.109367 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104563 2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 12:51:12.109367 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104566 2582 flags.go:64] FLAG: --healthz-port="10248" Oct 30 12:51:12.109367 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104570 2582 flags.go:64] FLAG: --help="false" Oct 30 12:51:12.109367 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104572 2582 flags.go:64] FLAG: --hostname-override="ip-10-0-137-21.ec2.internal" Oct 30 12:51:12.109367 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104575 2582 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 12:51:12.109367 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104578 2582 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 12:51:12.109367 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104581 2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 12:51:12.109367 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104584 2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 12:51:12.109367 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104588 2582 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 12:51:12.109921 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104590 2582 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 12:51:12.109921 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104593 2582 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 12:51:12.109921 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104596 2582 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 12:51:12.109921 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104599 2582 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 12:51:12.109921 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104602 2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 12:51:12.109921 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104605 2582 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 12:51:12.109921 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104620 2582 flags.go:64] FLAG: --kube-reserved="" Oct 30 12:51:12.109921 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104623 2582 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 12:51:12.109921 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104626 2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 12:51:12.109921 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104629 2582 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 12:51:12.109921 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104632 2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 12:51:12.109921 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104635 2582 flags.go:64] FLAG: --lock-file="" Oct 30 12:51:12.109921 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104638 2582 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 12:51:12.109921 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104641 2582 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 12:51:12.109921 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104644 2582 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 12:51:12.109921 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104650 2582 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 12:51:12.109921 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104652 2582 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 12:51:12.109921 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104655 2582 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 12:51:12.109921 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104659 2582 flags.go:64] FLAG: --logging-format="text" Oct 30 12:51:12.109921 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104662 2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 12:51:12.109921 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104665 2582 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 12:51:12.109921 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104668 2582 flags.go:64] FLAG: --manifest-url="" Oct 30 12:51:12.109921 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104671 2582 flags.go:64] FLAG: --manifest-url-header="" Oct 30 12:51:12.109921 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104675 2582 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 12:51:12.109921 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104678 2582 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 12:51:12.110526 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104683 2582 flags.go:64] FLAG: --max-pods="110" Oct 30 12:51:12.110526 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104686 2582 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 12:51:12.110526 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104689 2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 12:51:12.110526 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104692 2582 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 12:51:12.110526 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104695 2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 12:51:12.110526 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104698 2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 12:51:12.110526 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104701 2582 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 12:51:12.110526 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104704 2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 12:51:12.110526 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104710 2582 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 12:51:12.110526 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104713 2582 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 12:51:12.110526 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104716 2582 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 12:51:12.110526 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104719 2582 flags.go:64] FLAG: --pod-cidr="" Oct 30 12:51:12.110526 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104722 2582 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 12:51:12.110526 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104725 2582 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 12:51:12.110526 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104728 2582 flags.go:64] FLAG: --pods-per-core="0" Oct 30 12:51:12.110526 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104731 2582 flags.go:64] FLAG: --port="10250" Oct 30 12:51:12.110526 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104734 2582 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 12:51:12.110526 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104737 2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-057450241d331bc81" Oct 30 12:51:12.110526 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104740 2582 flags.go:64] FLAG: --qos-reserved="" Oct 30 12:51:12.110526 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104743 2582 flags.go:64] FLAG: --read-only-port="10255" Oct 30 12:51:12.110526 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104746 2582 flags.go:64] FLAG: --register-node="true" Oct 30 12:51:12.110526 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104748 2582 flags.go:64] FLAG: --register-with-taints="" Oct 30 12:51:12.110526 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104752 2582 flags.go:64] FLAG: --registry-burst="10" Oct 30 12:51:12.110526 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104755 2582 flags.go:64] FLAG: --registry-qps="5" Oct 30 12:51:12.110526 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104757 2582 flags.go:64] FLAG: --reserved-cpus="" Oct 30 12:51:12.110526 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104763 2582 flags.go:64] FLAG: --reserved-memory="" Oct 30 12:51:12.111139 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104767 2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 12:51:12.111139 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104770 2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 12:51:12.111139 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104772 2582 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 12:51:12.111139 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104775 2582 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 12:51:12.111139 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104778 2582 flags.go:64] FLAG: --runonce="false" Oct 30 12:51:12.111139 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104781 2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 12:51:12.111139 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104784 2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 12:51:12.111139 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104788 2582 flags.go:64] FLAG: --seccomp-default="false" Oct 30 12:51:12.111139 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104791 2582 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 12:51:12.111139 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104794 2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 12:51:12.111139 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104797 2582 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 12:51:12.111139 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104800 2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 12:51:12.111139 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104803 2582 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 12:51:12.111139 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104806 2582 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 12:51:12.111139 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104809 2582 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 12:51:12.111139 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104812 2582 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 12:51:12.111139 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104815 2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 12:51:12.111139 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104818 2582 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 12:51:12.111139 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104821 2582 flags.go:64] FLAG: --system-cgroups="" Oct 30 12:51:12.111139 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104823 2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 12:51:12.111139 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104829 2582 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 12:51:12.111139 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104832 2582 flags.go:64] FLAG: --tls-cert-file="" Oct 30 12:51:12.111139 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104834 2582 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 12:51:12.111139 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104839 2582 flags.go:64] FLAG: --tls-min-version="" Oct 30 12:51:12.111139 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104842 2582 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 12:51:12.111722 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104844 2582 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 12:51:12.111722 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104847 2582 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 12:51:12.111722 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104851 2582 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 12:51:12.111722 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104854 2582 flags.go:64] FLAG: --v="2" Oct 30 12:51:12.111722 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104858 2582 flags.go:64] FLAG: --version="false" Oct 30 12:51:12.111722 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104862 2582 flags.go:64] FLAG: --vmodule="" Oct 30 12:51:12.111722 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104867 2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 12:51:12.111722 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.104870 2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 12:51:12.111722 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105007 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:51:12.111722 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105011 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:51:12.111722 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105014 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:51:12.111722 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105016 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:51:12.111722 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105020 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:51:12.111722 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105023 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:51:12.111722 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105026 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:51:12.111722 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105029 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:51:12.111722 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105032 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:51:12.111722 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105035 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:51:12.111722 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105037 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:51:12.111722 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105040 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:51:12.111722 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105043 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:51:12.112223 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105045 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:51:12.112223 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105048 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:51:12.112223 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105051 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:51:12.112223 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105053 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:51:12.112223 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105056 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:51:12.112223 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105059 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:51:12.112223 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105061 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:51:12.112223 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105064 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:51:12.112223 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105066 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:51:12.112223 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105069 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:51:12.112223 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105072 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:51:12.112223 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105074 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:51:12.112223 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105077 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:51:12.112223 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105079 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:51:12.112223 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105082 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:51:12.112223 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105084 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:51:12.112223 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105087 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:51:12.112223 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105091 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:51:12.112223 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105093 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:51:12.112223 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105096 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:51:12.112719 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105098 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:51:12.112719 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105101 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:51:12.112719 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105104 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:51:12.112719 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105108 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:51:12.112719 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105110 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:51:12.112719 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105113 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:51:12.112719 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105115 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:51:12.112719 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105118 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:51:12.112719 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105121 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:51:12.112719 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105124 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:51:12.112719 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105126 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:51:12.112719 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105129 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:51:12.112719 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105132 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:51:12.112719 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105134 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:51:12.112719 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105137 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:51:12.112719 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105139 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:51:12.112719 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105142 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:51:12.112719 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105144 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:51:12.112719 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105147 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:51:12.112719 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105150 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:51:12.113221 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105152 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:51:12.113221 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105154 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:51:12.113221 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105157 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:51:12.113221 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105160 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:51:12.113221 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105162 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:51:12.113221 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105165 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:51:12.113221 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105168 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:51:12.113221 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105170 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:51:12.113221 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105173 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:51:12.113221 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105176 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:51:12.113221 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105179 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:51:12.113221 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105181 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:51:12.113221 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105184 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:51:12.113221 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105186 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:51:12.113221 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105190 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:51:12.113221 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105195 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:51:12.113221 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105199 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:51:12.113221 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105202 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:51:12.113221 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105206 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:51:12.113683 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105209 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:51:12.113683 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105212 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:51:12.113683 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105215 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:51:12.113683 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105218 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:51:12.113683 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105220 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:51:12.113683 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105223 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:51:12.113683 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105225 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:51:12.113683 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105228 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:51:12.113683 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105230 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:51:12.113683 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105233 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:51:12.113683 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105236 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:51:12.113683 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105238 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:51:12.113683 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105241 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:51:12.113683 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105243 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:51:12.113683 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105246 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:51:12.113683 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105249 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:51:12.113683 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105251 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:51:12.113683 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105254 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:51:12.113683 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105256 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:51:12.113683 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105259 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:51:12.114146 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105262 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:51:12.114146 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105265 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:51:12.114146 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105269 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:51:12.114146 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105271 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:51:12.114146 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105274 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:51:12.114146 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105277 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:51:12.114146 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105280 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:51:12.114146 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105284 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:51:12.114146 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105289 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:51:12.114146 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105292 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:51:12.114146 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105294 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:51:12.114146 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105297 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:51:12.114146 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105299 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:51:12.114146 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105302 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:51:12.114146 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.105329 2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 12:51:12.114498 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105363 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:51:12.114498 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105367 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:51:12.114498 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105370 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:51:12.114498 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105373 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:51:12.114498 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105376 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:51:12.114498 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105378 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:51:12.114498 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105381 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:51:12.114498 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105383 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:51:12.114498 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105387 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:51:12.114498 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105389 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:51:12.114498 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105391 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:51:12.114498 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105396 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:51:12.114498 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105399 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:51:12.114498 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105402 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:51:12.114498 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105405 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:51:12.114498 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105408 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:51:12.114498 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105411 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:51:12.114498 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105413 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:51:12.114498 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105416 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:51:12.115001 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105420 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:51:12.115001 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105422 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:51:12.115001 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105425 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:51:12.115001 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105427 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:51:12.115001 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105430 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:51:12.115001 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105432 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:51:12.115001 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105435 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:51:12.115001 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105439 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:51:12.115001 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105442 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:51:12.115001 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105444 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:51:12.115001 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105447 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:51:12.115001 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105450 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:51:12.115001 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105452 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:51:12.115001 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105455 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:51:12.115001 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105458 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:51:12.115001 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105460 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:51:12.115001 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105463 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:51:12.115001 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105466 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:51:12.115001 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105468 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:51:12.115445 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105471 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:51:12.115445 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105473 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:51:12.115445 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105476 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:51:12.115445 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105478 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:51:12.115445 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105481 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:51:12.115445 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105483 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:51:12.115445 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105486 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:51:12.115445 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105488 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:51:12.115445 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105491 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:51:12.115445 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105493 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:51:12.115445 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105496 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:51:12.115445 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105498 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:51:12.115445 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105501 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:51:12.115445 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105505 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:51:12.115445 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105507 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:51:12.115445 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105510 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:51:12.115445 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105512 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:51:12.115445 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105515 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:51:12.115445 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105518 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:51:12.115445 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105520 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:51:12.115918 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105524 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:51:12.115918 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105527 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:51:12.115918 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105529 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:51:12.115918 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105532 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:51:12.115918 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105535 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:51:12.115918 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105538 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:51:12.115918 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105540 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:51:12.115918 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105543 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:51:12.115918 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105546 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:51:12.115918 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105548 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:51:12.115918 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105551 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:51:12.115918 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105553 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:51:12.115918 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105556 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:51:12.115918 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105558 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:51:12.115918 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105561 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:51:12.115918 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105563 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:51:12.115918 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105566 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:51:12.115918 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105568 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:51:12.115918 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105571 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:51:12.115918 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105573 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:51:12.116386 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105576 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:51:12.116386 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105578 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:51:12.116386 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105581 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:51:12.116386 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105584 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:51:12.116386 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105586 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:51:12.116386 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105589 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:51:12.116386 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105592 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:51:12.116386 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105595 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:51:12.116386 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105599 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:51:12.116386 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105602 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:51:12.116386 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105605 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:51:12.116386 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105608 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:51:12.116386 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105623 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:51:12.116386 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105625 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:51:12.116386 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105628 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:51:12.116386 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105631 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:51:12.116386 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105633 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:51:12.116386 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105636 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:51:12.116386 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105639 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:51:12.116862 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105641 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:51:12.116862 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105644 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:51:12.116862 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105647 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:51:12.116862 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105649 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:51:12.116862 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105652 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:51:12.116862 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105654 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:51:12.116862 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105658 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:51:12.116862 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105660 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:51:12.116862 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.105663 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:51:12.116862 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.111714 2582 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 12:51:12.116862 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.111728 2582 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 12:51:12.116862 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.111744 2582 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 12:51:12.116862 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.111748 2582 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 12:51:12.116862 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111807 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:51:12.116862 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111813 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:51:12.116862 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111816 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:51:12.116862 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111820 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:51:12.116862 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111822 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:51:12.116862 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111827 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:51:12.117303 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111831 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:51:12.117303 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111834 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:51:12.117303 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111837 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:51:12.117303 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111840 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:51:12.117303 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111843 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:51:12.117303 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111846 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:51:12.117303 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111849 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:51:12.117303 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111851 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:51:12.117303 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111854 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:51:12.117303 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111857 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:51:12.117303 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111860 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:51:12.117303 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111862 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:51:12.117303 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111865 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:51:12.117303 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111869 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:51:12.117303 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111872 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:51:12.117303 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111875 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:51:12.117303 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111878 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:51:12.117303 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111881 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:51:12.117303 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111883 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:51:12.117778 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111886 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:51:12.117778 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111888 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:51:12.117778 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111891 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:51:12.117778 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111893 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:51:12.117778 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111897 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:51:12.117778 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111900 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:51:12.117778 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111902 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:51:12.117778 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111905 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:51:12.117778 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111908 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:51:12.117778 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111911 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:51:12.117778 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111913 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:51:12.117778 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111916 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:51:12.117778 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111919 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:51:12.117778 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111921 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:51:12.117778 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111924 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:51:12.117778 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111927 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:51:12.117778 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111929 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:51:12.117778 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111932 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:51:12.117778 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111934 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:51:12.117778 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111937 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:51:12.118247 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111939 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:51:12.118247 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111942 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:51:12.118247 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111944 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:51:12.118247 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111947 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:51:12.118247 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111949 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:51:12.118247 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111952 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:51:12.118247 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111955 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:51:12.118247 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111959 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:51:12.118247 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111963 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:51:12.118247 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111966 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:51:12.118247 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111969 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:51:12.118247 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111971 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:51:12.118247 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111974 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:51:12.118247 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111976 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:51:12.118247 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111979 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:51:12.118247 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111981 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:51:12.118247 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111985 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:51:12.118247 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111987 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:51:12.118247 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111990 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:51:12.118793 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111993 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:51:12.118793 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111995 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:51:12.118793 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.111998 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:51:12.118793 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112001 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:51:12.118793 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112003 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:51:12.118793 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112006 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:51:12.118793 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112008 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:51:12.118793 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112011 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:51:12.118793 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112013 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:51:12.118793 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112016 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:51:12.118793 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112018 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:51:12.118793 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112021 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:51:12.118793 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112023 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:51:12.118793 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112027 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:51:12.118793 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112029 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:51:12.118793 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112032 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:51:12.118793 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112034 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:51:12.118793 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112037 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:51:12.118793 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112040 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:51:12.119310 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112043 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:51:12.119310 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112045 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:51:12.119310 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112048 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:51:12.119310 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112051 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:51:12.119310 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112053 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:51:12.119310 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112056 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:51:12.119310 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112059 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:51:12.119310 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112061 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:51:12.119310 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112064 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:51:12.119310 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112066 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:51:12.119310 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112069 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:51:12.119310 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112072 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:51:12.119310 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112074 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:51:12.119310 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112077 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:51:12.119310 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112079 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:51:12.119310 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112082 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:51:12.119310 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112085 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:51:12.119310 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112087 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:51:12.119310 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112090 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:51:12.120005 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112092 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:51:12.120005 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112095 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:51:12.120005 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112097 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:51:12.120005 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112100 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:51:12.120005 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.112127 2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 12:51:12.120005 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112257 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:51:12.120005 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112261 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:51:12.120005 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112265 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:51:12.120005 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112268 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:51:12.120005 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112270 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:51:12.120005 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112273 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:51:12.120005 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112275 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:51:12.120005 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112278 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:51:12.120005 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112280 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:51:12.120005 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112283 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:51:12.120005 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112286 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:51:12.120396 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112290 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:51:12.120396 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112294 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:51:12.120396 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112297 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:51:12.120396 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112301 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:51:12.120396 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112304 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:51:12.120396 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112307 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:51:12.120396 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112310 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:51:12.120396 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112313 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:51:12.120396 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112316 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:51:12.120396 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112319 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:51:12.120396 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112321 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:51:12.120396 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112324 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:51:12.120396 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112326 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:51:12.120396 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112329 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:51:12.120396 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112331 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:51:12.120396 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112334 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:51:12.120396 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112336 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:51:12.120396 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112339 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:51:12.120396 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112341 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:51:12.120960 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112344 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:51:12.120960 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112346 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:51:12.120960 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112348 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:51:12.120960 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112351 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:51:12.120960 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112353 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:51:12.120960 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112356 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:51:12.120960 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112358 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:51:12.120960 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112361 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:51:12.120960 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112363 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:51:12.120960 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112365 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:51:12.120960 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112368 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:51:12.120960 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112371 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:51:12.120960 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112373 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:51:12.120960 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112376 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:51:12.120960 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112378 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:51:12.120960 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112381 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:51:12.120960 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112384 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:51:12.120960 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112386 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:51:12.120960 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112389 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:51:12.120960 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112391 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:51:12.121743 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112393 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:51:12.121743 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112396 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:51:12.121743 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112399 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:51:12.121743 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112401 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:51:12.121743 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112404 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:51:12.121743 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112407 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:51:12.121743 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112409 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:51:12.121743 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112412 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:51:12.121743 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112414 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:51:12.121743 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112417 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:51:12.121743 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112419 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:51:12.121743 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112422 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:51:12.121743 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112425 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:51:12.121743 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112427 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:51:12.121743 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112430 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:51:12.121743 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112432 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:51:12.121743 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112435 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:51:12.121743 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112437 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:51:12.121743 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112439 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:51:12.121743 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112442 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:51:12.122244 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112444 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:51:12.122244 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112447 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:51:12.122244 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112450 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:51:12.122244 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112461 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:51:12.122244 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112464 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:51:12.122244 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112466 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:51:12.122244 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112469 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:51:12.122244 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112472 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:51:12.122244 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112479 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:51:12.122244 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112482 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:51:12.122244 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112485 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:51:12.122244 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112489 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:51:12.122244 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112492 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:51:12.122244 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112496 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:51:12.122244 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112499 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:51:12.122244 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112502 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:51:12.122244 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112505 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:51:12.122244 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112508 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:51:12.122244 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112511 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:51:12.122756 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112513 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:51:12.122756 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112516 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:51:12.122756 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112518 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:51:12.122756 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112521 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:51:12.122756 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112523 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:51:12.122756 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112526 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:51:12.122756 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112528 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:51:12.122756 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112531 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:51:12.122756 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112533 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:51:12.122756 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112535 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:51:12.122756 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112538 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:51:12.122756 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112540 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:51:12.122756 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112543 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:51:12.122756 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112545 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:51:12.122756 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112548 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:51:12.122756 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112550 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:51:12.122756 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112553 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:51:12.123172 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.112577 2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 12:51:12.123172 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112608 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:51:12.123172 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112628 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:51:12.123172 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112631 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:51:12.123172 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112633 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:51:12.123172 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112637 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:51:12.123172 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112640 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:51:12.123172 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112643 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:51:12.123172 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112647 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:51:12.123172 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112649 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:51:12.123172 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112652 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:51:12.123172 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112654 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:51:12.123172 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112657 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:51:12.123172 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112660 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:51:12.123172 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112662 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:51:12.123172 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112665 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:51:12.123540 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112667 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:51:12.123540 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112670 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:51:12.123540 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112672 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:51:12.123540 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112675 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:51:12.123540 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112677 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:51:12.123540 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112680 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:51:12.123540 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112682 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:51:12.123540 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112685 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:51:12.123540 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112688 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:51:12.123540 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112690 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:51:12.123540 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112692 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:51:12.123540 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112695 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:51:12.123540 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112698 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:51:12.123540 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112700 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:51:12.123540 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112703 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:51:12.123540 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112705 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:51:12.123540 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112708 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:51:12.123540 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112710 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:51:12.123540 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112713 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:51:12.124003 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112716 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:51:12.124003 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112718 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:51:12.124003 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112720 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:51:12.124003 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112725 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:51:12.124003 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112728 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:51:12.124003 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112731 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:51:12.124003 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112734 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:51:12.124003 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112737 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:51:12.124003 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112739 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:51:12.124003 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112742 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:51:12.124003 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112744 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:51:12.124003 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112747 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:51:12.124003 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112749 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:51:12.124003 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112752 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:51:12.124003 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112754 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:51:12.124003 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112757 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:51:12.124003 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112759 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:51:12.124003 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112762 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:51:12.124003 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112764 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:51:12.124442 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112767 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:51:12.124442 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112769 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:51:12.124442 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112771 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:51:12.124442 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112774 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:51:12.124442 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112777 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:51:12.124442 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112780 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:51:12.124442 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112782 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:51:12.124442 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112788 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:51:12.124442 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112790 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:51:12.124442 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112793 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:51:12.124442 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112796 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:51:12.124442 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112798 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:51:12.124442 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112801 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:51:12.124442 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112803 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:51:12.124442 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112806 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:51:12.124442 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112808 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:51:12.124442 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112811 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:51:12.124442 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112814 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:51:12.124442 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112816 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:51:12.124442 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112819 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:51:12.124982 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112822 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:51:12.124982 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112826 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:51:12.124982 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112829 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:51:12.124982 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112832 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:51:12.124982 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112834 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:51:12.124982 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112837 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:51:12.124982 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112840 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:51:12.124982 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112842 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:51:12.124982 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112845 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:51:12.124982 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112847 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:51:12.124982 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112849 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:51:12.124982 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112852 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:51:12.124982 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112855 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:51:12.124982 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112858 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:51:12.124982 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112860 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:51:12.124982 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112863 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:51:12.124982 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112866 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:51:12.124982 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112868 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:51:12.124982 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112871 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:51:12.125504 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112875 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:51:12.125504 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112877 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:51:12.125504 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112880 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:51:12.125504 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112882 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:51:12.125504 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112885 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:51:12.125504 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112887 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:51:12.125504 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112890 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:51:12.125504 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112892 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:51:12.125504 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112895 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:51:12.125504 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112897 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:51:12.125504 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112900 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:51:12.125504 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112903 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:51:12.125504 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112906 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:51:12.125504 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:12.112908 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:51:12.125504 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.112957 2582 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 12:51:12.125504 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.115359 2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 12:51:12.125504 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.116232 2582 server.go:1014] "Starting client certificate rotation" Oct 30 12:51:12.125504 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.116326 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 12:51:12.125504 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.116355 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 12:51:12.126113 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.119692 2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 12:51:12.126113 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.122271 2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 12:51:12.130442 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.130426 2582 log.go:25] "Validated CRI v1 runtime API" Oct 30 12:51:12.133667 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.133595 2582 log.go:25] "Validated CRI v1 image API" Oct 30 12:51:12.134417 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.134396 2582 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 12:51:12.140525 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.140506 2582 fs.go:135] Filesystem UUIDs: map[5bcae973-a38e-4b89-9185-ba40144ffa06:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e8189750-30b2-4de3-9a9b-c16d3fe68d8c:/dev/nvme0n1p3] Oct 30 12:51:12.140586 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.140525 2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 12:51:12.145539 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.145434 2582 manager.go:217] Machine: {Timestamp:2026-10-30 12:51:12.144071674 +0000 UTC m=+0.319542565 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099258 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20dd1909b8e7a8a21ca82f86d2faeb SystemUUID:ec20dd19-09b8-e7a8-a21c-a82f86d2faeb BootID:eb44e9f4-016b-4b3c-8320-cea1e9014cf9 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f2:23:ad:9d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f2:23:ad:9d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:06:35:fe:23:01:82 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 12:51:12.145539 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.145537 2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 12:51:12.145664 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.145648 2582 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 12:51:12.146623 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.146592 2582 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 12:51:12.146750 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.146629 2582 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-21.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 12:51:12.146789 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.146752 2582 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 12:51:12.146789 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.146760 2582 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 12:51:12.146841 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.146800 2582 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 12:51:12.146841 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.146818 2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 12:51:12.147700 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.147689 2582 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 12:51:12.151435 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.151424 2582 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 12:51:12.151546 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.151538 2582 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 12:51:12.152903 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.152893 2582 kubelet.go:493] "Attempting to sync node with API server" Oct 30 12:51:12.152934 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.152908 2582 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 12:51:12.152934 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.152921 2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 12:51:12.152934 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.152930 2582 kubelet.go:405] "Adding apiserver pod source" Oct 30 12:51:12.153040 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.152938 2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 12:51:12.153224 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.153211 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 12:51:12.155277 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.155259 2582 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 12:51:12.161084 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.161066 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 12:51:12.161834 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.161817 2582 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 12:51:12.161873 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.161859 2582 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 12:51:12.162011 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.162001 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 12:51:12.162040 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.162016 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 12:51:12.162040 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.162022 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 12:51:12.162040 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.162028 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 12:51:12.162040 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.162034 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 12:51:12.162040 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.162040 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 12:51:12.162636 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.162627 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 12:51:12.162671 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.162637 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 12:51:12.162671 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.162645 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 12:51:12.162671 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.162651 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 12:51:12.162671 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.162659 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 12:51:12.162671 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.162668 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 12:51:12.163984 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.163965 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 12:51:12.163984 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.163977 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 12:51:12.167555 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.167542 2582 server.go:1263] "Started kubelet" Oct 30 12:51:12.167662 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.167585 2582 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 12:51:12.167717 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.167634 2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 12:51:12.167717 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.167694 2582 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 12:51:12.168301 ip-10-0-137-21 systemd[1]: Started Kubernetes Kubelet. Oct 30 12:51:12.168833 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.168812 2582 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 12:51:12.168928 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.168888 2582 server.go:317] "Adding debug handlers to kubelet server" Oct 30 12:51:12.175084 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.175065 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 12:51:12.175537 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.175516 2582 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 12:51:12.175987 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:12.175945 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-21.ec2.internal\" not found" Oct 30 12:51:12.176912 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.176895 2582 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 12:51:12.176912 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.176911 2582 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 12:51:12.177040 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.177030 2582 reconstruct.go:97] "Volume reconstruction finished" Oct 30 12:51:12.177040 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.177036 2582 reconciler.go:29] "Reconciler: start to sync state" Oct 30 12:51:12.177353 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.177335 2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 12:51:12.177482 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:12.177457 2582 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 12:51:12.177723 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.177703 2582 factory.go:55] Registering systemd factory Oct 30 12:51:12.177723 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.177721 2582 factory.go:223] Registration of the systemd container factory successfully Oct 30 12:51:12.177965 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.177947 2582 factory.go:153] Registering CRI-O factory Oct 30 12:51:12.178033 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.177968 2582 factory.go:223] Registration of the crio container factory successfully Oct 30 12:51:12.178033 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.178031 2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 12:51:12.178101 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.178051 2582 factory.go:103] Registering Raw factory Oct 30 12:51:12.178101 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.178072 2582 manager.go:1199] Started watching for new ooms in manager Oct 30 12:51:12.178994 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.178979 2582 manager.go:319] Starting recovery of all containers Oct 30 12:51:12.191422 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.191406 2582 manager.go:324] Recovery completed Oct 30 12:51:12.195503 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.195490 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:51:12.197449 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.197425 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-21.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:51:12.197508 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.197461 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-21.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:51:12.197508 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.197471 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-21.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:51:12.198635 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.198604 2582 cpu_manager.go:226] "Starting" policy="none" Oct 30 12:51:12.198635 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.198633 2582 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 12:51:12.198635 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.198608 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:51:12.198773 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.198654 2582 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 12:51:12.199368 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.199350 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-21.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:51:12.199439 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.199379 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-21.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:51:12.199439 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.199389 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-21.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:51:12.200848 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.200832 2582 policy_none.go:50] "Start" Oct 30 12:51:12.200916 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.200855 2582 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 12:51:12.200916 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.200869 2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 12:51:12.202903 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.202890 2582 policy_none.go:44] "Start" Oct 30 12:51:12.204124 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:12.204096 2582 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-21.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 12:51:12.204124 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.204109 2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-21.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 12:51:12.204973 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:12.204099 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-21.ec2.internal.18e34faaed479a86 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-21.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-21.ec2.internal,},FirstTimestamp:2026-10-30 12:51:12.167516806 +0000 UTC m=+0.342987696,LastTimestamp:2026-10-30 12:51:12.167516806 +0000 UTC m=+0.342987696,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-21.ec2.internal,}" Oct 30 12:51:12.222034 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:12.221997 2582 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-21.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 12:51:12.222034 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:12.221966 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-21.ec2.internal.18e34faaef105134 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-21.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-21.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-21.ec2.internal,},FirstTimestamp:2026-10-30 12:51:12.197447988 +0000 UTC m=+0.372918878,LastTimestamp:2026-10-30 12:51:12.197447988 +0000 UTC m=+0.372918878,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-21.ec2.internal,}" Oct 30 12:51:12.222034 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:12.222021 2582 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 12:51:12.222212 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:12.222152 2582 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 12:51:12.247339 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.247307 2582 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 12:51:12.248394 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:12.248331 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-21.ec2.internal.18e34faaef10974a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-21.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-21.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-21.ec2.internal,},FirstTimestamp:2026-10-30 12:51:12.19746593 +0000 UTC m=+0.372936821,LastTimestamp:2026-10-30 12:51:12.19746593 +0000 UTC m=+0.372936821,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-21.ec2.internal,}" Oct 30 12:51:12.271747 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.252559 2582 manager.go:341] "Starting Device Plugin manager" Oct 30 12:51:12.271747 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:12.252591 2582 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 12:51:12.271747 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.252603 2582 server.go:87] "Starting device plugin registration server" Oct 30 12:51:12.271747 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.252895 2582 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 12:51:12.271747 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.252906 2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 12:51:12.271747 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.253030 2582 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 12:51:12.271747 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.253116 2582 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 12:51:12.271747 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.253124 2582 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 12:51:12.271747 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:12.253688 2582 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 12:51:12.271747 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:12.253749 2582 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-21.ec2.internal\" not found" Oct 30 12:51:12.273620 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.272782 2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 12:51:12.273620 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.272818 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-b6tzd" Oct 30 12:51:12.273620 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:12.273474 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-21.ec2.internal.18e34faaef10b99a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-21.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-137-21.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-137-21.ec2.internal,},FirstTimestamp:2026-10-30 12:51:12.197474714 +0000 UTC m=+0.372945605,LastTimestamp:2026-10-30 12:51:12.197474714 +0000 UTC m=+0.372945605,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-21.ec2.internal,}" Oct 30 12:51:12.284688 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.284672 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-b6tzd" Oct 30 12:51:12.290811 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.290789 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 12:51:12.291983 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.291967 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 12:51:12.292060 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.291988 2582 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 12:51:12.292060 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.292005 2582 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 12:51:12.292060 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:12.292033 2582 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 12:51:12.295573 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:12.295513 2582 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-21.ec2.internal.18e34faaef105134\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-21.ec2.internal.18e34faaef105134 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-21.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-21.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-21.ec2.internal,},FirstTimestamp:2026-10-30 12:51:12.197447988 +0000 UTC m=+0.372918878,LastTimestamp:2026-10-30 12:51:12.199366869 +0000 UTC m=+0.374837762,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-21.ec2.internal,}" Oct 30 12:51:12.318522 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.318500 2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:51:12.353293 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.353254 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:51:12.354061 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.354047 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-21.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:51:12.354124 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.354073 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-21.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:51:12.354124 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.354082 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-21.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:51:12.354124 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.354103 2582 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-21.ec2.internal" Oct 30 12:51:12.370386 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.370365 2582 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-21.ec2.internal" Oct 30 12:51:12.370472 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:12.370390 2582 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-21.ec2.internal\": node \"ip-10-0-137-21.ec2.internal\" not found" Oct 30 12:51:12.393087 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.393054 2582 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-21.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-21.ec2.internal"] Oct 30 12:51:12.393157 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.393122 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:51:12.393855 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.393839 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-21.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:51:12.393914 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.393870 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-21.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:51:12.393914 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.393880 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-21.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:51:12.395763 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.395752 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:51:12.395917 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.395904 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-21.ec2.internal" Oct 30 12:51:12.395956 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.395932 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:51:12.396540 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.396520 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-21.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:51:12.396540 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.396531 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-21.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:51:12.396540 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.396545 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-21.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:51:12.396701 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.396552 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-21.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:51:12.396701 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.396555 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-21.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:51:12.396701 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.396562 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-21.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:51:12.398495 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.398481 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-21.ec2.internal" Oct 30 12:51:12.398535 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.398507 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:51:12.399234 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.399220 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-21.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:51:12.399295 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.399247 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-21.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:51:12.399295 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.399257 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-21.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:51:12.410153 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:12.410136 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-21.ec2.internal\" not found" Oct 30 12:51:12.431798 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:12.431782 2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-21.ec2.internal\" not found" node="ip-10-0-137-21.ec2.internal" Oct 30 12:51:12.434469 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:12.434455 2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-21.ec2.internal\" not found" node="ip-10-0-137-21.ec2.internal" Oct 30 12:51:12.510670 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:12.510653 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-21.ec2.internal\" not found" Oct 30 12:51:12.578766 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.578745 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c7d15293c60c5848ccd82e7130b45838-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-21.ec2.internal\" (UID: \"c7d15293c60c5848ccd82e7130b45838\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-21.ec2.internal" Oct 30 12:51:12.578835 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.578769 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7d15293c60c5848ccd82e7130b45838-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-21.ec2.internal\" (UID: \"c7d15293c60c5848ccd82e7130b45838\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-21.ec2.internal" Oct 30 12:51:12.578835 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.578785 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/100438b9fc00ed25c10ae1227a1de159-config\") pod \"kube-apiserver-proxy-ip-10-0-137-21.ec2.internal\" (UID: \"100438b9fc00ed25c10ae1227a1de159\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-21.ec2.internal" Oct 30 12:51:12.611334 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:12.611297 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-21.ec2.internal\" not found" Oct 30 12:51:12.679352 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.679334 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c7d15293c60c5848ccd82e7130b45838-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-21.ec2.internal\" (UID: \"c7d15293c60c5848ccd82e7130b45838\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-21.ec2.internal" Oct 30 12:51:12.679414 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.679357 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7d15293c60c5848ccd82e7130b45838-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-21.ec2.internal\" (UID: \"c7d15293c60c5848ccd82e7130b45838\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-21.ec2.internal" Oct 30 12:51:12.679414 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.679373 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/100438b9fc00ed25c10ae1227a1de159-config\") pod \"kube-apiserver-proxy-ip-10-0-137-21.ec2.internal\" (UID: \"100438b9fc00ed25c10ae1227a1de159\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-21.ec2.internal" Oct 30 12:51:12.679487 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.679426 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/100438b9fc00ed25c10ae1227a1de159-config\") pod \"kube-apiserver-proxy-ip-10-0-137-21.ec2.internal\" (UID: \"100438b9fc00ed25c10ae1227a1de159\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-21.ec2.internal" Oct 30 12:51:12.679487 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.679437 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c7d15293c60c5848ccd82e7130b45838-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-21.ec2.internal\" (UID: \"c7d15293c60c5848ccd82e7130b45838\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-21.ec2.internal" Oct 30 12:51:12.679487 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.679444 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7d15293c60c5848ccd82e7130b45838-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-21.ec2.internal\" (UID: \"c7d15293c60c5848ccd82e7130b45838\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-21.ec2.internal" Oct 30 12:51:12.711577 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:12.711559 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-21.ec2.internal\" not found" Oct 30 12:51:12.734835 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.734819 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-21.ec2.internal" Oct 30 12:51:12.737733 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:12.737719 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-21.ec2.internal" Oct 30 12:51:12.812156 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:12.812134 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-21.ec2.internal\" not found" Oct 30 12:51:12.912678 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:12.912634 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-21.ec2.internal\" not found" Oct 30 12:51:13.013153 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:13.013132 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-21.ec2.internal\" not found" Oct 30 12:51:13.113657 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:13.113634 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-21.ec2.internal\" not found" Oct 30 12:51:13.115781 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:13.115768 2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 12:51:13.115898 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:13.115880 2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 12:51:13.214512 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:13.214282 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-21.ec2.internal\" not found" Oct 30 12:51:13.264048 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:13.264020 2582 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:51:13.288698 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:13.288638 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 12:46:12 +0000 UTC" deadline="2028-05-15 12:59:42.597934011 +0000 UTC" Oct 30 12:51:13.288698 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:13.288688 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13512h8m29.309248663s" Oct 30 12:51:13.315223 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:13.315203 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-21.ec2.internal\" not found" Oct 30 12:51:13.325695 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:13.325657 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod100438b9fc00ed25c10ae1227a1de159.slice/crio-c8edd7fed57efb3f2fe9eb4b5689338b4c619b94606f3223fac094499fd99a54 WatchSource:0}: Error finding container c8edd7fed57efb3f2fe9eb4b5689338b4c619b94606f3223fac094499fd99a54: Status 404 returned error can't find the container with id c8edd7fed57efb3f2fe9eb4b5689338b4c619b94606f3223fac094499fd99a54 Oct 30 12:51:13.329715 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:13.329682 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7d15293c60c5848ccd82e7130b45838.slice/crio-cfaf6eb0e32f16cf0289059ab69c1e2d5690928fa9d7a16ea1a6cecf114b053c WatchSource:0}: Error finding container cfaf6eb0e32f16cf0289059ab69c1e2d5690928fa9d7a16ea1a6cecf114b053c: Status 404 returned error can't find the container with id cfaf6eb0e32f16cf0289059ab69c1e2d5690928fa9d7a16ea1a6cecf114b053c Oct 30 12:51:13.333820 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:13.333800 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:51:13.333919 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:13.333888 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 12:51:13.335338 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:13.335322 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-21.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:51:13.335410 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:13.335349 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-21.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:51:13.335410 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:13.335361 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-21.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:51:13.415866 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:13.415840 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-21.ec2.internal\" not found" Oct 30 12:51:13.517998 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:13.517947 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-21.ec2.internal\" not found" Oct 30 12:51:13.582263 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:13.582242 2582 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:51:13.620133 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:13.620110 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-21.ec2.internal\" not found" Oct 30 12:51:13.721081 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:13.721037 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-21.ec2.internal\" not found" Oct 30 12:51:13.814799 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:13.814723 2582 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:51:13.877395 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:13.877359 2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-21.ec2.internal" Oct 30 12:51:13.920116 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:13.920089 2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 12:51:13.921190 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:13.921168 2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-21.ec2.internal" Oct 30 12:51:13.949388 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:13.949364 2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 12:51:14.154313 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.154237 2582 apiserver.go:52] "Watching apiserver" Oct 30 12:51:14.179013 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.178960 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 12:51:14.197638 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.197606 2582 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 12:51:14.198229 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.198206 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-9szkr","openshift-multus/multus-additional-cni-plugins-75bqd","openshift-multus/multus-sw8rj","openshift-network-operator/iptables-alerter-zvw7s","openshift-cluster-node-tuning-operator/tuned-k76bj","openshift-image-registry/node-ca-bcbg8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-21.ec2.internal","openshift-ovn-kubernetes/ovnkube-node-nwrx6","kube-system/global-pull-secret-syncer-s7fsp","openshift-multus/network-metrics-daemon-tpz8b","openshift-network-diagnostics/network-check-target-95n9l","kube-system/konnectivity-agent-2nqhp","kube-system/kube-apiserver-proxy-ip-10-0-137-21.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66"] Oct 30 12:51:14.201167 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.201138 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpz8b" Oct 30 12:51:14.201289 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:14.201248 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpz8b" podUID="1fcccc7b-a83c-4c52-a648-cb549e70b54d" Oct 30 12:51:14.203406 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.203382 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95n9l" Oct 30 12:51:14.203530 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:14.203460 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95n9l" podUID="1bb48bfa-827b-4179-b9fa-418fc14f0f44" Oct 30 12:51:14.205688 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.205659 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.208052 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.208032 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zvw7s" Oct 30 12:51:14.210498 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.210472 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:51:14.213830 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.213811 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bcbg8" Oct 30 12:51:14.216349 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.216328 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:51:14.216430 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:14.216404 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7fsp" podUID="50bec85b-5592-437d-aa6f-4899aeb494a5" Oct 30 12:51:14.220886 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.220482 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" Oct 30 12:51:14.223059 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.223035 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-75bqd" Oct 30 12:51:14.225470 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.225448 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9szkr" Oct 30 12:51:14.227978 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.227958 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.229664 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.229644 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-sfhnn\"" Oct 30 12:51:14.230916 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.230900 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2nqhp" Oct 30 12:51:14.232417 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.232370 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-48745\"" Oct 30 12:51:14.236293 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.234927 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 12:51:14.236293 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.235184 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 12:51:14.236293 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.235250 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 12:51:14.236293 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.235321 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:51:14.236293 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.235472 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 12:51:14.236293 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.235522 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:51:14.236293 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.235599 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 12:51:14.236293 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.235981 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 12:51:14.236293 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.236029 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 12:51:14.236293 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.236041 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 12:51:14.236293 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.236108 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 12:51:14.236293 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.236120 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 12:51:14.280090 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.280051 2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 12:51:14.289425 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.289399 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bea99b12-e97c-4c0e-9360-07e542a4294d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dsh66\" (UID: \"bea99b12-e97c-4c0e-9360-07e542a4294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" Oct 30 12:51:14.289532 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.289442 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fe19a15d-5112-4b76-a226-4b8462e2fb33-hosts-file\") pod \"node-resolver-9szkr\" (UID: \"fe19a15d-5112-4b76-a226-4b8462e2fb33\") " pod="openshift-dns/node-resolver-9szkr" Oct 30 12:51:14.289532 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.289484 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-host-slash\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.289672 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.289552 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.289672 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.289610 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-hostroot\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.289672 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.289662 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50bec85b-5592-437d-aa6f-4899aeb494a5-original-pull-secret\") pod \"global-pull-secret-syncer-s7fsp\" (UID: \"50bec85b-5592-437d-aa6f-4899aeb494a5\") " pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:51:14.289827 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.289691 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bea99b12-e97c-4c0e-9360-07e542a4294d-sys-fs\") pod \"aws-ebs-csi-driver-node-dsh66\" (UID: \"bea99b12-e97c-4c0e-9360-07e542a4294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" Oct 30 12:51:14.289827 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.289715 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-multus-cni-dir\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.289827 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.289740 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-multus-socket-dir-parent\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.289827 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.289797 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-host-var-lib-kubelet\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.290033 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.289981 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9x52\" (UniqueName: \"kubernetes.io/projected/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-kube-api-access-x9x52\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.290079 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.290052 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee23e57d-7b77-4dba-95a5-84ad59133981-system-cni-dir\") pod \"multus-additional-cni-plugins-75bqd\" (UID: \"ee23e57d-7b77-4dba-95a5-84ad59133981\") " pod="openshift-multus/multus-additional-cni-plugins-75bqd" Oct 30 12:51:14.290133 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.290104 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwddx\" (UniqueName: \"kubernetes.io/projected/ee23e57d-7b77-4dba-95a5-84ad59133981-kube-api-access-hwddx\") pod \"multus-additional-cni-plugins-75bqd\" (UID: \"ee23e57d-7b77-4dba-95a5-84ad59133981\") " pod="openshift-multus/multus-additional-cni-plugins-75bqd" Oct 30 12:51:14.290191 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.290134 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3f38af1e-cd92-4cd8-a643-151b10b8da12-agent-certs\") pod \"konnectivity-agent-2nqhp\" (UID: \"3f38af1e-cd92-4cd8-a643-151b10b8da12\") " pod="kube-system/konnectivity-agent-2nqhp" Oct 30 12:51:14.290191 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.290159 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bea99b12-e97c-4c0e-9360-07e542a4294d-socket-dir\") pod \"aws-ebs-csi-driver-node-dsh66\" (UID: \"bea99b12-e97c-4c0e-9360-07e542a4294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" Oct 30 12:51:14.290294 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.290223 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-run-openvswitch\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.290294 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.290255 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-host-cni-netd\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.290294 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.290289 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d9b03988-7080-4352-ba1e-08ee68ba3df8-env-overrides\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.290448 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.290314 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d9b03988-7080-4352-ba1e-08ee68ba3df8-ovnkube-script-lib\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.290448 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.290340 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-cni-binary-copy\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.290448 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.290366 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ee23e57d-7b77-4dba-95a5-84ad59133981-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-75bqd\" (UID: \"ee23e57d-7b77-4dba-95a5-84ad59133981\") " pod="openshift-multus/multus-additional-cni-plugins-75bqd" Oct 30 12:51:14.290448 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.290418 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fcccc7b-a83c-4c52-a648-cb549e70b54d-metrics-certs\") pod \"network-metrics-daemon-tpz8b\" (UID: \"1fcccc7b-a83c-4c52-a648-cb549e70b54d\") " pod="openshift-multus/network-metrics-daemon-tpz8b" Oct 30 12:51:14.290658 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.290448 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11a0b508-2f9c-4be6-b5c7-b3d74766e801-host-slash\") pod \"iptables-alerter-zvw7s\" (UID: \"11a0b508-2f9c-4be6-b5c7-b3d74766e801\") " pod="openshift-network-operator/iptables-alerter-zvw7s" Oct 30 12:51:14.290658 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.290473 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bea99b12-e97c-4c0e-9360-07e542a4294d-etc-selinux\") pod \"aws-ebs-csi-driver-node-dsh66\" (UID: \"bea99b12-e97c-4c0e-9360-07e542a4294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" Oct 30 12:51:14.290658 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.290501 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-host-var-lib-cni-multus\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.290658 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.290528 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-etc-sysconfig\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:51:14.290658 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.290552 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/66172ef8-aaad-4875-b3b3-bd81b9e9ba59-serviceca\") pod \"node-ca-bcbg8\" (UID: \"66172ef8-aaad-4875-b3b3-bd81b9e9ba59\") " pod="openshift-image-registry/node-ca-bcbg8" Oct 30 12:51:14.290658 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.290613 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvxmx\" (UniqueName: \"kubernetes.io/projected/66172ef8-aaad-4875-b3b3-bd81b9e9ba59-kube-api-access-wvxmx\") pod \"node-ca-bcbg8\" (UID: \"66172ef8-aaad-4875-b3b3-bd81b9e9ba59\") " pod="openshift-image-registry/node-ca-bcbg8" Oct 30 12:51:14.291005 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.290645 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bea99b12-e97c-4c0e-9360-07e542a4294d-device-dir\") pod \"aws-ebs-csi-driver-node-dsh66\" (UID: \"bea99b12-e97c-4c0e-9360-07e542a4294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" Oct 30 12:51:14.291005 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.290692 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-host-cni-bin\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.291005 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.290726 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-host-run-multus-certs\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.291005 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.290774 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-sys\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:51:14.291005 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.290801 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-host-kubelet\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.291005 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.290826 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-host-run-k8s-cni-cncf-io\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.291005 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.290851 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-host-run-netns\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.291005 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.290877 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-host-var-lib-cni-bin\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.291005 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.290903 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/accde534-15e4-42cf-94df-c0aa085f0603-tmp\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:51:14.291005 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.290930 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-etc-openvswitch\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.291005 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.290955 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-run-ovn\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.291005 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.290981 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-log-socket\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.291596 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.291079 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-system-cni-dir\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.291596 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.291114 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-multus-daemon-config\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.291596 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.291141 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/50bec85b-5592-437d-aa6f-4899aeb494a5-kubelet-config\") pod \"global-pull-secret-syncer-s7fsp\" (UID: \"50bec85b-5592-437d-aa6f-4899aeb494a5\") " pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:51:14.291596 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.291170 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/accde534-15e4-42cf-94df-c0aa085f0603-etc-tuned\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:51:14.291596 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.291197 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp4zz\" (UniqueName: \"kubernetes.io/projected/d9b03988-7080-4352-ba1e-08ee68ba3df8-kube-api-access-qp4zz\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.291596 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.291224 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqvdp\" (UniqueName: \"kubernetes.io/projected/1bb48bfa-827b-4179-b9fa-418fc14f0f44-kube-api-access-dqvdp\") pod \"network-check-target-95n9l\" (UID: \"1bb48bfa-827b-4179-b9fa-418fc14f0f44\") " pod="openshift-network-diagnostics/network-check-target-95n9l" Oct 30 12:51:14.291596 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.291257 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee23e57d-7b77-4dba-95a5-84ad59133981-tuning-conf-dir\") pod \"multus-additional-cni-plugins-75bqd\" (UID: \"ee23e57d-7b77-4dba-95a5-84ad59133981\") " pod="openshift-multus/multus-additional-cni-plugins-75bqd" Oct 30 12:51:14.291596 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.291284 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3f38af1e-cd92-4cd8-a643-151b10b8da12-konnectivity-ca\") pod \"konnectivity-agent-2nqhp\" (UID: \"3f38af1e-cd92-4cd8-a643-151b10b8da12\") " pod="kube-system/konnectivity-agent-2nqhp" Oct 30 12:51:14.291596 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.291317 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxwlx\" (UniqueName: \"kubernetes.io/projected/1fcccc7b-a83c-4c52-a648-cb549e70b54d-kube-api-access-xxwlx\") pod \"network-metrics-daemon-tpz8b\" (UID: \"1fcccc7b-a83c-4c52-a648-cb549e70b54d\") " pod="openshift-multus/network-metrics-daemon-tpz8b" Oct 30 12:51:14.291596 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.291365 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-lib-modules\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:51:14.291596 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.291404 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bea99b12-e97c-4c0e-9360-07e542a4294d-registration-dir\") pod \"aws-ebs-csi-driver-node-dsh66\" (UID: \"bea99b12-e97c-4c0e-9360-07e542a4294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" Oct 30 12:51:14.291596 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.291435 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe19a15d-5112-4b76-a226-4b8462e2fb33-tmp-dir\") pod \"node-resolver-9szkr\" (UID: \"fe19a15d-5112-4b76-a226-4b8462e2fb33\") " pod="openshift-dns/node-resolver-9szkr" Oct 30 12:51:14.291596 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.291470 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-run-systemd\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.291596 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.291510 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-var-lib-openvswitch\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.291596 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.291540 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d9b03988-7080-4352-ba1e-08ee68ba3df8-ovn-node-metrics-cert\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.291596 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.291583 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/50bec85b-5592-437d-aa6f-4899aeb494a5-dbus\") pod \"global-pull-secret-syncer-s7fsp\" (UID: \"50bec85b-5592-437d-aa6f-4899aeb494a5\") " pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:51:14.292356 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.291626 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee23e57d-7b77-4dba-95a5-84ad59133981-cnibin\") pod \"multus-additional-cni-plugins-75bqd\" (UID: \"ee23e57d-7b77-4dba-95a5-84ad59133981\") " pod="openshift-multus/multus-additional-cni-plugins-75bqd" Oct 30 12:51:14.292356 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.291671 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-etc-sysctl-conf\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:51:14.292356 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.291714 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-node-log\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.292356 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.291774 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-run\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:51:14.292356 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.291815 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-var-lib-kubelet\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:51:14.292356 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.291852 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-host-run-netns\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.292356 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.291895 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d9b03988-7080-4352-ba1e-08ee68ba3df8-ovnkube-config\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.292356 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.291920 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-etc-systemd\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:51:14.292356 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.292025 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8fhd\" (UniqueName: \"kubernetes.io/projected/bea99b12-e97c-4c0e-9360-07e542a4294d-kube-api-access-m8fhd\") pod \"aws-ebs-csi-driver-node-dsh66\" (UID: \"bea99b12-e97c-4c0e-9360-07e542a4294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" Oct 30 12:51:14.292356 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.292073 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-host-run-ovn-kubernetes\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.292356 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.292115 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-cnibin\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.292356 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.292158 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/11a0b508-2f9c-4be6-b5c7-b3d74766e801-iptables-alerter-script\") pod \"iptables-alerter-zvw7s\" (UID: \"11a0b508-2f9c-4be6-b5c7-b3d74766e801\") " pod="openshift-network-operator/iptables-alerter-zvw7s" Oct 30 12:51:14.292356 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.292186 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-etc-kubernetes\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:51:14.292356 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.292213 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wksmn\" (UniqueName: \"kubernetes.io/projected/accde534-15e4-42cf-94df-c0aa085f0603-kube-api-access-wksmn\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:51:14.292356 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.292240 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/66172ef8-aaad-4875-b3b3-bd81b9e9ba59-host\") pod \"node-ca-bcbg8\" (UID: \"66172ef8-aaad-4875-b3b3-bd81b9e9ba59\") " pod="openshift-image-registry/node-ca-bcbg8" Oct 30 12:51:14.292356 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.292310 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpgrm\" (UniqueName: \"kubernetes.io/projected/fe19a15d-5112-4b76-a226-4b8462e2fb33-kube-api-access-qpgrm\") pod \"node-resolver-9szkr\" (UID: \"fe19a15d-5112-4b76-a226-4b8462e2fb33\") " pod="openshift-dns/node-resolver-9szkr" Oct 30 12:51:14.293167 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.292352 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-multus-conf-dir\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.293167 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.292386 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-etc-kubernetes\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.293167 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.292418 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkldf\" (UniqueName: \"kubernetes.io/projected/11a0b508-2f9c-4be6-b5c7-b3d74766e801-kube-api-access-tkldf\") pod \"iptables-alerter-zvw7s\" (UID: \"11a0b508-2f9c-4be6-b5c7-b3d74766e801\") " pod="openshift-network-operator/iptables-alerter-zvw7s" Oct 30 12:51:14.293336 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.293164 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee23e57d-7b77-4dba-95a5-84ad59133981-cni-binary-copy\") pod \"multus-additional-cni-plugins-75bqd\" (UID: \"ee23e57d-7b77-4dba-95a5-84ad59133981\") " pod="openshift-multus/multus-additional-cni-plugins-75bqd" Oct 30 12:51:14.293336 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.293206 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ee23e57d-7b77-4dba-95a5-84ad59133981-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-75bqd\" (UID: \"ee23e57d-7b77-4dba-95a5-84ad59133981\") " pod="openshift-multus/multus-additional-cni-plugins-75bqd" Oct 30 12:51:14.293336 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.293239 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-etc-modprobe-d\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:51:14.293336 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.293266 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-host\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:51:14.293336 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.293294 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-systemd-units\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.293336 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.293319 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-etc-sysctl-d\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:51:14.296730 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.296646 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-21.ec2.internal" event={"ID":"c7d15293c60c5848ccd82e7130b45838","Type":"ContainerStarted","Data":"cfaf6eb0e32f16cf0289059ab69c1e2d5690928fa9d7a16ea1a6cecf114b053c"} Oct 30 12:51:14.298070 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.298045 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-21.ec2.internal" event={"ID":"100438b9fc00ed25c10ae1227a1de159","Type":"ContainerStarted","Data":"c8edd7fed57efb3f2fe9eb4b5689338b4c619b94606f3223fac094499fd99a54"} Oct 30 12:51:14.338475 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.338444 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 12:51:14.338588 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.338568 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 12:51:14.338722 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.338701 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xj7mb\"" Oct 30 12:51:14.338812 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.338734 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-m2lrf\"" Oct 30 12:51:14.338812 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.338765 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 12:51:14.338910 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.338826 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7j54b\"" Oct 30 12:51:14.338910 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.338841 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-87w8v\"" Oct 30 12:51:14.338910 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.338868 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qjjfj\"" Oct 30 12:51:14.338910 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.338900 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-v4xk8\"" Oct 30 12:51:14.339056 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.338944 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9ppdm\"" Oct 30 12:51:14.343179 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.343158 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 12:51:14.343294 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.343280 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 12:51:14.343578 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.343553 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 12:51:14.343696 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.343628 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 12:51:14.343696 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.343655 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 12:51:14.343696 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.343662 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 12:51:14.343867 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.343773 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 12:51:14.343867 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.343782 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 12:51:14.343867 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.343777 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 12:51:14.343867 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.343778 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 12:51:14.343867 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.343801 2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 12:51:14.343867 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.343851 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4xhw5" Oct 30 12:51:14.344122 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.343875 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 12:51:14.344236 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.344218 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 12:51:14.394337 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.394315 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-host-slash\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.394474 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.394342 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.394474 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.394368 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-hostroot\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.394474 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.394395 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50bec85b-5592-437d-aa6f-4899aeb494a5-original-pull-secret\") pod \"global-pull-secret-syncer-s7fsp\" (UID: \"50bec85b-5592-437d-aa6f-4899aeb494a5\") " pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:51:14.394474 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.394414 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bea99b12-e97c-4c0e-9360-07e542a4294d-sys-fs\") pod \"aws-ebs-csi-driver-node-dsh66\" (UID: \"bea99b12-e97c-4c0e-9360-07e542a4294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" Oct 30 12:51:14.394474 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.394436 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-multus-cni-dir\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.394474 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.394449 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.394836 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.394523 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-hostroot\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.394836 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:14.394554 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 12:51:14.394836 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.394565 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-host-slash\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.394836 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.394461 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-multus-socket-dir-parent\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.394836 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.394607 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-host-var-lib-kubelet\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.394836 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.394655 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9x52\" (UniqueName: \"kubernetes.io/projected/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-kube-api-access-x9x52\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.394836 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.394682 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee23e57d-7b77-4dba-95a5-84ad59133981-system-cni-dir\") pod \"multus-additional-cni-plugins-75bqd\" (UID: \"ee23e57d-7b77-4dba-95a5-84ad59133981\") " pod="openshift-multus/multus-additional-cni-plugins-75bqd" Oct 30 12:51:14.394836 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.394726 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-multus-socket-dir-parent\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.394836 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:14.394806 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50bec85b-5592-437d-aa6f-4899aeb494a5-original-pull-secret podName:50bec85b-5592-437d-aa6f-4899aeb494a5 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:14.894779845 +0000 UTC m=+3.070250743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/50bec85b-5592-437d-aa6f-4899aeb494a5-original-pull-secret") pod "global-pull-secret-syncer-s7fsp" (UID: "50bec85b-5592-437d-aa6f-4899aeb494a5") : object "kube-system"/"original-pull-secret" not registered Oct 30 12:51:14.395295 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.394919 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bea99b12-e97c-4c0e-9360-07e542a4294d-sys-fs\") pod \"aws-ebs-csi-driver-node-dsh66\" (UID: \"bea99b12-e97c-4c0e-9360-07e542a4294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" Oct 30 12:51:14.395295 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.394967 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-multus-cni-dir\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.395295 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.395003 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee23e57d-7b77-4dba-95a5-84ad59133981-system-cni-dir\") pod \"multus-additional-cni-plugins-75bqd\" (UID: \"ee23e57d-7b77-4dba-95a5-84ad59133981\") " pod="openshift-multus/multus-additional-cni-plugins-75bqd" Oct 30 12:51:14.395295 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.395030 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-host-var-lib-kubelet\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.395295 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.395036 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwddx\" (UniqueName: \"kubernetes.io/projected/ee23e57d-7b77-4dba-95a5-84ad59133981-kube-api-access-hwddx\") pod \"multus-additional-cni-plugins-75bqd\" (UID: \"ee23e57d-7b77-4dba-95a5-84ad59133981\") " pod="openshift-multus/multus-additional-cni-plugins-75bqd" Oct 30 12:51:14.395295 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.395083 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3f38af1e-cd92-4cd8-a643-151b10b8da12-agent-certs\") pod \"konnectivity-agent-2nqhp\" (UID: \"3f38af1e-cd92-4cd8-a643-151b10b8da12\") " pod="kube-system/konnectivity-agent-2nqhp" Oct 30 12:51:14.395295 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.395112 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bea99b12-e97c-4c0e-9360-07e542a4294d-socket-dir\") pod \"aws-ebs-csi-driver-node-dsh66\" (UID: \"bea99b12-e97c-4c0e-9360-07e542a4294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" Oct 30 12:51:14.395295 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.395151 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-run-openvswitch\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.395295 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.395175 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-host-cni-netd\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.395714 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.395315 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-host-cni-netd\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.395714 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.395357 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bea99b12-e97c-4c0e-9360-07e542a4294d-socket-dir\") pod \"aws-ebs-csi-driver-node-dsh66\" (UID: \"bea99b12-e97c-4c0e-9360-07e542a4294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" Oct 30 12:51:14.395714 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.395356 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d9b03988-7080-4352-ba1e-08ee68ba3df8-env-overrides\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.395714 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.395385 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d9b03988-7080-4352-ba1e-08ee68ba3df8-ovnkube-script-lib\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.395714 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.395377 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-run-openvswitch\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.395714 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.395402 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-cni-binary-copy\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.395714 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.395426 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ee23e57d-7b77-4dba-95a5-84ad59133981-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-75bqd\" (UID: \"ee23e57d-7b77-4dba-95a5-84ad59133981\") " pod="openshift-multus/multus-additional-cni-plugins-75bqd" Oct 30 12:51:14.395714 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.395452 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fcccc7b-a83c-4c52-a648-cb549e70b54d-metrics-certs\") pod \"network-metrics-daemon-tpz8b\" (UID: \"1fcccc7b-a83c-4c52-a648-cb549e70b54d\") " pod="openshift-multus/network-metrics-daemon-tpz8b" Oct 30 12:51:14.395714 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.395477 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11a0b508-2f9c-4be6-b5c7-b3d74766e801-host-slash\") pod \"iptables-alerter-zvw7s\" (UID: \"11a0b508-2f9c-4be6-b5c7-b3d74766e801\") " pod="openshift-network-operator/iptables-alerter-zvw7s" Oct 30 12:51:14.395714 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.395501 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bea99b12-e97c-4c0e-9360-07e542a4294d-etc-selinux\") pod \"aws-ebs-csi-driver-node-dsh66\" (UID: \"bea99b12-e97c-4c0e-9360-07e542a4294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" Oct 30 12:51:14.395714 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.395526 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-host-var-lib-cni-multus\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.395714 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.395551 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-etc-sysconfig\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:51:14.395714 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.395574 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/66172ef8-aaad-4875-b3b3-bd81b9e9ba59-serviceca\") pod \"node-ca-bcbg8\" (UID: \"66172ef8-aaad-4875-b3b3-bd81b9e9ba59\") " pod="openshift-image-registry/node-ca-bcbg8" Oct 30 12:51:14.395714 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:14.395597 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:51:14.395714 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.395603 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvxmx\" (UniqueName: \"kubernetes.io/projected/66172ef8-aaad-4875-b3b3-bd81b9e9ba59-kube-api-access-wvxmx\") pod \"node-ca-bcbg8\" (UID: \"66172ef8-aaad-4875-b3b3-bd81b9e9ba59\") " pod="openshift-image-registry/node-ca-bcbg8" Oct 30 12:51:14.395714 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.395627 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bea99b12-e97c-4c0e-9360-07e542a4294d-device-dir\") pod \"aws-ebs-csi-driver-node-dsh66\" (UID: \"bea99b12-e97c-4c0e-9360-07e542a4294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" Oct 30 12:51:14.395714 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:14.395671 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fcccc7b-a83c-4c52-a648-cb549e70b54d-metrics-certs podName:1fcccc7b-a83c-4c52-a648-cb549e70b54d nodeName:}" failed. No retries permitted until 2026-10-30 12:51:14.895654381 +0000 UTC m=+3.071125283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1fcccc7b-a83c-4c52-a648-cb549e70b54d-metrics-certs") pod "network-metrics-daemon-tpz8b" (UID: "1fcccc7b-a83c-4c52-a648-cb549e70b54d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:51:14.396556 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.395692 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-host-cni-bin\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.396556 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.395703 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bea99b12-e97c-4c0e-9360-07e542a4294d-device-dir\") pod \"aws-ebs-csi-driver-node-dsh66\" (UID: \"bea99b12-e97c-4c0e-9360-07e542a4294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" Oct 30 12:51:14.396556 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.395723 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-host-run-multus-certs\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.396556 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.395748 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-sys\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:51:14.396556 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.395749 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11a0b508-2f9c-4be6-b5c7-b3d74766e801-host-slash\") pod \"iptables-alerter-zvw7s\" (UID: \"11a0b508-2f9c-4be6-b5c7-b3d74766e801\") " pod="openshift-network-operator/iptables-alerter-zvw7s" Oct 30 12:51:14.396556 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.395841 2582 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 12:51:14.396556 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.395836 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bea99b12-e97c-4c0e-9360-07e542a4294d-etc-selinux\") pod \"aws-ebs-csi-driver-node-dsh66\" (UID: \"bea99b12-e97c-4c0e-9360-07e542a4294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" Oct 30 12:51:14.396556 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.395879 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-etc-sysconfig\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:51:14.396556 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.395897 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-host-kubelet\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.396556 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.395927 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-host-run-k8s-cni-cncf-io\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.396556 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.395938 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-host-var-lib-cni-multus\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.396556 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.395952 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-host-run-netns\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.396556 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.395978 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-host-var-lib-cni-bin\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.396556 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.396002 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/accde534-15e4-42cf-94df-c0aa085f0603-tmp\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:51:14.396556 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.396026 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-etc-openvswitch\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.396556 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.396051 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-run-ovn\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.396556 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.396078 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-log-socket\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.396556 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.396088 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d9b03988-7080-4352-ba1e-08ee68ba3df8-env-overrides\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.397532 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.396102 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-system-cni-dir\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.397532 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.396131 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-multus-daemon-config\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.397532 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.396158 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/50bec85b-5592-437d-aa6f-4899aeb494a5-kubelet-config\") pod \"global-pull-secret-syncer-s7fsp\" (UID: \"50bec85b-5592-437d-aa6f-4899aeb494a5\") " pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:51:14.397532 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.396183 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/accde534-15e4-42cf-94df-c0aa085f0603-etc-tuned\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:51:14.397532 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.396197 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-cni-binary-copy\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.397532 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.396210 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp4zz\" (UniqueName: \"kubernetes.io/projected/d9b03988-7080-4352-ba1e-08ee68ba3df8-kube-api-access-qp4zz\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.397532 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.396254 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-etc-openvswitch\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.397532 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.396252 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d9b03988-7080-4352-ba1e-08ee68ba3df8-ovnkube-script-lib\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.397532 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.396310 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-host-run-netns\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.397532 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.396357 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-host-var-lib-cni-bin\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.397532 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.396373 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ee23e57d-7b77-4dba-95a5-84ad59133981-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-75bqd\" (UID: \"ee23e57d-7b77-4dba-95a5-84ad59133981\") " pod="openshift-multus/multus-additional-cni-plugins-75bqd" Oct 30 12:51:14.397532 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.396401 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-host-kubelet\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.397532 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.396436 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-host-cni-bin\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.397532 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.396456 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-host-run-k8s-cni-cncf-io\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.397532 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.396462 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/66172ef8-aaad-4875-b3b3-bd81b9e9ba59-serviceca\") pod \"node-ca-bcbg8\" (UID: \"66172ef8-aaad-4875-b3b3-bd81b9e9ba59\") " pod="openshift-image-registry/node-ca-bcbg8" Oct 30 12:51:14.397532 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.396485 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-sys\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:51:14.397532 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.396502 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-host-run-multus-certs\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.397532 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.396492 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-log-socket\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.398447 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.396519 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-run-ovn\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.398447 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.396571 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-system-cni-dir\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.398447 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.396559 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/50bec85b-5592-437d-aa6f-4899aeb494a5-kubelet-config\") pod \"global-pull-secret-syncer-s7fsp\" (UID: \"50bec85b-5592-437d-aa6f-4899aeb494a5\") " pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:51:14.398447 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.396604 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqvdp\" (UniqueName: \"kubernetes.io/projected/1bb48bfa-827b-4179-b9fa-418fc14f0f44-kube-api-access-dqvdp\") pod \"network-check-target-95n9l\" (UID: \"1bb48bfa-827b-4179-b9fa-418fc14f0f44\") " pod="openshift-network-diagnostics/network-check-target-95n9l" Oct 30 12:51:14.398447 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.396638 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee23e57d-7b77-4dba-95a5-84ad59133981-tuning-conf-dir\") pod \"multus-additional-cni-plugins-75bqd\" (UID: \"ee23e57d-7b77-4dba-95a5-84ad59133981\") " pod="openshift-multus/multus-additional-cni-plugins-75bqd" Oct 30 12:51:14.398447 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.396671 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3f38af1e-cd92-4cd8-a643-151b10b8da12-konnectivity-ca\") pod \"konnectivity-agent-2nqhp\" (UID: \"3f38af1e-cd92-4cd8-a643-151b10b8da12\") " pod="kube-system/konnectivity-agent-2nqhp" Oct 30 12:51:14.398447 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.396723 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxwlx\" (UniqueName: \"kubernetes.io/projected/1fcccc7b-a83c-4c52-a648-cb549e70b54d-kube-api-access-xxwlx\") pod \"network-metrics-daemon-tpz8b\" (UID: \"1fcccc7b-a83c-4c52-a648-cb549e70b54d\") " pod="openshift-multus/network-metrics-daemon-tpz8b" Oct 30 12:51:14.398447 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.396768 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-lib-modules\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:51:14.398447 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.396803 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bea99b12-e97c-4c0e-9360-07e542a4294d-registration-dir\") pod \"aws-ebs-csi-driver-node-dsh66\" (UID: \"bea99b12-e97c-4c0e-9360-07e542a4294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" Oct 30 12:51:14.398447 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.396835 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe19a15d-5112-4b76-a226-4b8462e2fb33-tmp-dir\") pod \"node-resolver-9szkr\" (UID: \"fe19a15d-5112-4b76-a226-4b8462e2fb33\") " pod="openshift-dns/node-resolver-9szkr" Oct 30 12:51:14.398447 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.396862 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-run-systemd\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.398447 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.396887 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-var-lib-openvswitch\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.398447 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.396910 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-lib-modules\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:51:14.398447 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.396914 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d9b03988-7080-4352-ba1e-08ee68ba3df8-ovn-node-metrics-cert\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.398447 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.396963 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/50bec85b-5592-437d-aa6f-4899aeb494a5-dbus\") pod \"global-pull-secret-syncer-s7fsp\" (UID: \"50bec85b-5592-437d-aa6f-4899aeb494a5\") " pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:51:14.398447 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.396986 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-multus-daemon-config\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.398447 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.396993 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee23e57d-7b77-4dba-95a5-84ad59133981-cnibin\") pod \"multus-additional-cni-plugins-75bqd\" (UID: \"ee23e57d-7b77-4dba-95a5-84ad59133981\") " pod="openshift-multus/multus-additional-cni-plugins-75bqd" Oct 30 12:51:14.399354 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397019 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-etc-sysctl-conf\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:51:14.399354 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397047 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-node-log\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.399354 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397058 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/50bec85b-5592-437d-aa6f-4899aeb494a5-dbus\") pod \"global-pull-secret-syncer-s7fsp\" (UID: \"50bec85b-5592-437d-aa6f-4899aeb494a5\") " pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:51:14.399354 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397071 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-run\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:51:14.399354 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397097 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-var-lib-kubelet\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:51:14.399354 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397127 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-host-run-netns\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.399354 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397152 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d9b03988-7080-4352-ba1e-08ee68ba3df8-ovnkube-config\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.399354 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397178 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-etc-systemd\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:51:14.399354 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397197 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-etc-sysctl-conf\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:51:14.399354 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397206 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8fhd\" (UniqueName: \"kubernetes.io/projected/bea99b12-e97c-4c0e-9360-07e542a4294d-kube-api-access-m8fhd\") pod \"aws-ebs-csi-driver-node-dsh66\" (UID: \"bea99b12-e97c-4c0e-9360-07e542a4294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" Oct 30 12:51:14.399354 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397226 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3f38af1e-cd92-4cd8-a643-151b10b8da12-konnectivity-ca\") pod \"konnectivity-agent-2nqhp\" (UID: \"3f38af1e-cd92-4cd8-a643-151b10b8da12\") " pod="kube-system/konnectivity-agent-2nqhp" Oct 30 12:51:14.399354 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397241 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-host-run-ovn-kubernetes\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.399354 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397269 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-cnibin\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.399354 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397276 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-node-log\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.399354 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397277 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bea99b12-e97c-4c0e-9360-07e542a4294d-registration-dir\") pod \"aws-ebs-csi-driver-node-dsh66\" (UID: \"bea99b12-e97c-4c0e-9360-07e542a4294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" Oct 30 12:51:14.399354 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397295 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/11a0b508-2f9c-4be6-b5c7-b3d74766e801-iptables-alerter-script\") pod \"iptables-alerter-zvw7s\" (UID: \"11a0b508-2f9c-4be6-b5c7-b3d74766e801\") " pod="openshift-network-operator/iptables-alerter-zvw7s" Oct 30 12:51:14.399354 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397331 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-etc-kubernetes\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:51:14.400256 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397361 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wksmn\" (UniqueName: \"kubernetes.io/projected/accde534-15e4-42cf-94df-c0aa085f0603-kube-api-access-wksmn\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:51:14.400256 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397341 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-run\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:51:14.400256 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397415 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-var-lib-kubelet\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:51:14.400256 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397420 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee23e57d-7b77-4dba-95a5-84ad59133981-cnibin\") pod \"multus-additional-cni-plugins-75bqd\" (UID: \"ee23e57d-7b77-4dba-95a5-84ad59133981\") " pod="openshift-multus/multus-additional-cni-plugins-75bqd" Oct 30 12:51:14.400256 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397440 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-host-run-ovn-kubernetes\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.400256 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397230 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe19a15d-5112-4b76-a226-4b8462e2fb33-tmp-dir\") pod \"node-resolver-9szkr\" (UID: \"fe19a15d-5112-4b76-a226-4b8462e2fb33\") " pod="openshift-dns/node-resolver-9szkr" Oct 30 12:51:14.400256 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397477 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-cnibin\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.400256 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397513 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee23e57d-7b77-4dba-95a5-84ad59133981-tuning-conf-dir\") pod \"multus-additional-cni-plugins-75bqd\" (UID: \"ee23e57d-7b77-4dba-95a5-84ad59133981\") " pod="openshift-multus/multus-additional-cni-plugins-75bqd" Oct 30 12:51:14.400256 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397530 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-etc-systemd\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:51:14.400256 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397570 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-host-run-netns\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.400256 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397565 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-run-systemd\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.400256 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397608 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-var-lib-openvswitch\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.400256 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397614 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/66172ef8-aaad-4875-b3b3-bd81b9e9ba59-host\") pod \"node-ca-bcbg8\" (UID: \"66172ef8-aaad-4875-b3b3-bd81b9e9ba59\") " pod="openshift-image-registry/node-ca-bcbg8" Oct 30 12:51:14.400256 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397639 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpgrm\" (UniqueName: \"kubernetes.io/projected/fe19a15d-5112-4b76-a226-4b8462e2fb33-kube-api-access-qpgrm\") pod \"node-resolver-9szkr\" (UID: \"fe19a15d-5112-4b76-a226-4b8462e2fb33\") " pod="openshift-dns/node-resolver-9szkr" Oct 30 12:51:14.400256 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397651 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-etc-kubernetes\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:51:14.400256 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397659 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-multus-conf-dir\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.400256 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397683 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-etc-kubernetes\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.400256 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397726 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkldf\" (UniqueName: \"kubernetes.io/projected/11a0b508-2f9c-4be6-b5c7-b3d74766e801-kube-api-access-tkldf\") pod \"iptables-alerter-zvw7s\" (UID: \"11a0b508-2f9c-4be6-b5c7-b3d74766e801\") " pod="openshift-network-operator/iptables-alerter-zvw7s" Oct 30 12:51:14.401029 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397748 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee23e57d-7b77-4dba-95a5-84ad59133981-cni-binary-copy\") pod \"multus-additional-cni-plugins-75bqd\" (UID: \"ee23e57d-7b77-4dba-95a5-84ad59133981\") " pod="openshift-multus/multus-additional-cni-plugins-75bqd" Oct 30 12:51:14.401029 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397771 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ee23e57d-7b77-4dba-95a5-84ad59133981-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-75bqd\" (UID: \"ee23e57d-7b77-4dba-95a5-84ad59133981\") " pod="openshift-multus/multus-additional-cni-plugins-75bqd" Oct 30 12:51:14.401029 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397793 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-etc-modprobe-d\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:51:14.401029 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397813 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-host\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:51:14.401029 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397837 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-systemd-units\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.401029 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397860 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-etc-sysctl-d\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:51:14.401029 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397881 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bea99b12-e97c-4c0e-9360-07e542a4294d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dsh66\" (UID: \"bea99b12-e97c-4c0e-9360-07e542a4294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" Oct 30 12:51:14.401029 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397904 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fe19a15d-5112-4b76-a226-4b8462e2fb33-hosts-file\") pod \"node-resolver-9szkr\" (UID: \"fe19a15d-5112-4b76-a226-4b8462e2fb33\") " pod="openshift-dns/node-resolver-9szkr" Oct 30 12:51:14.401029 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397963 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d9b03988-7080-4352-ba1e-08ee68ba3df8-ovnkube-config\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.401029 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397986 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fe19a15d-5112-4b76-a226-4b8462e2fb33-hosts-file\") pod \"node-resolver-9szkr\" (UID: \"fe19a15d-5112-4b76-a226-4b8462e2fb33\") " pod="openshift-dns/node-resolver-9szkr" Oct 30 12:51:14.401029 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.397984 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/11a0b508-2f9c-4be6-b5c7-b3d74766e801-iptables-alerter-script\") pod \"iptables-alerter-zvw7s\" (UID: \"11a0b508-2f9c-4be6-b5c7-b3d74766e801\") " pod="openshift-network-operator/iptables-alerter-zvw7s" Oct 30 12:51:14.401029 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.398030 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/66172ef8-aaad-4875-b3b3-bd81b9e9ba59-host\") pod \"node-ca-bcbg8\" (UID: \"66172ef8-aaad-4875-b3b3-bd81b9e9ba59\") " pod="openshift-image-registry/node-ca-bcbg8" Oct 30 12:51:14.401029 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.398061 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-host\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:51:14.401029 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.398102 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-etc-sysctl-d\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:51:14.401029 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.398104 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-etc-modprobe-d\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:51:14.401029 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.398154 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bea99b12-e97c-4c0e-9360-07e542a4294d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dsh66\" (UID: \"bea99b12-e97c-4c0e-9360-07e542a4294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" Oct 30 12:51:14.401029 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.398193 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-multus-conf-dir\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.401029 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.398230 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-systemd-units\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.401788 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.398241 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-etc-kubernetes\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.401788 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.398541 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee23e57d-7b77-4dba-95a5-84ad59133981-cni-binary-copy\") pod \"multus-additional-cni-plugins-75bqd\" (UID: \"ee23e57d-7b77-4dba-95a5-84ad59133981\") " pod="openshift-multus/multus-additional-cni-plugins-75bqd" Oct 30 12:51:14.401788 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.398599 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ee23e57d-7b77-4dba-95a5-84ad59133981-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-75bqd\" (UID: \"ee23e57d-7b77-4dba-95a5-84ad59133981\") " pod="openshift-multus/multus-additional-cni-plugins-75bqd" Oct 30 12:51:14.401788 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.400671 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/accde534-15e4-42cf-94df-c0aa085f0603-tmp\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:51:14.401788 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.400969 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3f38af1e-cd92-4cd8-a643-151b10b8da12-agent-certs\") pod \"konnectivity-agent-2nqhp\" (UID: \"3f38af1e-cd92-4cd8-a643-151b10b8da12\") " pod="kube-system/konnectivity-agent-2nqhp" Oct 30 12:51:14.401788 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.401421 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/accde534-15e4-42cf-94df-c0aa085f0603-etc-tuned\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:51:14.402121 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.402098 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d9b03988-7080-4352-ba1e-08ee68ba3df8-ovn-node-metrics-cert\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.429829 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.429764 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9x52\" (UniqueName: \"kubernetes.io/projected/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-kube-api-access-x9x52\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.430132 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.430112 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwddx\" (UniqueName: \"kubernetes.io/projected/ee23e57d-7b77-4dba-95a5-84ad59133981-kube-api-access-hwddx\") pod \"multus-additional-cni-plugins-75bqd\" (UID: \"ee23e57d-7b77-4dba-95a5-84ad59133981\") " pod="openshift-multus/multus-additional-cni-plugins-75bqd" Oct 30 12:51:14.431526 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.431503 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvxmx\" (UniqueName: \"kubernetes.io/projected/66172ef8-aaad-4875-b3b3-bd81b9e9ba59-kube-api-access-wvxmx\") pod \"node-ca-bcbg8\" (UID: \"66172ef8-aaad-4875-b3b3-bd81b9e9ba59\") " pod="openshift-image-registry/node-ca-bcbg8" Oct 30 12:51:14.435319 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:14.435296 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 12:51:14.435319 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:14.435326 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 12:51:14.435489 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:14.435337 2582 projected.go:196] Error preparing data for projected volume kube-api-access-dqvdp for pod openshift-network-diagnostics/network-check-target-95n9l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:51:14.435489 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:14.435386 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bb48bfa-827b-4179-b9fa-418fc14f0f44-kube-api-access-dqvdp podName:1bb48bfa-827b-4179-b9fa-418fc14f0f44 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:14.935372839 +0000 UTC m=+3.110843719 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dqvdp" (UniqueName: "kubernetes.io/projected/1bb48bfa-827b-4179-b9fa-418fc14f0f44-kube-api-access-dqvdp") pod "network-check-target-95n9l" (UID: "1bb48bfa-827b-4179-b9fa-418fc14f0f44") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:51:14.438467 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.438441 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wksmn\" (UniqueName: \"kubernetes.io/projected/accde534-15e4-42cf-94df-c0aa085f0603-kube-api-access-wksmn\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:51:14.438572 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.438451 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxwlx\" (UniqueName: \"kubernetes.io/projected/1fcccc7b-a83c-4c52-a648-cb549e70b54d-kube-api-access-xxwlx\") pod \"network-metrics-daemon-tpz8b\" (UID: \"1fcccc7b-a83c-4c52-a648-cb549e70b54d\") " pod="openshift-multus/network-metrics-daemon-tpz8b" Oct 30 12:51:14.438779 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.438754 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8fhd\" (UniqueName: \"kubernetes.io/projected/bea99b12-e97c-4c0e-9360-07e542a4294d-kube-api-access-m8fhd\") pod \"aws-ebs-csi-driver-node-dsh66\" (UID: \"bea99b12-e97c-4c0e-9360-07e542a4294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" Oct 30 12:51:14.439839 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.439815 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpgrm\" (UniqueName: \"kubernetes.io/projected/fe19a15d-5112-4b76-a226-4b8462e2fb33-kube-api-access-qpgrm\") pod \"node-resolver-9szkr\" (UID: \"fe19a15d-5112-4b76-a226-4b8462e2fb33\") " pod="openshift-dns/node-resolver-9szkr" Oct 30 12:51:14.440398 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.440374 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp4zz\" (UniqueName: \"kubernetes.io/projected/d9b03988-7080-4352-ba1e-08ee68ba3df8-kube-api-access-qp4zz\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.440755 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.440738 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkldf\" (UniqueName: \"kubernetes.io/projected/11a0b508-2f9c-4be6-b5c7-b3d74766e801-kube-api-access-tkldf\") pod \"iptables-alerter-zvw7s\" (UID: \"11a0b508-2f9c-4be6-b5c7-b3d74766e801\") " pod="openshift-network-operator/iptables-alerter-zvw7s" Oct 30 12:51:14.522251 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.522217 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sw8rj" Oct 30 12:51:14.533427 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.533401 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zvw7s" Oct 30 12:51:14.534065 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:14.534027 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddad46b5c_e3e3_4efa_b8f1_b13de224ef8c.slice/crio-ae4a53261c8f2d08d8af9130166c20d999ab7bf0ed529f7d154cbc0ae514e6ed WatchSource:0}: Error finding container ae4a53261c8f2d08d8af9130166c20d999ab7bf0ed529f7d154cbc0ae514e6ed: Status 404 returned error can't find the container with id ae4a53261c8f2d08d8af9130166c20d999ab7bf0ed529f7d154cbc0ae514e6ed Oct 30 12:51:14.544191 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:14.544161 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11a0b508_2f9c_4be6_b5c7_b3d74766e801.slice/crio-1697328fc8c5672d44ef75b4e614af0feacc2e137b5e29cc32a0554ac92829eb WatchSource:0}: Error finding container 1697328fc8c5672d44ef75b4e614af0feacc2e137b5e29cc32a0554ac92829eb: Status 404 returned error can't find the container with id 1697328fc8c5672d44ef75b4e614af0feacc2e137b5e29cc32a0554ac92829eb Oct 30 12:51:14.545002 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.544978 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:51:14.550113 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.550086 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bcbg8" Oct 30 12:51:14.559220 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:14.559197 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaccde534_15e4_42cf_94df_c0aa085f0603.slice/crio-4e9d163d1c47968f39eb321929aee25faba0731ed3f680879663c1a2a60decff WatchSource:0}: Error finding container 4e9d163d1c47968f39eb321929aee25faba0731ed3f680879663c1a2a60decff: Status 404 returned error can't find the container with id 4e9d163d1c47968f39eb321929aee25faba0731ed3f680879663c1a2a60decff Oct 30 12:51:14.562598 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:14.562572 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66172ef8_aaad_4875_b3b3_bd81b9e9ba59.slice/crio-b72d684ebf8e4040794f375efa08178308d583211935ee32ab65d99a3a74d8e6 WatchSource:0}: Error finding container b72d684ebf8e4040794f375efa08178308d583211935ee32ab65d99a3a74d8e6: Status 404 returned error can't find the container with id b72d684ebf8e4040794f375efa08178308d583211935ee32ab65d99a3a74d8e6 Oct 30 12:51:14.563739 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.563720 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" Oct 30 12:51:14.576997 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:14.576925 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbea99b12_e97c_4c0e_9360_07e542a4294d.slice/crio-076576ce8f705e376e3e20d5ba028afd9046651ed42cefcc0e1400fc95ef000a WatchSource:0}: Error finding container 076576ce8f705e376e3e20d5ba028afd9046651ed42cefcc0e1400fc95ef000a: Status 404 returned error can't find the container with id 076576ce8f705e376e3e20d5ba028afd9046651ed42cefcc0e1400fc95ef000a Oct 30 12:51:14.581258 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.581236 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9szkr" Oct 30 12:51:14.588483 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.588457 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-75bqd" Oct 30 12:51:14.592104 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:14.592082 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe19a15d_5112_4b76_a226_4b8462e2fb33.slice/crio-2d03adb62aeda4a8cc61bb3dca86865ed63bd63ff6e3b7d346b3c6d438fcd903 WatchSource:0}: Error finding container 2d03adb62aeda4a8cc61bb3dca86865ed63bd63ff6e3b7d346b3c6d438fcd903: Status 404 returned error can't find the container with id 2d03adb62aeda4a8cc61bb3dca86865ed63bd63ff6e3b7d346b3c6d438fcd903 Oct 30 12:51:14.594916 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.594880 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2nqhp" Oct 30 12:51:14.598307 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:14.598285 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee23e57d_7b77_4dba_95a5_84ad59133981.slice/crio-6ff02a1f79b7714a3c55113244bdf4c133dd72421d6929105f6298f05ccb7636 WatchSource:0}: Error finding container 6ff02a1f79b7714a3c55113244bdf4c133dd72421d6929105f6298f05ccb7636: Status 404 returned error can't find the container with id 6ff02a1f79b7714a3c55113244bdf4c133dd72421d6929105f6298f05ccb7636 Oct 30 12:51:14.605980 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.605954 2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:51:14.606452 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:14.606429 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f38af1e_cd92_4cd8_a643_151b10b8da12.slice/crio-52495134362f8b43db53d257786d659819f2329d5a13f341d1b9c0cc4a20619b WatchSource:0}: Error finding container 52495134362f8b43db53d257786d659819f2329d5a13f341d1b9c0cc4a20619b: Status 404 returned error can't find the container with id 52495134362f8b43db53d257786d659819f2329d5a13f341d1b9c0cc4a20619b Oct 30 12:51:14.610207 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.610183 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:14.622085 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:14.622056 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9b03988_7080_4352_ba1e_08ee68ba3df8.slice/crio-8b99f37e6db72b8da01b867c94cb74058eb342a3d1d037fbcbd4dcf00672bdbf WatchSource:0}: Error finding container 8b99f37e6db72b8da01b867c94cb74058eb342a3d1d037fbcbd4dcf00672bdbf: Status 404 returned error can't find the container with id 8b99f37e6db72b8da01b867c94cb74058eb342a3d1d037fbcbd4dcf00672bdbf Oct 30 12:51:14.901931 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.901902 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50bec85b-5592-437d-aa6f-4899aeb494a5-original-pull-secret\") pod \"global-pull-secret-syncer-s7fsp\" (UID: \"50bec85b-5592-437d-aa6f-4899aeb494a5\") " pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:51:14.902047 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:14.901956 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fcccc7b-a83c-4c52-a648-cb549e70b54d-metrics-certs\") pod \"network-metrics-daemon-tpz8b\" (UID: \"1fcccc7b-a83c-4c52-a648-cb549e70b54d\") " pod="openshift-multus/network-metrics-daemon-tpz8b" Oct 30 12:51:14.902120 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:14.902104 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:51:14.902172 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:14.902158 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fcccc7b-a83c-4c52-a648-cb549e70b54d-metrics-certs podName:1fcccc7b-a83c-4c52-a648-cb549e70b54d nodeName:}" failed. No retries permitted until 2026-10-30 12:51:15.902141358 +0000 UTC m=+4.077612239 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1fcccc7b-a83c-4c52-a648-cb549e70b54d-metrics-certs") pod "network-metrics-daemon-tpz8b" (UID: "1fcccc7b-a83c-4c52-a648-cb549e70b54d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:51:14.902239 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:14.902231 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 12:51:14.902298 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:14.902266 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50bec85b-5592-437d-aa6f-4899aeb494a5-original-pull-secret podName:50bec85b-5592-437d-aa6f-4899aeb494a5 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:15.902254197 +0000 UTC m=+4.077725080 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/50bec85b-5592-437d-aa6f-4899aeb494a5-original-pull-secret") pod "global-pull-secret-syncer-s7fsp" (UID: "50bec85b-5592-437d-aa6f-4899aeb494a5") : object "kube-system"/"original-pull-secret" not registered Oct 30 12:51:15.003062 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:15.003031 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqvdp\" (UniqueName: \"kubernetes.io/projected/1bb48bfa-827b-4179-b9fa-418fc14f0f44-kube-api-access-dqvdp\") pod \"network-check-target-95n9l\" (UID: \"1bb48bfa-827b-4179-b9fa-418fc14f0f44\") " pod="openshift-network-diagnostics/network-check-target-95n9l" Oct 30 12:51:15.003213 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:15.003197 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 12:51:15.003276 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:15.003221 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 12:51:15.003276 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:15.003234 2582 projected.go:196] Error preparing data for projected volume kube-api-access-dqvdp for pod openshift-network-diagnostics/network-check-target-95n9l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:51:15.003388 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:15.003299 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bb48bfa-827b-4179-b9fa-418fc14f0f44-kube-api-access-dqvdp podName:1bb48bfa-827b-4179-b9fa-418fc14f0f44 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:16.003279679 +0000 UTC m=+4.178750559 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dqvdp" (UniqueName: "kubernetes.io/projected/1bb48bfa-827b-4179-b9fa-418fc14f0f44-kube-api-access-dqvdp") pod "network-check-target-95n9l" (UID: "1bb48bfa-827b-4179-b9fa-418fc14f0f44") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:51:15.319353 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:15.319130 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zvw7s" event={"ID":"11a0b508-2f9c-4be6-b5c7-b3d74766e801","Type":"ContainerStarted","Data":"1697328fc8c5672d44ef75b4e614af0feacc2e137b5e29cc32a0554ac92829eb"} Oct 30 12:51:15.325426 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:15.325286 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sw8rj" event={"ID":"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c","Type":"ContainerStarted","Data":"ae4a53261c8f2d08d8af9130166c20d999ab7bf0ed529f7d154cbc0ae514e6ed"} Oct 30 12:51:15.339477 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:15.339447 2582 generic.go:356] "Generic (PLEG): container finished" podID="c7d15293c60c5848ccd82e7130b45838" containerID="3cdd4b7edc10db5ddf3920923ece442e41dbb7163b81a835bf9d485f1fbd0f0c" exitCode=0 Oct 30 12:51:15.339590 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:15.339534 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-21.ec2.internal" event={"ID":"c7d15293c60c5848ccd82e7130b45838","Type":"ContainerDied","Data":"3cdd4b7edc10db5ddf3920923ece442e41dbb7163b81a835bf9d485f1fbd0f0c"} Oct 30 12:51:15.347287 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:15.346906 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:46:14 +0000 UTC" deadline="2028-08-14 23:11:56.954503694 +0000 UTC" Oct 30 12:51:15.347287 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:15.346965 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15706h20m41.607543201s" Oct 30 12:51:15.348743 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:15.348717 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-75bqd" event={"ID":"ee23e57d-7b77-4dba-95a5-84ad59133981","Type":"ContainerStarted","Data":"6ff02a1f79b7714a3c55113244bdf4c133dd72421d6929105f6298f05ccb7636"} Oct 30 12:51:15.353439 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:15.353392 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bcbg8" event={"ID":"66172ef8-aaad-4875-b3b3-bd81b9e9ba59","Type":"ContainerStarted","Data":"b72d684ebf8e4040794f375efa08178308d583211935ee32ab65d99a3a74d8e6"} Oct 30 12:51:15.374456 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:15.374424 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k76bj" event={"ID":"accde534-15e4-42cf-94df-c0aa085f0603","Type":"ContainerStarted","Data":"4e9d163d1c47968f39eb321929aee25faba0731ed3f680879663c1a2a60decff"} Oct 30 12:51:15.383722 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:15.383692 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-21.ec2.internal" event={"ID":"100438b9fc00ed25c10ae1227a1de159","Type":"ContainerStarted","Data":"32809343fc4d53e65653c04b2a4bdb750b6595098b05ab4b1020bb6d4cfc7deb"} Oct 30 12:51:15.388116 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:15.388031 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" event={"ID":"d9b03988-7080-4352-ba1e-08ee68ba3df8","Type":"ContainerStarted","Data":"8b99f37e6db72b8da01b867c94cb74058eb342a3d1d037fbcbd4dcf00672bdbf"} Oct 30 12:51:15.396322 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:15.396042 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2nqhp" event={"ID":"3f38af1e-cd92-4cd8-a643-151b10b8da12","Type":"ContainerStarted","Data":"52495134362f8b43db53d257786d659819f2329d5a13f341d1b9c0cc4a20619b"} Oct 30 12:51:15.400955 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:15.400876 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9szkr" event={"ID":"fe19a15d-5112-4b76-a226-4b8462e2fb33","Type":"ContainerStarted","Data":"2d03adb62aeda4a8cc61bb3dca86865ed63bd63ff6e3b7d346b3c6d438fcd903"} Oct 30 12:51:15.418010 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:15.417159 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" event={"ID":"bea99b12-e97c-4c0e-9360-07e542a4294d","Type":"ContainerStarted","Data":"076576ce8f705e376e3e20d5ba028afd9046651ed42cefcc0e1400fc95ef000a"} Oct 30 12:51:15.443718 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:15.443631 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-21.ec2.internal" podStartSLOduration=1.44361514 podStartE2EDuration="1.44361514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:51:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:15.44280638 +0000 UTC m=+3.618277284" watchObservedRunningTime="2026-10-30 12:51:15.44361514 +0000 UTC m=+3.619086047" Oct 30 12:51:15.911283 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:15.911170 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50bec85b-5592-437d-aa6f-4899aeb494a5-original-pull-secret\") pod \"global-pull-secret-syncer-s7fsp\" (UID: \"50bec85b-5592-437d-aa6f-4899aeb494a5\") " pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:51:15.911283 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:15.911215 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fcccc7b-a83c-4c52-a648-cb549e70b54d-metrics-certs\") pod \"network-metrics-daemon-tpz8b\" (UID: \"1fcccc7b-a83c-4c52-a648-cb549e70b54d\") " pod="openshift-multus/network-metrics-daemon-tpz8b" Oct 30 12:51:15.911529 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:15.911336 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:51:15.911529 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:15.911400 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fcccc7b-a83c-4c52-a648-cb549e70b54d-metrics-certs podName:1fcccc7b-a83c-4c52-a648-cb549e70b54d nodeName:}" failed. No retries permitted until 2026-10-30 12:51:17.911384698 +0000 UTC m=+6.086855598 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1fcccc7b-a83c-4c52-a648-cb549e70b54d-metrics-certs") pod "network-metrics-daemon-tpz8b" (UID: "1fcccc7b-a83c-4c52-a648-cb549e70b54d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:51:15.911800 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:15.911336 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 12:51:15.911800 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:15.911772 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50bec85b-5592-437d-aa6f-4899aeb494a5-original-pull-secret podName:50bec85b-5592-437d-aa6f-4899aeb494a5 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:17.911746534 +0000 UTC m=+6.087217429 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/50bec85b-5592-437d-aa6f-4899aeb494a5-original-pull-secret") pod "global-pull-secret-syncer-s7fsp" (UID: "50bec85b-5592-437d-aa6f-4899aeb494a5") : object "kube-system"/"original-pull-secret" not registered Oct 30 12:51:16.013090 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:16.012459 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqvdp\" (UniqueName: \"kubernetes.io/projected/1bb48bfa-827b-4179-b9fa-418fc14f0f44-kube-api-access-dqvdp\") pod \"network-check-target-95n9l\" (UID: \"1bb48bfa-827b-4179-b9fa-418fc14f0f44\") " pod="openshift-network-diagnostics/network-check-target-95n9l" Oct 30 12:51:16.013090 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:16.012608 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 12:51:16.013090 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:16.012628 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 12:51:16.013090 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:16.012641 2582 projected.go:196] Error preparing data for projected volume kube-api-access-dqvdp for pod openshift-network-diagnostics/network-check-target-95n9l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:51:16.013090 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:16.012720 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bb48bfa-827b-4179-b9fa-418fc14f0f44-kube-api-access-dqvdp podName:1bb48bfa-827b-4179-b9fa-418fc14f0f44 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:18.012700802 +0000 UTC m=+6.188171701 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dqvdp" (UniqueName: "kubernetes.io/projected/1bb48bfa-827b-4179-b9fa-418fc14f0f44-kube-api-access-dqvdp") pod "network-check-target-95n9l" (UID: "1bb48bfa-827b-4179-b9fa-418fc14f0f44") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:51:16.293817 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:16.293679 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95n9l" Oct 30 12:51:16.294000 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:16.293816 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95n9l" podUID="1bb48bfa-827b-4179-b9fa-418fc14f0f44" Oct 30 12:51:16.294469 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:16.294155 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:51:16.294469 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:16.294277 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7fsp" podUID="50bec85b-5592-437d-aa6f-4899aeb494a5" Oct 30 12:51:16.294469 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:16.294305 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpz8b" Oct 30 12:51:16.294469 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:16.294391 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpz8b" podUID="1fcccc7b-a83c-4c52-a648-cb549e70b54d" Oct 30 12:51:16.347666 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:16.347543 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:46:14 +0000 UTC" deadline="2028-07-11 20:28:50.38958599 +0000 UTC" Oct 30 12:51:16.347666 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:16.347599 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14887h37m34.041990773s" Oct 30 12:51:16.442119 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:16.442083 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-21.ec2.internal" event={"ID":"c7d15293c60c5848ccd82e7130b45838","Type":"ContainerStarted","Data":"710b176a6be7fb03f6faa77f6d2b4d3eb0b08e2a53531865a612ade2f6b8631a"} Oct 30 12:51:17.453802 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:17.453767 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-75bqd" event={"ID":"ee23e57d-7b77-4dba-95a5-84ad59133981","Type":"ContainerStarted","Data":"e6077a088ebe4035b199c3a7d23af9d47cf57a32ab3c7a37135ca5a33196bd00"} Oct 30 12:51:17.538191 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:17.538128 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-21.ec2.internal" podStartSLOduration=3.538105587 podStartE2EDuration="3.538105587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:51:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:16.490028155 +0000 UTC m=+4.665499061" watchObservedRunningTime="2026-10-30 12:51:17.538105587 +0000 UTC m=+5.713576493" Oct 30 12:51:17.928226 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:17.928190 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50bec85b-5592-437d-aa6f-4899aeb494a5-original-pull-secret\") pod \"global-pull-secret-syncer-s7fsp\" (UID: \"50bec85b-5592-437d-aa6f-4899aeb494a5\") " pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:51:17.928390 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:17.928247 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fcccc7b-a83c-4c52-a648-cb549e70b54d-metrics-certs\") pod \"network-metrics-daemon-tpz8b\" (UID: \"1fcccc7b-a83c-4c52-a648-cb549e70b54d\") " pod="openshift-multus/network-metrics-daemon-tpz8b" Oct 30 12:51:17.928390 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:17.928376 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:51:17.928523 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:17.928436 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fcccc7b-a83c-4c52-a648-cb549e70b54d-metrics-certs podName:1fcccc7b-a83c-4c52-a648-cb549e70b54d nodeName:}" failed. No retries permitted until 2026-10-30 12:51:21.928417934 +0000 UTC m=+10.103888820 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1fcccc7b-a83c-4c52-a648-cb549e70b54d-metrics-certs") pod "network-metrics-daemon-tpz8b" (UID: "1fcccc7b-a83c-4c52-a648-cb549e70b54d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:51:17.928872 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:17.928852 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 12:51:17.928955 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:17.928908 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50bec85b-5592-437d-aa6f-4899aeb494a5-original-pull-secret podName:50bec85b-5592-437d-aa6f-4899aeb494a5 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:21.928893168 +0000 UTC m=+10.104364063 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/50bec85b-5592-437d-aa6f-4899aeb494a5-original-pull-secret") pod "global-pull-secret-syncer-s7fsp" (UID: "50bec85b-5592-437d-aa6f-4899aeb494a5") : object "kube-system"/"original-pull-secret" not registered Oct 30 12:51:18.029150 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:18.029116 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqvdp\" (UniqueName: \"kubernetes.io/projected/1bb48bfa-827b-4179-b9fa-418fc14f0f44-kube-api-access-dqvdp\") pod \"network-check-target-95n9l\" (UID: \"1bb48bfa-827b-4179-b9fa-418fc14f0f44\") " pod="openshift-network-diagnostics/network-check-target-95n9l" Oct 30 12:51:18.029329 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:18.029282 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 12:51:18.029329 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:18.029302 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 12:51:18.029329 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:18.029316 2582 projected.go:196] Error preparing data for projected volume kube-api-access-dqvdp for pod openshift-network-diagnostics/network-check-target-95n9l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:51:18.029499 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:18.029378 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bb48bfa-827b-4179-b9fa-418fc14f0f44-kube-api-access-dqvdp podName:1bb48bfa-827b-4179-b9fa-418fc14f0f44 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:22.029358824 +0000 UTC m=+10.204829709 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dqvdp" (UniqueName: "kubernetes.io/projected/1bb48bfa-827b-4179-b9fa-418fc14f0f44-kube-api-access-dqvdp") pod "network-check-target-95n9l" (UID: "1bb48bfa-827b-4179-b9fa-418fc14f0f44") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:51:18.294599 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:18.294527 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95n9l" Oct 30 12:51:18.294765 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:18.294670 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95n9l" podUID="1bb48bfa-827b-4179-b9fa-418fc14f0f44" Oct 30 12:51:18.295043 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:18.295024 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:51:18.295137 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:18.295120 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7fsp" podUID="50bec85b-5592-437d-aa6f-4899aeb494a5" Oct 30 12:51:18.295193 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:18.295179 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpz8b" Oct 30 12:51:18.295465 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:18.295261 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpz8b" podUID="1fcccc7b-a83c-4c52-a648-cb549e70b54d" Oct 30 12:51:18.459227 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:18.458410 2582 generic.go:356] "Generic (PLEG): container finished" podID="ee23e57d-7b77-4dba-95a5-84ad59133981" containerID="e6077a088ebe4035b199c3a7d23af9d47cf57a32ab3c7a37135ca5a33196bd00" exitCode=0 Oct 30 12:51:18.459227 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:18.458465 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-75bqd" event={"ID":"ee23e57d-7b77-4dba-95a5-84ad59133981","Type":"ContainerDied","Data":"e6077a088ebe4035b199c3a7d23af9d47cf57a32ab3c7a37135ca5a33196bd00"} Oct 30 12:51:19.464335 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:19.464297 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2nqhp" event={"ID":"3f38af1e-cd92-4cd8-a643-151b10b8da12","Type":"ContainerStarted","Data":"2113b5dbe9bb76c2624d4b43f3145324d35fa2b91a55d4eb0bb3d43e89c90f86"} Oct 30 12:51:19.509160 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:19.509110 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2nqhp" podStartSLOduration=3.52292814 podStartE2EDuration="7.509093324s" totalImagesPullingTime="3.986165184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:19.508519798 +0000 UTC m=+7.683990721" watchObservedRunningTime="2026-10-30 12:51:19.509093324 +0000 UTC m=+7.684564226" Oct 30 12:51:20.294531 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:20.294500 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpz8b" Oct 30 12:51:20.294675 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:20.294652 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpz8b" podUID="1fcccc7b-a83c-4c52-a648-cb549e70b54d" Oct 30 12:51:20.295040 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:20.295017 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95n9l" Oct 30 12:51:20.295197 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:20.295115 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95n9l" podUID="1bb48bfa-827b-4179-b9fa-418fc14f0f44" Oct 30 12:51:20.295197 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:20.295174 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:51:20.295299 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:20.295242 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7fsp" podUID="50bec85b-5592-437d-aa6f-4899aeb494a5" Oct 30 12:51:20.469141 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:20.469069 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bcbg8" event={"ID":"66172ef8-aaad-4875-b3b3-bd81b9e9ba59","Type":"ContainerStarted","Data":"fb73e21ed1fa8fc406d8b76f7264ada7066df9a5d9faad1e50930d0734148caf"} Oct 30 12:51:20.472654 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:20.472573 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" event={"ID":"bea99b12-e97c-4c0e-9360-07e542a4294d","Type":"ContainerStarted","Data":"94f05401ada88b3f0d4c859ddf462ac716fc2d1f3ff24a2a0497e12861ea1fe6"} Oct 30 12:51:20.537712 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:20.537655 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bcbg8" podStartSLOduration=3.761574054 podStartE2EDuration="8.537637565s" totalImagesPullingTime="4.776063511s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:20.53697057 +0000 UTC m=+8.712441472" watchObservedRunningTime="2026-10-30 12:51:20.537637565 +0000 UTC m=+8.713108465" Oct 30 12:51:21.005655 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:21.005352 2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2nqhp" Oct 30 12:51:21.005921 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:21.005901 2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2nqhp" Oct 30 12:51:21.844533 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:21.844506 2582 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 12:51:21.963799 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:21.963771 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50bec85b-5592-437d-aa6f-4899aeb494a5-original-pull-secret\") pod \"global-pull-secret-syncer-s7fsp\" (UID: \"50bec85b-5592-437d-aa6f-4899aeb494a5\") " pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:51:21.964006 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:21.963825 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fcccc7b-a83c-4c52-a648-cb549e70b54d-metrics-certs\") pod \"network-metrics-daemon-tpz8b\" (UID: \"1fcccc7b-a83c-4c52-a648-cb549e70b54d\") " pod="openshift-multus/network-metrics-daemon-tpz8b" Oct 30 12:51:21.964006 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:21.963939 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:51:21.964006 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:21.964002 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fcccc7b-a83c-4c52-a648-cb549e70b54d-metrics-certs podName:1fcccc7b-a83c-4c52-a648-cb549e70b54d nodeName:}" failed. No retries permitted until 2026-10-30 12:51:29.963981523 +0000 UTC m=+18.139452426 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1fcccc7b-a83c-4c52-a648-cb549e70b54d-metrics-certs") pod "network-metrics-daemon-tpz8b" (UID: "1fcccc7b-a83c-4c52-a648-cb549e70b54d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:51:21.964188 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:21.963939 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 12:51:21.964188 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:21.964064 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50bec85b-5592-437d-aa6f-4899aeb494a5-original-pull-secret podName:50bec85b-5592-437d-aa6f-4899aeb494a5 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:29.964052401 +0000 UTC m=+18.139523283 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/50bec85b-5592-437d-aa6f-4899aeb494a5-original-pull-secret") pod "global-pull-secret-syncer-s7fsp" (UID: "50bec85b-5592-437d-aa6f-4899aeb494a5") : object "kube-system"/"original-pull-secret" not registered Oct 30 12:51:22.065765 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:22.065178 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqvdp\" (UniqueName: \"kubernetes.io/projected/1bb48bfa-827b-4179-b9fa-418fc14f0f44-kube-api-access-dqvdp\") pod \"network-check-target-95n9l\" (UID: \"1bb48bfa-827b-4179-b9fa-418fc14f0f44\") " pod="openshift-network-diagnostics/network-check-target-95n9l" Oct 30 12:51:22.065765 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:22.065335 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 12:51:22.065765 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:22.065355 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 12:51:22.065765 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:22.065368 2582 projected.go:196] Error preparing data for projected volume kube-api-access-dqvdp for pod openshift-network-diagnostics/network-check-target-95n9l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:51:22.065765 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:22.065421 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bb48bfa-827b-4179-b9fa-418fc14f0f44-kube-api-access-dqvdp podName:1bb48bfa-827b-4179-b9fa-418fc14f0f44 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:30.065404173 +0000 UTC m=+18.240875066 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dqvdp" (UniqueName: "kubernetes.io/projected/1bb48bfa-827b-4179-b9fa-418fc14f0f44-kube-api-access-dqvdp") pod "network-check-target-95n9l" (UID: "1bb48bfa-827b-4179-b9fa-418fc14f0f44") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:51:22.258386 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:22.258232 2582 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T12:51:21.844527609Z","UUID":"9f87b1a6-c1e4-43a3-953e-d0e77053ff66","Handler":null,"Name":"","Endpoint":""} Oct 30 12:51:22.262198 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:22.262175 2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 12:51:22.262311 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:22.262206 2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 12:51:22.293207 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:22.293142 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95n9l" Oct 30 12:51:22.293335 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:22.293243 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95n9l" podUID="1bb48bfa-827b-4179-b9fa-418fc14f0f44" Oct 30 12:51:22.293431 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:22.293379 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:51:22.293501 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:22.293477 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7fsp" podUID="50bec85b-5592-437d-aa6f-4899aeb494a5" Oct 30 12:51:22.293554 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:22.293542 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpz8b" Oct 30 12:51:22.293698 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:22.293646 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpz8b" podUID="1fcccc7b-a83c-4c52-a648-cb549e70b54d" Oct 30 12:51:22.480880 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:22.480828 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" event={"ID":"bea99b12-e97c-4c0e-9360-07e542a4294d","Type":"ContainerStarted","Data":"9eb7e97485b6a78367fb089c41c2345fbe51ed2cee654f1d06dd725036d5bc84"} Oct 30 12:51:22.484267 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:22.484243 2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 12:51:22.485270 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:22.485213 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sw8rj" event={"ID":"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c","Type":"ContainerStarted","Data":"61525ae398f7343b97d77750d7276cd7a4ccbd0af494d059452428b35666b5f8"} Oct 30 12:51:22.534176 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:22.534067 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-sw8rj" podStartSLOduration=2.755340591 podStartE2EDuration="10.5340466s" totalImagesPullingTime="7.778706009s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:22.532892595 +0000 UTC m=+10.708363499" watchObservedRunningTime="2026-10-30 12:51:22.5340466 +0000 UTC m=+10.709517502" Oct 30 12:51:23.487866 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:23.487600 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9szkr" event={"ID":"fe19a15d-5112-4b76-a226-4b8462e2fb33","Type":"ContainerStarted","Data":"523b5189bc73611cd9e7b8e0e85c87a622fe3084297c42471597ddf5d5e04dcd"} Oct 30 12:51:23.490055 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:23.489999 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" event={"ID":"bea99b12-e97c-4c0e-9360-07e542a4294d","Type":"ContainerStarted","Data":"6f085b2f3cf9943b3f3539b8f2ab01c7b2bcedbb0bf226fd66be71b9535eba25"} Oct 30 12:51:23.491909 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:23.491804 2582 generic.go:356] "Generic (PLEG): container finished" podID="ee23e57d-7b77-4dba-95a5-84ad59133981" containerID="1e586f0383e556ec9d0ab1d50915441e40fa894f7a987b5c1c65d9a52af31cc1" exitCode=0 Oct 30 12:51:23.491909 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:23.491850 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-75bqd" event={"ID":"ee23e57d-7b77-4dba-95a5-84ad59133981","Type":"ContainerDied","Data":"1e586f0383e556ec9d0ab1d50915441e40fa894f7a987b5c1c65d9a52af31cc1"} Oct 30 12:51:23.531652 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:23.531569 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-9szkr" podStartSLOduration=3.530720893 podStartE2EDuration="11.531552814s" totalImagesPullingTime="8.000831921s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:23.530974859 +0000 UTC m=+11.706445759" watchObservedRunningTime="2026-10-30 12:51:23.531552814 +0000 UTC m=+11.707023716" Oct 30 12:51:23.634644 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:23.632838 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" podStartSLOduration=3.7212722830000002 podStartE2EDuration="11.632819688s" totalImagesPullingTime="7.911547405s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:51:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:23.629968486 +0000 UTC m=+11.805439389" watchObservedRunningTime="2026-10-30 12:51:23.632819688 +0000 UTC m=+11.808290590" Oct 30 12:51:24.293543 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:24.293016 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:51:24.293543 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:24.293044 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpz8b" Oct 30 12:51:24.293543 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:24.293017 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95n9l" Oct 30 12:51:24.293543 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:24.293141 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7fsp" podUID="50bec85b-5592-437d-aa6f-4899aeb494a5" Oct 30 12:51:24.293543 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:24.293201 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95n9l" podUID="1bb48bfa-827b-4179-b9fa-418fc14f0f44" Oct 30 12:51:24.293543 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:24.293314 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpz8b" podUID="1fcccc7b-a83c-4c52-a648-cb549e70b54d" Oct 30 12:51:24.495767 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:24.495733 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zvw7s" event={"ID":"11a0b508-2f9c-4be6-b5c7-b3d74766e801","Type":"ContainerStarted","Data":"e720aa6adb6473a09b86b49ab972b6c4b3e568ffec4376baf8475b5fc6dc58b1"} Oct 30 12:51:24.539414 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:24.539363 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zvw7s" podStartSLOduration=4.72123198 podStartE2EDuration="12.539345951s" totalImagesPullingTime="7.818113971s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:24.539025937 +0000 UTC m=+12.714496839" watchObservedRunningTime="2026-10-30 12:51:24.539345951 +0000 UTC m=+12.714816852" Oct 30 12:51:25.499841 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:25.499804 2582 generic.go:356] "Generic (PLEG): container finished" podID="ee23e57d-7b77-4dba-95a5-84ad59133981" containerID="424f941b15e3c0aabdcee253632f615ca93f53d7abb333388daada2accd2a1aa" exitCode=0 Oct 30 12:51:25.500645 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:25.499891 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-75bqd" event={"ID":"ee23e57d-7b77-4dba-95a5-84ad59133981","Type":"ContainerDied","Data":"424f941b15e3c0aabdcee253632f615ca93f53d7abb333388daada2accd2a1aa"} Oct 30 12:51:25.502002 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:25.501658 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k76bj" event={"ID":"accde534-15e4-42cf-94df-c0aa085f0603","Type":"ContainerStarted","Data":"dcc133e7c745bbca6923770662b7a10b40e40afa02ad1f4f39be92d7cd0db196"} Oct 30 12:51:25.624520 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:25.624464 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-k76bj" podStartSLOduration=3.444291513 podStartE2EDuration="13.624446644s" totalImagesPullingTime="10.180155131s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:25.623939972 +0000 UTC m=+13.799410873" watchObservedRunningTime="2026-10-30 12:51:25.624446644 +0000 UTC m=+13.799917545" Oct 30 12:51:26.294778 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:26.294750 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95n9l" Oct 30 12:51:26.294887 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:26.294757 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:51:26.294938 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:26.294878 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95n9l" podUID="1bb48bfa-827b-4179-b9fa-418fc14f0f44" Oct 30 12:51:26.294938 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:26.294757 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpz8b" Oct 30 12:51:26.295028 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:26.294933 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7fsp" podUID="50bec85b-5592-437d-aa6f-4899aeb494a5" Oct 30 12:51:26.295028 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:26.294996 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpz8b" podUID="1fcccc7b-a83c-4c52-a648-cb549e70b54d" Oct 30 12:51:26.507332 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:26.507253 2582 generic.go:356] "Generic (PLEG): container finished" podID="ee23e57d-7b77-4dba-95a5-84ad59133981" containerID="a49e71e67cfd0c016fdeeb9daa5c8a2547728d3527ae2f5b3c6ad715168a895d" exitCode=0 Oct 30 12:51:26.507890 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:26.507335 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-75bqd" event={"ID":"ee23e57d-7b77-4dba-95a5-84ad59133981","Type":"ContainerDied","Data":"a49e71e67cfd0c016fdeeb9daa5c8a2547728d3527ae2f5b3c6ad715168a895d"} Oct 30 12:51:28.292798 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:28.292587 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:51:28.293163 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:28.292598 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95n9l" Oct 30 12:51:28.293163 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:28.292905 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7fsp" podUID="50bec85b-5592-437d-aa6f-4899aeb494a5" Oct 30 12:51:28.293163 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:28.292645 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpz8b" Oct 30 12:51:28.293163 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:28.292944 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95n9l" podUID="1bb48bfa-827b-4179-b9fa-418fc14f0f44" Oct 30 12:51:28.293163 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:28.293007 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpz8b" podUID="1fcccc7b-a83c-4c52-a648-cb549e70b54d" Oct 30 12:51:28.514829 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:28.514802 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-75bqd" event={"ID":"ee23e57d-7b77-4dba-95a5-84ad59133981","Type":"ContainerStarted","Data":"f6a15e7651ec6983a0084122fd46a5921cce20e0f2e5f34e2bf5c48b84f591f8"} Oct 30 12:51:29.519865 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:29.519638 2582 generic.go:356] "Generic (PLEG): container finished" podID="ee23e57d-7b77-4dba-95a5-84ad59133981" containerID="f6a15e7651ec6983a0084122fd46a5921cce20e0f2e5f34e2bf5c48b84f591f8" exitCode=0 Oct 30 12:51:29.520216 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:29.519680 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-75bqd" event={"ID":"ee23e57d-7b77-4dba-95a5-84ad59133981","Type":"ContainerDied","Data":"f6a15e7651ec6983a0084122fd46a5921cce20e0f2e5f34e2bf5c48b84f591f8"} Oct 30 12:51:29.524729 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:29.524709 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" event={"ID":"d9b03988-7080-4352-ba1e-08ee68ba3df8","Type":"ContainerStarted","Data":"ddb289ddaf5af0237b7f38a539c08556f0e9a72c50cfd44ee7880ab9387fed6c"} Oct 30 12:51:29.524784 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:29.524765 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" event={"ID":"d9b03988-7080-4352-ba1e-08ee68ba3df8","Type":"ContainerStarted","Data":"b2f17c1409f3dd3efae90bc55dccae19392dddeb3449cb5e389cfadba9373b08"} Oct 30 12:51:29.524824 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:29.524795 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" event={"ID":"d9b03988-7080-4352-ba1e-08ee68ba3df8","Type":"ContainerStarted","Data":"392db0e9b9b94e9848121198d3ea3e44f7d3736a6d062a4dae6f8df781ec174d"} Oct 30 12:51:29.524824 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:29.524822 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" event={"ID":"d9b03988-7080-4352-ba1e-08ee68ba3df8","Type":"ContainerStarted","Data":"ff406aad351d524e980fa8f20ad045ade11b9ee0ec7d7219981fdef8c7a4ed8f"} Oct 30 12:51:29.524893 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:29.524858 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" event={"ID":"d9b03988-7080-4352-ba1e-08ee68ba3df8","Type":"ContainerStarted","Data":"61132d619596819cf763dccea289ed42c4ed55414c719b6cf19477689a30e790"} Oct 30 12:51:29.524893 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:29.524885 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" event={"ID":"d9b03988-7080-4352-ba1e-08ee68ba3df8","Type":"ContainerStarted","Data":"5c9686fa886236a671d4cd406559d628689bb02b581c03bf284c1c2b0bbe784d"} Oct 30 12:51:30.036199 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:30.036172 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50bec85b-5592-437d-aa6f-4899aeb494a5-original-pull-secret\") pod \"global-pull-secret-syncer-s7fsp\" (UID: \"50bec85b-5592-437d-aa6f-4899aeb494a5\") " pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:51:30.036320 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:30.036205 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fcccc7b-a83c-4c52-a648-cb549e70b54d-metrics-certs\") pod \"network-metrics-daemon-tpz8b\" (UID: \"1fcccc7b-a83c-4c52-a648-cb549e70b54d\") " pod="openshift-multus/network-metrics-daemon-tpz8b" Oct 30 12:51:30.036320 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:30.036292 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:51:30.036320 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:30.036293 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 12:51:30.036414 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:30.036336 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fcccc7b-a83c-4c52-a648-cb549e70b54d-metrics-certs podName:1fcccc7b-a83c-4c52-a648-cb549e70b54d nodeName:}" failed. No retries permitted until 2026-10-30 12:51:46.036322292 +0000 UTC m=+34.211793169 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1fcccc7b-a83c-4c52-a648-cb549e70b54d-metrics-certs") pod "network-metrics-daemon-tpz8b" (UID: "1fcccc7b-a83c-4c52-a648-cb549e70b54d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:51:30.036414 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:30.036347 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50bec85b-5592-437d-aa6f-4899aeb494a5-original-pull-secret podName:50bec85b-5592-437d-aa6f-4899aeb494a5 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:46.036341818 +0000 UTC m=+34.211812696 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/50bec85b-5592-437d-aa6f-4899aeb494a5-original-pull-secret") pod "global-pull-secret-syncer-s7fsp" (UID: "50bec85b-5592-437d-aa6f-4899aeb494a5") : object "kube-system"/"original-pull-secret" not registered Oct 30 12:51:30.136635 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:30.136598 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqvdp\" (UniqueName: \"kubernetes.io/projected/1bb48bfa-827b-4179-b9fa-418fc14f0f44-kube-api-access-dqvdp\") pod \"network-check-target-95n9l\" (UID: \"1bb48bfa-827b-4179-b9fa-418fc14f0f44\") " pod="openshift-network-diagnostics/network-check-target-95n9l" Oct 30 12:51:30.136747 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:30.136728 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 12:51:30.136747 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:30.136743 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 12:51:30.136807 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:30.136750 2582 projected.go:196] Error preparing data for projected volume kube-api-access-dqvdp for pod openshift-network-diagnostics/network-check-target-95n9l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:51:30.136807 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:30.136787 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bb48bfa-827b-4179-b9fa-418fc14f0f44-kube-api-access-dqvdp podName:1bb48bfa-827b-4179-b9fa-418fc14f0f44 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:46.136776743 +0000 UTC m=+34.312247621 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dqvdp" (UniqueName: "kubernetes.io/projected/1bb48bfa-827b-4179-b9fa-418fc14f0f44-kube-api-access-dqvdp") pod "network-check-target-95n9l" (UID: "1bb48bfa-827b-4179-b9fa-418fc14f0f44") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:51:30.292893 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:30.292829 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:51:30.292893 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:30.292862 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95n9l" Oct 30 12:51:30.293052 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:30.292916 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7fsp" podUID="50bec85b-5592-437d-aa6f-4899aeb494a5" Oct 30 12:51:30.293052 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:30.292972 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpz8b" Oct 30 12:51:30.293149 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:30.293071 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpz8b" podUID="1fcccc7b-a83c-4c52-a648-cb549e70b54d" Oct 30 12:51:30.293149 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:30.293137 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95n9l" podUID="1bb48bfa-827b-4179-b9fa-418fc14f0f44" Oct 30 12:51:30.529409 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:30.529382 2582 generic.go:356] "Generic (PLEG): container finished" podID="ee23e57d-7b77-4dba-95a5-84ad59133981" containerID="ea8df7b68cacfe318f960db70cd745cf8fcb6033960ddb98df1df107cf254848" exitCode=0 Oct 30 12:51:30.529936 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:30.529425 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-75bqd" event={"ID":"ee23e57d-7b77-4dba-95a5-84ad59133981","Type":"ContainerDied","Data":"ea8df7b68cacfe318f960db70cd745cf8fcb6033960ddb98df1df107cf254848"} Oct 30 12:51:31.537282 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:31.537249 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-75bqd" event={"ID":"ee23e57d-7b77-4dba-95a5-84ad59133981","Type":"ContainerStarted","Data":"ce106d84bd5e72df1e573fb153a48de4a60b692b2b5cc5b9feca2a8d448181e2"} Oct 30 12:51:31.617444 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:31.617395 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-75bqd" podStartSLOduration=9.074229862 podStartE2EDuration="19.617382184s" totalImagesPullingTime="10.543152322s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:51:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:31.61691208 +0000 UTC m=+19.792382963" watchObservedRunningTime="2026-10-30 12:51:31.617382184 +0000 UTC m=+19.792853084" Oct 30 12:51:32.293368 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:32.293336 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95n9l" Oct 30 12:51:32.293510 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:32.293417 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:51:32.293510 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:32.293466 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpz8b" Oct 30 12:51:32.293510 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:32.293460 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95n9l" podUID="1bb48bfa-827b-4179-b9fa-418fc14f0f44" Oct 30 12:51:32.293637 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:32.293563 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpz8b" podUID="1fcccc7b-a83c-4c52-a648-cb549e70b54d" Oct 30 12:51:32.293718 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:32.293692 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7fsp" podUID="50bec85b-5592-437d-aa6f-4899aeb494a5" Oct 30 12:51:32.542435 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:32.542410 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" event={"ID":"d9b03988-7080-4352-ba1e-08ee68ba3df8","Type":"ContainerStarted","Data":"4b39be54d2e9f115b6dc15f8ab2e1d1086b5250bea943e0928dd38f8076648b3"} Oct 30 12:51:33.373937 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:33.373889 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:33.559275 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:33.559251 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:34.293299 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:34.293126 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpz8b" Oct 30 12:51:34.293429 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:34.293145 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:51:34.293429 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:34.293383 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpz8b" podUID="1fcccc7b-a83c-4c52-a648-cb549e70b54d" Oct 30 12:51:34.293514 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:34.293465 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7fsp" podUID="50bec85b-5592-437d-aa6f-4899aeb494a5" Oct 30 12:51:34.293514 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:34.293159 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95n9l" Oct 30 12:51:34.293585 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:34.293540 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95n9l" podUID="1bb48bfa-827b-4179-b9fa-418fc14f0f44" Oct 30 12:51:34.549340 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:34.549283 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" event={"ID":"d9b03988-7080-4352-ba1e-08ee68ba3df8","Type":"ContainerStarted","Data":"45438bdf20a72bfc0680a6a46a9fdf861ecaf3689c20f451d966b2b6a82d7ad1"} Oct 30 12:51:34.549531 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:34.549515 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:34.549590 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:34.549582 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:34.549661 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:34.549648 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:34.565060 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:34.565042 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:34.567329 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:34.567312 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:51:34.648710 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:34.648662 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" podStartSLOduration=8.064288232 podStartE2EDuration="22.648648423s" totalImagesPullingTime="14.584360191s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:34.648538798 +0000 UTC m=+22.824009704" watchObservedRunningTime="2026-10-30 12:51:34.648648423 +0000 UTC m=+22.824119324" Oct 30 12:51:35.396046 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:35.396018 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:35.548902 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:35.548870 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:35.722529 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:35.722502 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-s7fsp"] Oct 30 12:51:35.722943 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:35.722630 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:51:35.722943 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:35.722751 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7fsp" podUID="50bec85b-5592-437d-aa6f-4899aeb494a5" Oct 30 12:51:35.725851 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:35.725829 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-95n9l"] Oct 30 12:51:35.725945 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:35.725905 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95n9l" Oct 30 12:51:35.725991 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:35.725976 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95n9l" podUID="1bb48bfa-827b-4179-b9fa-418fc14f0f44" Oct 30 12:51:35.726328 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:35.726310 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tpz8b"] Oct 30 12:51:35.726399 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:35.726388 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpz8b" Oct 30 12:51:35.726499 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:35.726480 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpz8b" podUID="1fcccc7b-a83c-4c52-a648-cb549e70b54d" Oct 30 12:51:36.180704 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:36.180632 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:36.311305 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:36.311278 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:36.431393 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:36.431334 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:36.564529 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:36.564502 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:36.774956 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:36.774928 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:36.891821 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:36.891792 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:37.121035 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:37.120966 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:37.243106 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:37.243081 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:37.292747 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:37.292724 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpz8b" Oct 30 12:51:37.292877 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:37.292752 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95n9l" Oct 30 12:51:37.292877 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:37.292844 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpz8b" podUID="1fcccc7b-a83c-4c52-a648-cb549e70b54d" Oct 30 12:51:37.292940 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:37.292898 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:51:37.292996 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:37.292975 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95n9l" podUID="1bb48bfa-827b-4179-b9fa-418fc14f0f44" Oct 30 12:51:37.293065 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:37.293051 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7fsp" podUID="50bec85b-5592-437d-aa6f-4899aeb494a5" Oct 30 12:51:37.574418 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:37.574393 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:37.696552 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:37.696513 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:38.221935 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:38.221908 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:39.188698 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:39.188676 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9szkr_fe19a15d-5112-4b76-a226-4b8462e2fb33/dns-node-resolver/0.log" Oct 30 12:51:39.293029 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:39.293002 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpz8b" Oct 30 12:51:39.293487 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:39.293048 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:51:39.293487 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:39.293046 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95n9l" Oct 30 12:51:39.293487 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:39.293124 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpz8b" podUID="1fcccc7b-a83c-4c52-a648-cb549e70b54d" Oct 30 12:51:39.293487 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:39.293202 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7fsp" podUID="50bec85b-5592-437d-aa6f-4899aeb494a5" Oct 30 12:51:39.293487 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:39.293291 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95n9l" podUID="1bb48bfa-827b-4179-b9fa-418fc14f0f44" Oct 30 12:51:39.988767 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:39.988744 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bcbg8_66172ef8-aaad-4875-b3b3-bd81b9e9ba59/node-ca/0.log" Oct 30 12:51:41.055677 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:41.055643 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:41.169298 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:41.169236 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:41.287831 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:41.287789 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:41.292746 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:41.292730 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpz8b" Oct 30 12:51:41.292844 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:41.292829 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95n9l" Oct 30 12:51:41.292888 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:41.292837 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpz8b" podUID="1fcccc7b-a83c-4c52-a648-cb549e70b54d" Oct 30 12:51:41.292888 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:41.292857 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:51:41.292958 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:41.292917 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95n9l" podUID="1bb48bfa-827b-4179-b9fa-418fc14f0f44" Oct 30 12:51:41.292997 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:41.292969 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7fsp" podUID="50bec85b-5592-437d-aa6f-4899aeb494a5" Oct 30 12:51:41.408347 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:41.408320 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:41.542101 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:41.540053 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:41.675212 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:41.675190 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:41.828635 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:41.828556 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:42.052772 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:42.052740 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:42.286435 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:42.286403 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:42.399488 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:42.399456 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:42.969342 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:42.969277 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:43.165131 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:43.165105 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:43.293224 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:43.293156 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95n9l" Oct 30 12:51:43.293673 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:43.293234 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpz8b" Oct 30 12:51:43.293673 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:43.293258 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:51:43.293673 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:43.293362 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpz8b" podUID="1fcccc7b-a83c-4c52-a648-cb549e70b54d" Oct 30 12:51:43.293673 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:43.293393 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7fsp" podUID="50bec85b-5592-437d-aa6f-4899aeb494a5" Oct 30 12:51:43.293673 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:43.293440 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95n9l" podUID="1bb48bfa-827b-4179-b9fa-418fc14f0f44" Oct 30 12:51:43.368534 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:43.368501 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:43.572171 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:43.572098 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:43.762738 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:43.762711 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:44.148186 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:44.148163 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2nqhp" Oct 30 12:51:44.148333 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:44.148281 2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 12:51:44.148685 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:44.148669 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2nqhp" Oct 30 12:51:44.791774 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:44.791739 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:45.163362 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:45.163289 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:45.292942 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:45.292915 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpz8b" Oct 30 12:51:45.292942 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:45.292941 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:51:45.293134 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:45.293063 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpz8b" podUID="1fcccc7b-a83c-4c52-a648-cb549e70b54d" Oct 30 12:51:45.293134 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:45.293111 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95n9l" Oct 30 12:51:45.293234 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:45.293182 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7fsp" podUID="50bec85b-5592-437d-aa6f-4899aeb494a5" Oct 30 12:51:45.293284 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:45.293232 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95n9l" podUID="1bb48bfa-827b-4179-b9fa-418fc14f0f44" Oct 30 12:51:45.358494 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:45.358469 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:45.557221 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:45.557200 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:45.974420 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:45.974352 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:46.051547 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:46.051524 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50bec85b-5592-437d-aa6f-4899aeb494a5-original-pull-secret\") pod \"global-pull-secret-syncer-s7fsp\" (UID: \"50bec85b-5592-437d-aa6f-4899aeb494a5\") " pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:51:46.051646 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:46.051556 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fcccc7b-a83c-4c52-a648-cb549e70b54d-metrics-certs\") pod \"network-metrics-daemon-tpz8b\" (UID: \"1fcccc7b-a83c-4c52-a648-cb549e70b54d\") " pod="openshift-multus/network-metrics-daemon-tpz8b" Oct 30 12:51:46.051699 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:46.051685 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:51:46.051748 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:46.051738 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fcccc7b-a83c-4c52-a648-cb549e70b54d-metrics-certs podName:1fcccc7b-a83c-4c52-a648-cb549e70b54d nodeName:}" failed. No retries permitted until 2026-10-30 12:52:18.051727151 +0000 UTC m=+66.227198029 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1fcccc7b-a83c-4c52-a648-cb549e70b54d-metrics-certs") pod "network-metrics-daemon-tpz8b" (UID: "1fcccc7b-a83c-4c52-a648-cb549e70b54d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:51:46.051788 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:46.051685 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 12:51:46.051788 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:46.051776 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50bec85b-5592-437d-aa6f-4899aeb494a5-original-pull-secret podName:50bec85b-5592-437d-aa6f-4899aeb494a5 nodeName:}" failed. No retries permitted until 2026-10-30 12:52:18.05176646 +0000 UTC m=+66.227237339 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/50bec85b-5592-437d-aa6f-4899aeb494a5-original-pull-secret") pod "global-pull-secret-syncer-s7fsp" (UID: "50bec85b-5592-437d-aa6f-4899aeb494a5") : object "kube-system"/"original-pull-secret" not registered Oct 30 12:51:46.152534 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:46.152511 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqvdp\" (UniqueName: \"kubernetes.io/projected/1bb48bfa-827b-4179-b9fa-418fc14f0f44-kube-api-access-dqvdp\") pod \"network-check-target-95n9l\" (UID: \"1bb48bfa-827b-4179-b9fa-418fc14f0f44\") " pod="openshift-network-diagnostics/network-check-target-95n9l" Oct 30 12:51:46.152651 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:46.152626 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 12:51:46.152651 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:46.152639 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 12:51:46.152651 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:46.152648 2582 projected.go:196] Error preparing data for projected volume kube-api-access-dqvdp for pod openshift-network-diagnostics/network-check-target-95n9l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:51:46.152745 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:46.152686 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bb48bfa-827b-4179-b9fa-418fc14f0f44-kube-api-access-dqvdp podName:1bb48bfa-827b-4179-b9fa-418fc14f0f44 nodeName:}" failed. No retries permitted until 2026-10-30 12:52:18.152676697 +0000 UTC m=+66.328147575 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dqvdp" (UniqueName: "kubernetes.io/projected/1bb48bfa-827b-4179-b9fa-418fc14f0f44-kube-api-access-dqvdp") pod "network-check-target-95n9l" (UID: "1bb48bfa-827b-4179-b9fa-418fc14f0f44") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:51:46.559789 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:46.559764 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:47.212967 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:47.212940 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:47.292337 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:47.292309 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:51:47.292459 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:47.292341 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpz8b" Oct 30 12:51:47.292459 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:47.292395 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7fsp" podUID="50bec85b-5592-437d-aa6f-4899aeb494a5" Oct 30 12:51:47.292459 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:47.292425 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95n9l" Oct 30 12:51:47.292632 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:47.292505 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95n9l" podUID="1bb48bfa-827b-4179-b9fa-418fc14f0f44" Oct 30 12:51:47.292632 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:47.292595 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpz8b" podUID="1fcccc7b-a83c-4c52-a648-cb549e70b54d" Oct 30 12:51:47.446344 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:47.446323 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:47.561567 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:47.561521 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:47.768281 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:47.768253 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:47.964087 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:47.964064 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:48.501740 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:48.501712 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:48.613952 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:48.613907 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:49.165113 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:49.165081 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:49.293029 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:49.293002 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpz8b" Oct 30 12:51:49.293213 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:49.293092 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95n9l" Oct 30 12:51:49.293213 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:49.293115 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:51:49.293213 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:49.293095 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpz8b" podUID="1fcccc7b-a83c-4c52-a648-cb549e70b54d" Oct 30 12:51:49.293213 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:49.293194 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95n9l" podUID="1bb48bfa-827b-4179-b9fa-418fc14f0f44" Oct 30 12:51:49.293420 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:49.293254 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7fsp" podUID="50bec85b-5592-437d-aa6f-4899aeb494a5" Oct 30 12:51:49.360977 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:49.360951 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:49.469849 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:49.469820 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:49.758518 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:49.758466 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:49.961480 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:49.961451 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:50.558187 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:50.558161 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:51.158223 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:51.158195 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:51.293129 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:51.293102 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpz8b" Oct 30 12:51:51.293129 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:51.293120 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95n9l" Oct 30 12:51:51.293280 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:51.293196 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:51:51.293280 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:51.293206 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95n9l" podUID="1bb48bfa-827b-4179-b9fa-418fc14f0f44" Oct 30 12:51:51.293359 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:51.293279 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpz8b" podUID="1fcccc7b-a83c-4c52-a648-cb549e70b54d" Oct 30 12:51:51.293410 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:51.293364 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7fsp" podUID="50bec85b-5592-437d-aa6f-4899aeb494a5" Oct 30 12:51:51.362858 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:51.362832 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:51.556926 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:51.556899 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:51.757803 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:51.757780 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:51.935117 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:51.935064 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:52.474314 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:52.474283 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:52.583305 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:52.583284 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:52.853904 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:52.853841 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:52.976415 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:52.976391 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:53.292790 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:53.292763 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95n9l" Oct 30 12:51:53.292910 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:53.292875 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:51:53.292910 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:53.292884 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpz8b" Oct 30 12:51:53.292910 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:53.292881 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95n9l" podUID="1bb48bfa-827b-4179-b9fa-418fc14f0f44" Oct 30 12:51:53.293007 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:53.292942 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7fsp" podUID="50bec85b-5592-437d-aa6f-4899aeb494a5" Oct 30 12:51:53.293039 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:53.293025 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpz8b" podUID="1fcccc7b-a83c-4c52-a648-cb549e70b54d" Oct 30 12:51:54.769570 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:54.769539 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:54.884832 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:54.884801 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:54.993458 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:54.993428 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:55.276399 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:55.276369 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:55.292311 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:55.292291 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpz8b" Oct 30 12:51:55.292400 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:55.292385 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95n9l" Oct 30 12:51:55.292436 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:55.292412 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:51:55.292469 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:55.292387 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpz8b" podUID="1fcccc7b-a83c-4c52-a648-cb549e70b54d" Oct 30 12:51:55.292502 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:55.292467 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95n9l" podUID="1bb48bfa-827b-4179-b9fa-418fc14f0f44" Oct 30 12:51:55.292551 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:51:55.292535 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7fsp" podUID="50bec85b-5592-437d-aa6f-4899aeb494a5" Oct 30 12:51:55.399845 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:55.399822 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:56.283487 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:56.283454 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:56.404039 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:56.404011 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:56.603631 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:56.603465 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:56.713659 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:56.713598 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:57.094473 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.094443 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-21.ec2.internal" event="NodeReady" Oct 30 12:51:57.094593 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.094531 2582 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 12:51:57.189004 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.188972 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-854v4"] Oct 30 12:51:57.190848 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.190833 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-v7skv"] Oct 30 12:51:57.190973 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.190958 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-854v4" Oct 30 12:51:57.193690 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.193671 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v7skv" Oct 30 12:51:57.210328 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.210307 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 12:51:57.210525 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.210511 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-bhhxq\"" Oct 30 12:51:57.212377 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.212363 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 12:51:57.215658 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.215635 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 12:51:57.216290 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.216272 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-854v4"] Oct 30 12:51:57.222910 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.222893 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-v7skv"] Oct 30 12:51:57.223353 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.223337 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-cfgs4"] Oct 30 12:51:57.225327 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.225311 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cfgs4" Oct 30 12:51:57.247085 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.247065 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cfgs4"] Oct 30 12:51:57.292977 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.292945 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95n9l" Oct 30 12:51:57.292977 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.292964 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:51:57.293322 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.293230 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpz8b" Oct 30 12:51:57.308937 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.308912 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 12:51:57.309034 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.309021 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ktwvb\"" Oct 30 12:51:57.309094 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.309021 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-45hdr\"" Oct 30 12:51:57.309094 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.309062 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 12:51:57.310044 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.310029 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hszmd\"" Oct 30 12:51:57.310155 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.310141 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-wqgtx\"" Oct 30 12:51:57.311288 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.311273 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 12:51:57.311334 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.311292 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 12:51:57.311334 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.311308 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-68fxr\"" Oct 30 12:51:57.314370 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.314354 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 12:51:57.314430 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.314355 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 12:51:57.314430 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.314407 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 12:51:57.314531 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.314429 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 12:51:57.314531 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.314440 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 12:51:57.314531 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.314511 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 12:51:57.335197 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.335179 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbr4h\" (UniqueName: \"kubernetes.io/projected/ab451161-af25-47e2-860e-f7d5186d4491-kube-api-access-gbr4h\") pod \"ingress-canary-854v4\" (UID: \"ab451161-af25-47e2-860e-f7d5186d4491\") " pod="openshift-ingress-canary/ingress-canary-854v4" Oct 30 12:51:57.335262 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.335203 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/51a542f5-796e-46b1-95bd-31751d50b52b-config-volume\") pod \"dns-default-v7skv\" (UID: \"51a542f5-796e-46b1-95bd-31751d50b52b\") " pod="openshift-dns/dns-default-v7skv" Oct 30 12:51:57.335262 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.335220 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4dca982f-f509-454b-a057-d7fd9b3f34bd-data-volume\") pod \"insights-runtime-extractor-cfgs4\" (UID: \"4dca982f-f509-454b-a057-d7fd9b3f34bd\") " pod="openshift-insights/insights-runtime-extractor-cfgs4" Oct 30 12:51:57.335262 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.335237 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4dca982f-f509-454b-a057-d7fd9b3f34bd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cfgs4\" (UID: \"4dca982f-f509-454b-a057-d7fd9b3f34bd\") " pod="openshift-insights/insights-runtime-extractor-cfgs4" Oct 30 12:51:57.335360 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.335300 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4dca982f-f509-454b-a057-d7fd9b3f34bd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cfgs4\" (UID: \"4dca982f-f509-454b-a057-d7fd9b3f34bd\") " pod="openshift-insights/insights-runtime-extractor-cfgs4" Oct 30 12:51:57.335360 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.335337 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/51a542f5-796e-46b1-95bd-31751d50b52b-metrics-tls\") pod \"dns-default-v7skv\" (UID: \"51a542f5-796e-46b1-95bd-31751d50b52b\") " pod="openshift-dns/dns-default-v7skv" Oct 30 12:51:57.335360 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.335354 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m944\" (UniqueName: \"kubernetes.io/projected/4dca982f-f509-454b-a057-d7fd9b3f34bd-kube-api-access-5m944\") pod \"insights-runtime-extractor-cfgs4\" (UID: \"4dca982f-f509-454b-a057-d7fd9b3f34bd\") " pod="openshift-insights/insights-runtime-extractor-cfgs4" Oct 30 12:51:57.335472 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.335455 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4dca982f-f509-454b-a057-d7fd9b3f34bd-crio-socket\") pod \"insights-runtime-extractor-cfgs4\" (UID: \"4dca982f-f509-454b-a057-d7fd9b3f34bd\") " pod="openshift-insights/insights-runtime-extractor-cfgs4" Oct 30 12:51:57.335526 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.335513 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab451161-af25-47e2-860e-f7d5186d4491-cert\") pod \"ingress-canary-854v4\" (UID: \"ab451161-af25-47e2-860e-f7d5186d4491\") " pod="openshift-ingress-canary/ingress-canary-854v4" Oct 30 12:51:57.335558 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.335540 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvf5p\" (UniqueName: \"kubernetes.io/projected/51a542f5-796e-46b1-95bd-31751d50b52b-kube-api-access-tvf5p\") pod \"dns-default-v7skv\" (UID: \"51a542f5-796e-46b1-95bd-31751d50b52b\") " pod="openshift-dns/dns-default-v7skv" Oct 30 12:51:57.335597 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.335586 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51a542f5-796e-46b1-95bd-31751d50b52b-tmp-dir\") pod \"dns-default-v7skv\" (UID: \"51a542f5-796e-46b1-95bd-31751d50b52b\") " pod="openshift-dns/dns-default-v7skv" Oct 30 12:51:57.436047 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.435983 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbr4h\" (UniqueName: \"kubernetes.io/projected/ab451161-af25-47e2-860e-f7d5186d4491-kube-api-access-gbr4h\") pod \"ingress-canary-854v4\" (UID: \"ab451161-af25-47e2-860e-f7d5186d4491\") " pod="openshift-ingress-canary/ingress-canary-854v4" Oct 30 12:51:57.436047 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.436020 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/51a542f5-796e-46b1-95bd-31751d50b52b-config-volume\") pod \"dns-default-v7skv\" (UID: \"51a542f5-796e-46b1-95bd-31751d50b52b\") " pod="openshift-dns/dns-default-v7skv" Oct 30 12:51:57.436047 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.436035 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4dca982f-f509-454b-a057-d7fd9b3f34bd-data-volume\") pod \"insights-runtime-extractor-cfgs4\" (UID: \"4dca982f-f509-454b-a057-d7fd9b3f34bd\") " pod="openshift-insights/insights-runtime-extractor-cfgs4" Oct 30 12:51:57.436047 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.436050 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4dca982f-f509-454b-a057-d7fd9b3f34bd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cfgs4\" (UID: \"4dca982f-f509-454b-a057-d7fd9b3f34bd\") " pod="openshift-insights/insights-runtime-extractor-cfgs4" Oct 30 12:51:57.436289 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.436067 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4dca982f-f509-454b-a057-d7fd9b3f34bd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cfgs4\" (UID: \"4dca982f-f509-454b-a057-d7fd9b3f34bd\") " pod="openshift-insights/insights-runtime-extractor-cfgs4" Oct 30 12:51:57.436289 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.436083 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/51a542f5-796e-46b1-95bd-31751d50b52b-metrics-tls\") pod \"dns-default-v7skv\" (UID: \"51a542f5-796e-46b1-95bd-31751d50b52b\") " pod="openshift-dns/dns-default-v7skv" Oct 30 12:51:57.436289 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.436106 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m944\" (UniqueName: \"kubernetes.io/projected/4dca982f-f509-454b-a057-d7fd9b3f34bd-kube-api-access-5m944\") pod \"insights-runtime-extractor-cfgs4\" (UID: \"4dca982f-f509-454b-a057-d7fd9b3f34bd\") " pod="openshift-insights/insights-runtime-extractor-cfgs4" Oct 30 12:51:57.436289 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.436136 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4dca982f-f509-454b-a057-d7fd9b3f34bd-crio-socket\") pod \"insights-runtime-extractor-cfgs4\" (UID: \"4dca982f-f509-454b-a057-d7fd9b3f34bd\") " pod="openshift-insights/insights-runtime-extractor-cfgs4" Oct 30 12:51:57.436289 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.436182 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab451161-af25-47e2-860e-f7d5186d4491-cert\") pod \"ingress-canary-854v4\" (UID: \"ab451161-af25-47e2-860e-f7d5186d4491\") " pod="openshift-ingress-canary/ingress-canary-854v4" Oct 30 12:51:57.436507 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.436352 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4dca982f-f509-454b-a057-d7fd9b3f34bd-crio-socket\") pod \"insights-runtime-extractor-cfgs4\" (UID: \"4dca982f-f509-454b-a057-d7fd9b3f34bd\") " pod="openshift-insights/insights-runtime-extractor-cfgs4" Oct 30 12:51:57.436507 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.436395 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvf5p\" (UniqueName: \"kubernetes.io/projected/51a542f5-796e-46b1-95bd-31751d50b52b-kube-api-access-tvf5p\") pod \"dns-default-v7skv\" (UID: \"51a542f5-796e-46b1-95bd-31751d50b52b\") " pod="openshift-dns/dns-default-v7skv" Oct 30 12:51:57.436507 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.436427 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51a542f5-796e-46b1-95bd-31751d50b52b-tmp-dir\") pod \"dns-default-v7skv\" (UID: \"51a542f5-796e-46b1-95bd-31751d50b52b\") " pod="openshift-dns/dns-default-v7skv" Oct 30 12:51:57.436507 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.436442 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4dca982f-f509-454b-a057-d7fd9b3f34bd-data-volume\") pod \"insights-runtime-extractor-cfgs4\" (UID: \"4dca982f-f509-454b-a057-d7fd9b3f34bd\") " pod="openshift-insights/insights-runtime-extractor-cfgs4" Oct 30 12:51:57.436805 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.436757 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/51a542f5-796e-46b1-95bd-31751d50b52b-config-volume\") pod \"dns-default-v7skv\" (UID: \"51a542f5-796e-46b1-95bd-31751d50b52b\") " pod="openshift-dns/dns-default-v7skv" Oct 30 12:51:57.436805 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.436761 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51a542f5-796e-46b1-95bd-31751d50b52b-tmp-dir\") pod \"dns-default-v7skv\" (UID: \"51a542f5-796e-46b1-95bd-31751d50b52b\") " pod="openshift-dns/dns-default-v7skv" Oct 30 12:51:57.437100 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.437078 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4dca982f-f509-454b-a057-d7fd9b3f34bd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cfgs4\" (UID: \"4dca982f-f509-454b-a057-d7fd9b3f34bd\") " pod="openshift-insights/insights-runtime-extractor-cfgs4" Oct 30 12:51:57.439682 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.439662 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab451161-af25-47e2-860e-f7d5186d4491-cert\") pod \"ingress-canary-854v4\" (UID: \"ab451161-af25-47e2-860e-f7d5186d4491\") " pod="openshift-ingress-canary/ingress-canary-854v4" Oct 30 12:51:57.439823 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.439805 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4dca982f-f509-454b-a057-d7fd9b3f34bd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cfgs4\" (UID: \"4dca982f-f509-454b-a057-d7fd9b3f34bd\") " pod="openshift-insights/insights-runtime-extractor-cfgs4" Oct 30 12:51:57.439861 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.439822 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/51a542f5-796e-46b1-95bd-31751d50b52b-metrics-tls\") pod \"dns-default-v7skv\" (UID: \"51a542f5-796e-46b1-95bd-31751d50b52b\") " pod="openshift-dns/dns-default-v7skv" Oct 30 12:51:57.462352 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.462324 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvf5p\" (UniqueName: \"kubernetes.io/projected/51a542f5-796e-46b1-95bd-31751d50b52b-kube-api-access-tvf5p\") pod \"dns-default-v7skv\" (UID: \"51a542f5-796e-46b1-95bd-31751d50b52b\") " pod="openshift-dns/dns-default-v7skv" Oct 30 12:51:57.462539 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.462525 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m944\" (UniqueName: \"kubernetes.io/projected/4dca982f-f509-454b-a057-d7fd9b3f34bd-kube-api-access-5m944\") pod \"insights-runtime-extractor-cfgs4\" (UID: \"4dca982f-f509-454b-a057-d7fd9b3f34bd\") " pod="openshift-insights/insights-runtime-extractor-cfgs4" Oct 30 12:51:57.462578 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.462535 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbr4h\" (UniqueName: \"kubernetes.io/projected/ab451161-af25-47e2-860e-f7d5186d4491-kube-api-access-gbr4h\") pod \"ingress-canary-854v4\" (UID: \"ab451161-af25-47e2-860e-f7d5186d4491\") " pod="openshift-ingress-canary/ingress-canary-854v4" Oct 30 12:51:57.502445 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.502421 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-854v4" Oct 30 12:51:57.507925 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.507909 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v7skv" Oct 30 12:51:57.535651 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.535619 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cfgs4" Oct 30 12:51:57.777590 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.777563 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:57.836948 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.836927 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-854v4"] Oct 30 12:51:57.839725 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.839707 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-v7skv"] Oct 30 12:51:57.844681 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:57.844657 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51a542f5_796e_46b1_95bd_31751d50b52b.slice/crio-6bc842612ded17a701978fb17fb8dace799d5e75552b5c8b0104dd3c213b4857 WatchSource:0}: Error finding container 6bc842612ded17a701978fb17fb8dace799d5e75552b5c8b0104dd3c213b4857: Status 404 returned error can't find the container with id 6bc842612ded17a701978fb17fb8dace799d5e75552b5c8b0104dd3c213b4857 Oct 30 12:51:57.854743 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.854723 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cfgs4"] Oct 30 12:51:57.857378 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:51:57.857356 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dca982f_f509_454b_a057_d7fd9b3f34bd.slice/crio-92483d9ab9d6b9c70a1cf111febac24212e30ce539406891994098559142f86b WatchSource:0}: Error finding container 92483d9ab9d6b9c70a1cf111febac24212e30ce539406891994098559142f86b: Status 404 returned error can't find the container with id 92483d9ab9d6b9c70a1cf111febac24212e30ce539406891994098559142f86b Oct 30 12:51:57.890182 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:57.890163 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:58.598563 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:58.598416 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cfgs4" event={"ID":"4dca982f-f509-454b-a057-d7fd9b3f34bd","Type":"ContainerStarted","Data":"1b34be89db59e4d35831b6c69b2978448a8f84c3305459d53521c993ee87b5ad"} Oct 30 12:51:58.598563 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:58.598483 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cfgs4" event={"ID":"4dca982f-f509-454b-a057-d7fd9b3f34bd","Type":"ContainerStarted","Data":"7666939cb09657791ebd2f5a32def11ffa9459f18661080ebf95083053543267"} Oct 30 12:51:58.598563 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:58.598512 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cfgs4" event={"ID":"4dca982f-f509-454b-a057-d7fd9b3f34bd","Type":"ContainerStarted","Data":"92483d9ab9d6b9c70a1cf111febac24212e30ce539406891994098559142f86b"} Oct 30 12:51:58.600666 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:58.600639 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v7skv" event={"ID":"51a542f5-796e-46b1-95bd-31751d50b52b","Type":"ContainerStarted","Data":"6bc842612ded17a701978fb17fb8dace799d5e75552b5c8b0104dd3c213b4857"} Oct 30 12:51:58.601975 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:58.601949 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-854v4" event={"ID":"ab451161-af25-47e2-860e-f7d5186d4491","Type":"ContainerStarted","Data":"f2508189ec0d29ac883e4c6404a709ef657cdf80440bf5200b0fc427fa4f210d"} Oct 30 12:51:59.967472 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:51:59.967443 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:00.070160 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:00.070130 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-xxb2q"] Oct 30 12:52:00.072218 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:00.072195 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xxb2q" Oct 30 12:52:00.093601 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:00.093563 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:00.107081 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:00.106968 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 12:52:00.107081 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:00.107058 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 12:52:00.107566 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:00.107548 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 12:52:00.107675 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:00.107605 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 12:52:00.107851 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:00.107830 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 12:52:00.108845 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:00.108823 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-xxb2q"] Oct 30 12:52:00.120351 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:00.120333 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-vlkp5\"" Oct 30 12:52:00.205258 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:00.205233 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:00.257499 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:00.257468 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/56928a4d-7923-46e6-97bb-6690514f89a4-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-xxb2q\" (UID: \"56928a4d-7923-46e6-97bb-6690514f89a4\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xxb2q" Oct 30 12:52:00.257639 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:00.257515 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/56928a4d-7923-46e6-97bb-6690514f89a4-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-xxb2q\" (UID: \"56928a4d-7923-46e6-97bb-6690514f89a4\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xxb2q" Oct 30 12:52:00.257639 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:00.257537 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/56928a4d-7923-46e6-97bb-6690514f89a4-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-xxb2q\" (UID: \"56928a4d-7923-46e6-97bb-6690514f89a4\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xxb2q" Oct 30 12:52:00.257747 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:00.257675 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v6nm\" (UniqueName: \"kubernetes.io/projected/56928a4d-7923-46e6-97bb-6690514f89a4-kube-api-access-9v6nm\") pod \"prometheus-operator-9cd89b74c-xxb2q\" (UID: \"56928a4d-7923-46e6-97bb-6690514f89a4\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xxb2q" Oct 30 12:52:00.358646 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:00.358590 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/56928a4d-7923-46e6-97bb-6690514f89a4-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-xxb2q\" (UID: \"56928a4d-7923-46e6-97bb-6690514f89a4\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xxb2q" Oct 30 12:52:00.358796 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:00.358684 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/56928a4d-7923-46e6-97bb-6690514f89a4-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-xxb2q\" (UID: \"56928a4d-7923-46e6-97bb-6690514f89a4\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xxb2q" Oct 30 12:52:00.358796 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:00.358717 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/56928a4d-7923-46e6-97bb-6690514f89a4-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-xxb2q\" (UID: \"56928a4d-7923-46e6-97bb-6690514f89a4\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xxb2q" Oct 30 12:52:00.358796 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:00.358746 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v6nm\" (UniqueName: \"kubernetes.io/projected/56928a4d-7923-46e6-97bb-6690514f89a4-kube-api-access-9v6nm\") pod \"prometheus-operator-9cd89b74c-xxb2q\" (UID: \"56928a4d-7923-46e6-97bb-6690514f89a4\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xxb2q" Oct 30 12:52:00.359781 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:00.359717 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/56928a4d-7923-46e6-97bb-6690514f89a4-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-xxb2q\" (UID: \"56928a4d-7923-46e6-97bb-6690514f89a4\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xxb2q" Oct 30 12:52:00.361521 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:00.361495 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/56928a4d-7923-46e6-97bb-6690514f89a4-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-xxb2q\" (UID: \"56928a4d-7923-46e6-97bb-6690514f89a4\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xxb2q" Oct 30 12:52:00.361606 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:00.361501 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/56928a4d-7923-46e6-97bb-6690514f89a4-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-xxb2q\" (UID: \"56928a4d-7923-46e6-97bb-6690514f89a4\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xxb2q" Oct 30 12:52:00.386747 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:00.386724 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v6nm\" (UniqueName: \"kubernetes.io/projected/56928a4d-7923-46e6-97bb-6690514f89a4-kube-api-access-9v6nm\") pod \"prometheus-operator-9cd89b74c-xxb2q\" (UID: \"56928a4d-7923-46e6-97bb-6690514f89a4\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xxb2q" Oct 30 12:52:00.610776 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:00.610730 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cfgs4" event={"ID":"4dca982f-f509-454b-a057-d7fd9b3f34bd","Type":"ContainerStarted","Data":"f908284a875408360687c94b8077edbb517bcb1206a8b33a5930152b9498ca40"} Oct 30 12:52:00.614876 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:00.612764 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v7skv" event={"ID":"51a542f5-796e-46b1-95bd-31751d50b52b","Type":"ContainerStarted","Data":"b738bc9c88507f3c5e9bdcfe696c924180fd05f23f37d49b880ab0c9d1b5a0b1"} Oct 30 12:52:00.614876 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:00.612828 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v7skv" event={"ID":"51a542f5-796e-46b1-95bd-31751d50b52b","Type":"ContainerStarted","Data":"56a3c85e97253b20413bab512f2e5e7a8e8229f1e68364650dbff4c762d72424"} Oct 30 12:52:00.614876 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:00.612887 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-v7skv" Oct 30 12:52:00.614876 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:00.614411 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-854v4" event={"ID":"ab451161-af25-47e2-860e-f7d5186d4491","Type":"ContainerStarted","Data":"8d97ff04f2928d3937b5a84611165eaa602b8da53084f25dab137a237fa3d3c5"} Oct 30 12:52:00.682466 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:00.682416 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-cfgs4" podStartSLOduration=1.7550975389999999 podStartE2EDuration="3.682403365s" totalImagesPullingTime="1.927305826s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:51:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:00.68160087 +0000 UTC m=+48.857071770" watchObservedRunningTime="2026-10-30 12:52:00.682403365 +0000 UTC m=+48.857874264" Oct 30 12:52:00.686919 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:00.686892 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xxb2q" Oct 30 12:52:00.727738 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:00.727403 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-854v4" podStartSLOduration=1.100752903 podStartE2EDuration="3.727388262s" totalImagesPullingTime="2.626635359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:00.726776816 +0000 UTC m=+48.902247717" watchObservedRunningTime="2026-10-30 12:52:00.727388262 +0000 UTC m=+48.902859218" Oct 30 12:52:00.775804 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:00.775744 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-v7skv" podStartSLOduration=2.073492574 podStartE2EDuration="3.775729941s" totalImagesPullingTime="1.702237367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:00.775285793 +0000 UTC m=+48.950756693" watchObservedRunningTime="2026-10-30 12:52:00.775729941 +0000 UTC m=+48.951200841" Oct 30 12:52:00.980973 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:00.980901 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-xxb2q"] Oct 30 12:52:00.984147 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:52:00.984122 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56928a4d_7923_46e6_97bb_6690514f89a4.slice/crio-4407c6e829fd9619dbbaeb700be6308ccf26f6335725731cf1d5ddd53c2c6ad3 WatchSource:0}: Error finding container 4407c6e829fd9619dbbaeb700be6308ccf26f6335725731cf1d5ddd53c2c6ad3: Status 404 returned error can't find the container with id 4407c6e829fd9619dbbaeb700be6308ccf26f6335725731cf1d5ddd53c2c6ad3 Oct 30 12:52:01.618909 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:01.618870 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xxb2q" event={"ID":"56928a4d-7923-46e6-97bb-6690514f89a4","Type":"ContainerStarted","Data":"4407c6e829fd9619dbbaeb700be6308ccf26f6335725731cf1d5ddd53c2c6ad3"} Oct 30 12:52:02.627955 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:02.627915 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xxb2q" event={"ID":"56928a4d-7923-46e6-97bb-6690514f89a4","Type":"ContainerStarted","Data":"922903ec1a0e564fa7e65e50f04fbf8f074fc9a50b456efad52409382742465e"} Oct 30 12:52:02.628465 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:02.628013 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xxb2q" event={"ID":"56928a4d-7923-46e6-97bb-6690514f89a4","Type":"ContainerStarted","Data":"3ab56f6d872154ea9f19535afc0a39ed933852226e6f8a67a966b945bb2e55f8"} Oct 30 12:52:02.746158 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:02.746126 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:02.861980 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:02.861934 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:04.378761 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.378734 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:04.538073 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.538027 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:04.671126 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.671053 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:04.753173 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.753120 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xxb2q" podStartSLOduration=3.762413229 podStartE2EDuration="4.753105555s" totalImagesPullingTime="990.692326ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:02.85806153 +0000 UTC m=+51.033532429" watchObservedRunningTime="2026-10-30 12:52:04.753105555 +0000 UTC m=+52.928576456" Oct 30 12:52:04.753779 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.753762 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-4tq96"] Oct 30 12:52:04.755801 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.755786 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4tq96" Oct 30 12:52:04.767392 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.767372 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-c4jgj"] Oct 30 12:52:04.769191 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.769176 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:04.781307 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.781289 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-4tq96"] Oct 30 12:52:04.791603 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.791583 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:04.823570 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.823553 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 12:52:04.823670 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.823579 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 12:52:04.872412 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.872395 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 12:52:04.872813 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.872789 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 12:52:04.872813 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.872803 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 12:52:04.872936 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.872863 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rwc4t\"" Oct 30 12:52:04.872936 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.872879 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 12:52:04.872936 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.872918 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-66wcp\"" Oct 30 12:52:04.890186 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.890164 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fd0fdb6-c254-446d-8832-5a2fb284f03d-metrics-client-ca\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:04.890275 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.890197 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8fd0fdb6-c254-446d-8832-5a2fb284f03d-sys\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:04.890275 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.890228 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8fd0fdb6-c254-446d-8832-5a2fb284f03d-node-exporter-accelerators-collector-config\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:04.890344 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.890278 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8fd0fdb6-c254-446d-8832-5a2fb284f03d-root\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:04.890396 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.890379 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8fd0fdb6-c254-446d-8832-5a2fb284f03d-node-exporter-tls\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:04.893641 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.890470 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8fd0fdb6-c254-446d-8832-5a2fb284f03d-node-exporter-textfile\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:04.893641 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.890933 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/dd3d929e-3197-4794-9b31-794427a55693-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-4tq96\" (UID: \"dd3d929e-3197-4794-9b31-794427a55693\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4tq96" Oct 30 12:52:04.893641 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.890984 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27dm6\" (UniqueName: \"kubernetes.io/projected/8fd0fdb6-c254-446d-8832-5a2fb284f03d-kube-api-access-27dm6\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:04.893641 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.891032 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd3d929e-3197-4794-9b31-794427a55693-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-4tq96\" (UID: \"dd3d929e-3197-4794-9b31-794427a55693\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4tq96" Oct 30 12:52:04.893641 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.891066 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dd3d929e-3197-4794-9b31-794427a55693-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-4tq96\" (UID: \"dd3d929e-3197-4794-9b31-794427a55693\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4tq96" Oct 30 12:52:04.893641 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.891125 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd3d929e-3197-4794-9b31-794427a55693-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-4tq96\" (UID: \"dd3d929e-3197-4794-9b31-794427a55693\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4tq96" Oct 30 12:52:04.893641 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.891157 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/dd3d929e-3197-4794-9b31-794427a55693-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-4tq96\" (UID: \"dd3d929e-3197-4794-9b31-794427a55693\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4tq96" Oct 30 12:52:04.893641 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.891187 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8fd0fdb6-c254-446d-8832-5a2fb284f03d-node-exporter-wtmp\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:04.893641 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.891230 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8fd0fdb6-c254-446d-8832-5a2fb284f03d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:04.893641 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.891258 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8xd7\" (UniqueName: \"kubernetes.io/projected/dd3d929e-3197-4794-9b31-794427a55693-kube-api-access-m8xd7\") pod \"kube-state-metrics-5dc8d7d89b-4tq96\" (UID: \"dd3d929e-3197-4794-9b31-794427a55693\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4tq96" Oct 30 12:52:04.911551 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.911525 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:04.992277 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.992256 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8fd0fdb6-c254-446d-8832-5a2fb284f03d-sys\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:04.992359 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.992284 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8fd0fdb6-c254-446d-8832-5a2fb284f03d-node-exporter-accelerators-collector-config\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:04.992359 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.992301 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8fd0fdb6-c254-446d-8832-5a2fb284f03d-root\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:04.992359 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.992324 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8fd0fdb6-c254-446d-8832-5a2fb284f03d-node-exporter-tls\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:04.992359 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.992349 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8fd0fdb6-c254-446d-8832-5a2fb284f03d-node-exporter-textfile\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:04.992554 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.992358 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8fd0fdb6-c254-446d-8832-5a2fb284f03d-sys\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:04.992554 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.992373 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/dd3d929e-3197-4794-9b31-794427a55693-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-4tq96\" (UID: \"dd3d929e-3197-4794-9b31-794427a55693\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4tq96" Oct 30 12:52:04.992554 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.992391 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8fd0fdb6-c254-446d-8832-5a2fb284f03d-root\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:04.992554 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.992418 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27dm6\" (UniqueName: \"kubernetes.io/projected/8fd0fdb6-c254-446d-8832-5a2fb284f03d-kube-api-access-27dm6\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:04.992554 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.992453 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd3d929e-3197-4794-9b31-794427a55693-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-4tq96\" (UID: \"dd3d929e-3197-4794-9b31-794427a55693\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4tq96" Oct 30 12:52:04.992554 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:52:04.992464 2582 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Oct 30 12:52:04.992554 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.992477 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dd3d929e-3197-4794-9b31-794427a55693-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-4tq96\" (UID: \"dd3d929e-3197-4794-9b31-794427a55693\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4tq96" Oct 30 12:52:04.992554 ip-10-0-137-21 kubenswrapper[2582]: E1030 12:52:04.992520 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fd0fdb6-c254-446d-8832-5a2fb284f03d-node-exporter-tls podName:8fd0fdb6-c254-446d-8832-5a2fb284f03d nodeName:}" failed. No retries permitted until 2026-10-30 12:52:05.492505289 +0000 UTC m=+53.667976170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/8fd0fdb6-c254-446d-8832-5a2fb284f03d-node-exporter-tls") pod "node-exporter-c4jgj" (UID: "8fd0fdb6-c254-446d-8832-5a2fb284f03d") : secret "node-exporter-tls" not found Oct 30 12:52:04.992554 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.992549 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd3d929e-3197-4794-9b31-794427a55693-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-4tq96\" (UID: \"dd3d929e-3197-4794-9b31-794427a55693\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4tq96" Oct 30 12:52:04.993010 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.992575 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/dd3d929e-3197-4794-9b31-794427a55693-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-4tq96\" (UID: \"dd3d929e-3197-4794-9b31-794427a55693\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4tq96" Oct 30 12:52:04.993010 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.992601 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8fd0fdb6-c254-446d-8832-5a2fb284f03d-node-exporter-wtmp\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:04.993010 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.992644 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8fd0fdb6-c254-446d-8832-5a2fb284f03d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:04.993010 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.992666 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8xd7\" (UniqueName: \"kubernetes.io/projected/dd3d929e-3197-4794-9b31-794427a55693-kube-api-access-m8xd7\") pod \"kube-state-metrics-5dc8d7d89b-4tq96\" (UID: \"dd3d929e-3197-4794-9b31-794427a55693\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4tq96" Oct 30 12:52:04.993010 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.992717 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fd0fdb6-c254-446d-8832-5a2fb284f03d-metrics-client-ca\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:04.993010 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.992769 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8fd0fdb6-c254-446d-8832-5a2fb284f03d-node-exporter-textfile\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:04.993010 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.992813 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/dd3d929e-3197-4794-9b31-794427a55693-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-4tq96\" (UID: \"dd3d929e-3197-4794-9b31-794427a55693\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4tq96" Oct 30 12:52:04.993010 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.992948 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8fd0fdb6-c254-446d-8832-5a2fb284f03d-node-exporter-wtmp\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:04.993320 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.993049 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8fd0fdb6-c254-446d-8832-5a2fb284f03d-node-exporter-accelerators-collector-config\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:04.993320 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.993267 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fd0fdb6-c254-446d-8832-5a2fb284f03d-metrics-client-ca\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:04.993493 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.993462 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd3d929e-3197-4794-9b31-794427a55693-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-4tq96\" (UID: \"dd3d929e-3197-4794-9b31-794427a55693\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4tq96" Oct 30 12:52:04.993720 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.993694 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/dd3d929e-3197-4794-9b31-794427a55693-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-4tq96\" (UID: \"dd3d929e-3197-4794-9b31-794427a55693\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4tq96" Oct 30 12:52:04.994984 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.994965 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8fd0fdb6-c254-446d-8832-5a2fb284f03d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:04.995130 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.995114 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd3d929e-3197-4794-9b31-794427a55693-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-4tq96\" (UID: \"dd3d929e-3197-4794-9b31-794427a55693\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4tq96" Oct 30 12:52:04.995267 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:04.995245 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dd3d929e-3197-4794-9b31-794427a55693-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-4tq96\" (UID: \"dd3d929e-3197-4794-9b31-794427a55693\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4tq96" Oct 30 12:52:05.018530 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:05.018511 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27dm6\" (UniqueName: \"kubernetes.io/projected/8fd0fdb6-c254-446d-8832-5a2fb284f03d-kube-api-access-27dm6\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:05.019346 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:05.019321 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8xd7\" (UniqueName: \"kubernetes.io/projected/dd3d929e-3197-4794-9b31-794427a55693-kube-api-access-m8xd7\") pod \"kube-state-metrics-5dc8d7d89b-4tq96\" (UID: \"dd3d929e-3197-4794-9b31-794427a55693\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4tq96" Oct 30 12:52:05.067904 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:05.067876 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4tq96" Oct 30 12:52:05.094295 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:05.094260 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:05.207048 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:05.207003 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:05.331900 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:05.331847 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:05.376564 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:05.376534 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-4tq96"] Oct 30 12:52:05.379783 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:52:05.379763 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd3d929e_3197_4794_9b31_794427a55693.slice/crio-64493cb387412ea7ebe03ce195389352c6b410c126810317e7a7cef26cb9b066 WatchSource:0}: Error finding container 64493cb387412ea7ebe03ce195389352c6b410c126810317e7a7cef26cb9b066: Status 404 returned error can't find the container with id 64493cb387412ea7ebe03ce195389352c6b410c126810317e7a7cef26cb9b066 Oct 30 12:52:05.496562 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:05.496542 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8fd0fdb6-c254-446d-8832-5a2fb284f03d-node-exporter-tls\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:05.498580 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:05.498563 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8fd0fdb6-c254-446d-8832-5a2fb284f03d-node-exporter-tls\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:05.636942 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:05.636896 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4tq96" event={"ID":"dd3d929e-3197-4794-9b31-794427a55693","Type":"ContainerStarted","Data":"64493cb387412ea7ebe03ce195389352c6b410c126810317e7a7cef26cb9b066"} Oct 30 12:52:05.680930 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:05.680907 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:05.689790 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:52:05.689770 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fd0fdb6_c254_446d_8832_5a2fb284f03d.slice/crio-0fb70e7381decaff50290b92ee2c0505239b354e82ff99bf84e63be1e937c006 WatchSource:0}: Error finding container 0fb70e7381decaff50290b92ee2c0505239b354e82ff99bf84e63be1e937c006: Status 404 returned error can't find the container with id 0fb70e7381decaff50290b92ee2c0505239b354e82ff99bf84e63be1e937c006 Oct 30 12:52:05.791366 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:05.791334 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:05.907872 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:05.907758 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:06.041250 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.041218 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 12:52:06.043245 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.043200 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:06.043625 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.043588 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:06.062335 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.062303 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 12:52:06.062335 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.062320 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 12:52:06.062596 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.062564 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 12:52:06.062710 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.062647 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 12:52:06.062804 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.062776 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-khs4n\"" Oct 30 12:52:06.062877 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.062818 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 12:52:06.062933 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.062874 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 12:52:06.073843 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.073824 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 12:52:06.076580 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.076557 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 12:52:06.162125 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.162057 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 12:52:06.162125 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.162057 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 12:52:06.201524 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.201494 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:06.201759 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.201540 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:06.201759 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.201574 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-config-out\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:06.201759 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.201669 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:06.201759 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.201722 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:06.201952 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.201812 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:06.201952 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.201833 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-web-config\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:06.201952 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.201860 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:06.202093 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.201994 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:06.202093 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.202029 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:06.202093 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.202057 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82x4j\" (UniqueName: \"kubernetes.io/projected/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-kube-api-access-82x4j\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:06.202237 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.202109 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-config-volume\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:06.202237 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.202184 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:06.302794 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.302765 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:06.302945 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.302822 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:06.302945 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.302838 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:06.302945 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.302862 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82x4j\" (UniqueName: \"kubernetes.io/projected/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-kube-api-access-82x4j\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:06.302945 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.302884 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-config-volume\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:06.302945 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.302903 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:06.303181 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.303164 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:06.303346 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.303322 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:06.303437 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.303379 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:06.303437 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.303402 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-config-out\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:06.303437 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.303428 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:06.303574 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.303457 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:06.303574 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.303516 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:06.303574 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.303545 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-web-config\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:06.304385 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.304083 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:06.304503 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.304476 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:06.306019 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.305973 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:06.306196 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.306173 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-config-volume\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:06.306768 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.306747 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-web-config\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:06.306990 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.306968 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-config-out\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:06.308417 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.308341 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:06.308417 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.308385 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:06.308601 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.308580 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:06.309344 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.309312 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:06.310844 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.310820 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:06.331433 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.331407 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82x4j\" (UniqueName: \"kubernetes.io/projected/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-kube-api-access-82x4j\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:06.366213 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.366187 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:06.564949 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.564926 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:06.642860 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.642822 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c4jgj" event={"ID":"8fd0fdb6-c254-446d-8832-5a2fb284f03d","Type":"ContainerStarted","Data":"776eeda575dbe4788b49d97bc8d8d4b89d1d6a5b07824becaed6548ce820310b"} Oct 30 12:52:06.642959 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.642929 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c4jgj" event={"ID":"8fd0fdb6-c254-446d-8832-5a2fb284f03d","Type":"ContainerStarted","Data":"0fb70e7381decaff50290b92ee2c0505239b354e82ff99bf84e63be1e937c006"} Oct 30 12:52:06.646082 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.646049 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4tq96" event={"ID":"dd3d929e-3197-4794-9b31-794427a55693","Type":"ContainerStarted","Data":"bc5c0a51ec6b2f5e86459a321f83837c050dbaf6d7915cbbd9fa9a65501c087c"} Oct 30 12:52:06.646216 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.646114 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4tq96" event={"ID":"dd3d929e-3197-4794-9b31-794427a55693","Type":"ContainerStarted","Data":"dd7dccd398e0dc99cbb207352016e75dc930a49e11fe376b7596eb2d89ca0edd"} Oct 30 12:52:06.714003 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.713981 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 12:52:06.717821 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:52:06.717791 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc18cb3fe_0392_4dca_b636_4e3ab99eb0f6.slice/crio-58287815249aca0628fe9cfd989113fcd3966286481e51121bf107cd4599d14f WatchSource:0}: Error finding container 58287815249aca0628fe9cfd989113fcd3966286481e51121bf107cd4599d14f: Status 404 returned error can't find the container with id 58287815249aca0628fe9cfd989113fcd3966286481e51121bf107cd4599d14f Oct 30 12:52:06.785061 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.785027 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-85cbd9968c-p6js7"] Oct 30 12:52:06.787112 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.787088 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85cbd9968c-p6js7" Oct 30 12:52:06.805738 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.805715 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 12:52:06.805872 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.805792 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 12:52:06.805872 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.805848 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 12:52:06.805872 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.805853 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 12:52:06.805991 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.805949 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 12:52:06.808604 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.808579 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f89a7014-2966-43c7-a901-709c690b0003-console-oauth-config\") pod \"console-85cbd9968c-p6js7\" (UID: \"f89a7014-2966-43c7-a901-709c690b0003\") " pod="openshift-console/console-85cbd9968c-p6js7" Oct 30 12:52:06.808701 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.808685 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f89a7014-2966-43c7-a901-709c690b0003-service-ca\") pod \"console-85cbd9968c-p6js7\" (UID: \"f89a7014-2966-43c7-a901-709c690b0003\") " pod="openshift-console/console-85cbd9968c-p6js7" Oct 30 12:52:06.808771 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.808715 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f89a7014-2966-43c7-a901-709c690b0003-tmp\") pod \"console-85cbd9968c-p6js7\" (UID: \"f89a7014-2966-43c7-a901-709c690b0003\") " pod="openshift-console/console-85cbd9968c-p6js7" Oct 30 12:52:06.808834 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.808812 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssl6b\" (UniqueName: \"kubernetes.io/projected/f89a7014-2966-43c7-a901-709c690b0003-kube-api-access-ssl6b\") pod \"console-85cbd9968c-p6js7\" (UID: \"f89a7014-2966-43c7-a901-709c690b0003\") " pod="openshift-console/console-85cbd9968c-p6js7" Oct 30 12:52:06.808896 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.808879 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f89a7014-2966-43c7-a901-709c690b0003-oauth-serving-cert\") pod \"console-85cbd9968c-p6js7\" (UID: \"f89a7014-2966-43c7-a901-709c690b0003\") " pod="openshift-console/console-85cbd9968c-p6js7" Oct 30 12:52:06.809002 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.808988 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f89a7014-2966-43c7-a901-709c690b0003-console-serving-cert\") pod \"console-85cbd9968c-p6js7\" (UID: \"f89a7014-2966-43c7-a901-709c690b0003\") " pod="openshift-console/console-85cbd9968c-p6js7" Oct 30 12:52:06.809060 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.809045 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f89a7014-2966-43c7-a901-709c690b0003-console-config\") pod \"console-85cbd9968c-p6js7\" (UID: \"f89a7014-2966-43c7-a901-709c690b0003\") " pod="openshift-console/console-85cbd9968c-p6js7" Oct 30 12:52:06.810033 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.810014 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85cbd9968c-p6js7"] Oct 30 12:52:06.885015 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.884994 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 12:52:06.885135 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.885121 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7dvvl\"" Oct 30 12:52:06.885175 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.885144 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 12:52:06.909601 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.909581 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f89a7014-2966-43c7-a901-709c690b0003-oauth-serving-cert\") pod \"console-85cbd9968c-p6js7\" (UID: \"f89a7014-2966-43c7-a901-709c690b0003\") " pod="openshift-console/console-85cbd9968c-p6js7" Oct 30 12:52:06.909707 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.909637 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f89a7014-2966-43c7-a901-709c690b0003-console-serving-cert\") pod \"console-85cbd9968c-p6js7\" (UID: \"f89a7014-2966-43c7-a901-709c690b0003\") " pod="openshift-console/console-85cbd9968c-p6js7" Oct 30 12:52:06.909707 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.909682 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f89a7014-2966-43c7-a901-709c690b0003-console-config\") pod \"console-85cbd9968c-p6js7\" (UID: \"f89a7014-2966-43c7-a901-709c690b0003\") " pod="openshift-console/console-85cbd9968c-p6js7" Oct 30 12:52:06.909802 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.909720 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f89a7014-2966-43c7-a901-709c690b0003-console-oauth-config\") pod \"console-85cbd9968c-p6js7\" (UID: \"f89a7014-2966-43c7-a901-709c690b0003\") " pod="openshift-console/console-85cbd9968c-p6js7" Oct 30 12:52:06.909802 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.909742 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f89a7014-2966-43c7-a901-709c690b0003-service-ca\") pod \"console-85cbd9968c-p6js7\" (UID: \"f89a7014-2966-43c7-a901-709c690b0003\") " pod="openshift-console/console-85cbd9968c-p6js7" Oct 30 12:52:06.909802 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.909762 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f89a7014-2966-43c7-a901-709c690b0003-tmp\") pod \"console-85cbd9968c-p6js7\" (UID: \"f89a7014-2966-43c7-a901-709c690b0003\") " pod="openshift-console/console-85cbd9968c-p6js7" Oct 30 12:52:06.909922 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.909879 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssl6b\" (UniqueName: \"kubernetes.io/projected/f89a7014-2966-43c7-a901-709c690b0003-kube-api-access-ssl6b\") pod \"console-85cbd9968c-p6js7\" (UID: \"f89a7014-2966-43c7-a901-709c690b0003\") " pod="openshift-console/console-85cbd9968c-p6js7" Oct 30 12:52:06.910211 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.910185 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f89a7014-2966-43c7-a901-709c690b0003-tmp\") pod \"console-85cbd9968c-p6js7\" (UID: \"f89a7014-2966-43c7-a901-709c690b0003\") " pod="openshift-console/console-85cbd9968c-p6js7" Oct 30 12:52:06.910593 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.910572 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f89a7014-2966-43c7-a901-709c690b0003-oauth-serving-cert\") pod \"console-85cbd9968c-p6js7\" (UID: \"f89a7014-2966-43c7-a901-709c690b0003\") " pod="openshift-console/console-85cbd9968c-p6js7" Oct 30 12:52:06.910715 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.910695 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f89a7014-2966-43c7-a901-709c690b0003-console-config\") pod \"console-85cbd9968c-p6js7\" (UID: \"f89a7014-2966-43c7-a901-709c690b0003\") " pod="openshift-console/console-85cbd9968c-p6js7" Oct 30 12:52:06.910776 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.910695 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f89a7014-2966-43c7-a901-709c690b0003-service-ca\") pod \"console-85cbd9968c-p6js7\" (UID: \"f89a7014-2966-43c7-a901-709c690b0003\") " pod="openshift-console/console-85cbd9968c-p6js7" Oct 30 12:52:06.912182 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.912159 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f89a7014-2966-43c7-a901-709c690b0003-console-oauth-config\") pod \"console-85cbd9968c-p6js7\" (UID: \"f89a7014-2966-43c7-a901-709c690b0003\") " pod="openshift-console/console-85cbd9968c-p6js7" Oct 30 12:52:06.912261 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.912178 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f89a7014-2966-43c7-a901-709c690b0003-console-serving-cert\") pod \"console-85cbd9968c-p6js7\" (UID: \"f89a7014-2966-43c7-a901-709c690b0003\") " pod="openshift-console/console-85cbd9968c-p6js7" Oct 30 12:52:06.941388 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:06.941367 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssl6b\" (UniqueName: \"kubernetes.io/projected/f89a7014-2966-43c7-a901-709c690b0003-kube-api-access-ssl6b\") pod \"console-85cbd9968c-p6js7\" (UID: \"f89a7014-2966-43c7-a901-709c690b0003\") " pod="openshift-console/console-85cbd9968c-p6js7" Oct 30 12:52:07.099038 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:07.098786 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85cbd9968c-p6js7" Oct 30 12:52:07.400648 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:07.400565 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85cbd9968c-p6js7"] Oct 30 12:52:07.404140 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:52:07.404111 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf89a7014_2966_43c7_a901_709c690b0003.slice/crio-815dcea6522fbada33cae727dbd21ebe8040c2dced78a4d7324e4652362ad187 WatchSource:0}: Error finding container 815dcea6522fbada33cae727dbd21ebe8040c2dced78a4d7324e4652362ad187: Status 404 returned error can't find the container with id 815dcea6522fbada33cae727dbd21ebe8040c2dced78a4d7324e4652362ad187 Oct 30 12:52:07.561993 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:07.561969 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:07.650176 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:07.650150 2582 generic.go:356] "Generic (PLEG): container finished" podID="8fd0fdb6-c254-446d-8832-5a2fb284f03d" containerID="776eeda575dbe4788b49d97bc8d8d4b89d1d6a5b07824becaed6548ce820310b" exitCode=0 Oct 30 12:52:07.650521 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:07.650226 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c4jgj" event={"ID":"8fd0fdb6-c254-446d-8832-5a2fb284f03d","Type":"ContainerDied","Data":"776eeda575dbe4788b49d97bc8d8d4b89d1d6a5b07824becaed6548ce820310b"} Oct 30 12:52:07.651827 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:07.651807 2582 generic.go:356] "Generic (PLEG): container finished" podID="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" containerID="6665af12009c0962bf45c992c8dbd0324d4b4e5094dd12d19baa5bd145b23ecc" exitCode=0 Oct 30 12:52:07.651911 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:07.651858 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6","Type":"ContainerDied","Data":"6665af12009c0962bf45c992c8dbd0324d4b4e5094dd12d19baa5bd145b23ecc"} Oct 30 12:52:07.652016 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:07.651914 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6","Type":"ContainerStarted","Data":"58287815249aca0628fe9cfd989113fcd3966286481e51121bf107cd4599d14f"} Oct 30 12:52:07.654188 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:07.654162 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4tq96" event={"ID":"dd3d929e-3197-4794-9b31-794427a55693","Type":"ContainerStarted","Data":"f74a13239290aed74ededfcf178ff0b96db8467de9dc6a7ef7d86d52a069c490"} Oct 30 12:52:07.655431 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:07.655405 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85cbd9968c-p6js7" event={"ID":"f89a7014-2966-43c7-a901-709c690b0003","Type":"ContainerStarted","Data":"815dcea6522fbada33cae727dbd21ebe8040c2dced78a4d7324e4652362ad187"} Oct 30 12:52:07.690433 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:07.690412 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:07.747976 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:07.747929 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4tq96" podStartSLOduration=2.635176512 podStartE2EDuration="3.747914086s" totalImagesPullingTime="1.112737574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:07.74730536 +0000 UTC m=+55.922776260" watchObservedRunningTime="2026-10-30 12:52:07.747914086 +0000 UTC m=+55.923384987" Oct 30 12:52:07.965071 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:07.965028 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:08.078308 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:08.078257 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:08.195825 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:08.195787 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:08.487599 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:08.487511 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:08.666825 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:08.666154 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c4jgj" event={"ID":"8fd0fdb6-c254-446d-8832-5a2fb284f03d","Type":"ContainerStarted","Data":"4787a1bd5781fcc2012ff68bddd4dfbc76c4f99587599c9ffc8b5f5b1fcf9d4e"} Oct 30 12:52:08.666825 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:08.666211 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c4jgj" event={"ID":"8fd0fdb6-c254-446d-8832-5a2fb284f03d","Type":"ContainerStarted","Data":"ed1096304f8b598983c66b639d1f722e9a7176037b1d6eb9900ab588510b36c3"} Oct 30 12:52:08.730243 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:08.730183 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-c4jgj" podStartSLOduration=3.8280317029999997 podStartE2EDuration="4.730165531s" totalImagesPullingTime="827.332352ms" totalInitContainerRuntime="74.801476ms" isStatefulPod=true podCreationTimestamp="2026-10-30 12:52:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:08.729649065 +0000 UTC m=+56.905119993" watchObservedRunningTime="2026-10-30 12:52:08.730165531 +0000 UTC m=+56.905636434" Oct 30 12:52:09.278826 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.278801 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm"] Oct 30 12:52:09.280955 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.280934 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm" Oct 30 12:52:09.300568 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.300542 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 12:52:09.300708 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.300692 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 12:52:09.300831 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.300813 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-7j6fw\"" Oct 30 12:52:09.312844 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.312823 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 12:52:09.313066 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.313028 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2t7nptcd6h1f5\"" Oct 30 12:52:09.313190 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.313076 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 12:52:09.314086 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.314068 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm"] Oct 30 12:52:09.333243 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.333215 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c44f520d-f602-40e0-94d4-23277e56d67e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6d7f54bf5f-jkjtm\" (UID: \"c44f520d-f602-40e0-94d4-23277e56d67e\") " pod="openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm" Oct 30 12:52:09.333327 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.333288 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c44f520d-f602-40e0-94d4-23277e56d67e-metrics-server-audit-profiles\") pod \"metrics-server-6d7f54bf5f-jkjtm\" (UID: \"c44f520d-f602-40e0-94d4-23277e56d67e\") " pod="openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm" Oct 30 12:52:09.333392 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.333360 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c44f520d-f602-40e0-94d4-23277e56d67e-audit-log\") pod \"metrics-server-6d7f54bf5f-jkjtm\" (UID: \"c44f520d-f602-40e0-94d4-23277e56d67e\") " pod="openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm" Oct 30 12:52:09.333432 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.333422 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c44f520d-f602-40e0-94d4-23277e56d67e-secret-metrics-server-client-certs\") pod \"metrics-server-6d7f54bf5f-jkjtm\" (UID: \"c44f520d-f602-40e0-94d4-23277e56d67e\") " pod="openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm" Oct 30 12:52:09.333519 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.333498 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhkpc\" (UniqueName: \"kubernetes.io/projected/c44f520d-f602-40e0-94d4-23277e56d67e-kube-api-access-jhkpc\") pod \"metrics-server-6d7f54bf5f-jkjtm\" (UID: \"c44f520d-f602-40e0-94d4-23277e56d67e\") " pod="openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm" Oct 30 12:52:09.333646 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.333589 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c44f520d-f602-40e0-94d4-23277e56d67e-client-ca-bundle\") pod \"metrics-server-6d7f54bf5f-jkjtm\" (UID: \"c44f520d-f602-40e0-94d4-23277e56d67e\") " pod="openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm" Oct 30 12:52:09.333923 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.333648 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c44f520d-f602-40e0-94d4-23277e56d67e-secret-metrics-server-tls\") pod \"metrics-server-6d7f54bf5f-jkjtm\" (UID: \"c44f520d-f602-40e0-94d4-23277e56d67e\") " pod="openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm" Oct 30 12:52:09.434767 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.434675 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhkpc\" (UniqueName: \"kubernetes.io/projected/c44f520d-f602-40e0-94d4-23277e56d67e-kube-api-access-jhkpc\") pod \"metrics-server-6d7f54bf5f-jkjtm\" (UID: \"c44f520d-f602-40e0-94d4-23277e56d67e\") " pod="openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm" Oct 30 12:52:09.435306 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.435277 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c44f520d-f602-40e0-94d4-23277e56d67e-client-ca-bundle\") pod \"metrics-server-6d7f54bf5f-jkjtm\" (UID: \"c44f520d-f602-40e0-94d4-23277e56d67e\") " pod="openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm" Oct 30 12:52:09.435436 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.435349 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c44f520d-f602-40e0-94d4-23277e56d67e-secret-metrics-server-tls\") pod \"metrics-server-6d7f54bf5f-jkjtm\" (UID: \"c44f520d-f602-40e0-94d4-23277e56d67e\") " pod="openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm" Oct 30 12:52:09.435436 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.435393 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c44f520d-f602-40e0-94d4-23277e56d67e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6d7f54bf5f-jkjtm\" (UID: \"c44f520d-f602-40e0-94d4-23277e56d67e\") " pod="openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm" Oct 30 12:52:09.435546 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.435456 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c44f520d-f602-40e0-94d4-23277e56d67e-metrics-server-audit-profiles\") pod \"metrics-server-6d7f54bf5f-jkjtm\" (UID: \"c44f520d-f602-40e0-94d4-23277e56d67e\") " pod="openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm" Oct 30 12:52:09.435546 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.435486 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c44f520d-f602-40e0-94d4-23277e56d67e-audit-log\") pod \"metrics-server-6d7f54bf5f-jkjtm\" (UID: \"c44f520d-f602-40e0-94d4-23277e56d67e\") " pod="openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm" Oct 30 12:52:09.435729 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.435538 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c44f520d-f602-40e0-94d4-23277e56d67e-secret-metrics-server-client-certs\") pod \"metrics-server-6d7f54bf5f-jkjtm\" (UID: \"c44f520d-f602-40e0-94d4-23277e56d67e\") " pod="openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm" Oct 30 12:52:09.436245 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.436203 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c44f520d-f602-40e0-94d4-23277e56d67e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6d7f54bf5f-jkjtm\" (UID: \"c44f520d-f602-40e0-94d4-23277e56d67e\") " pod="openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm" Oct 30 12:52:09.436245 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.436229 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c44f520d-f602-40e0-94d4-23277e56d67e-audit-log\") pod \"metrics-server-6d7f54bf5f-jkjtm\" (UID: \"c44f520d-f602-40e0-94d4-23277e56d67e\") " pod="openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm" Oct 30 12:52:09.436606 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.436577 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c44f520d-f602-40e0-94d4-23277e56d67e-metrics-server-audit-profiles\") pod \"metrics-server-6d7f54bf5f-jkjtm\" (UID: \"c44f520d-f602-40e0-94d4-23277e56d67e\") " pod="openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm" Oct 30 12:52:09.438271 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.438246 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c44f520d-f602-40e0-94d4-23277e56d67e-client-ca-bundle\") pod \"metrics-server-6d7f54bf5f-jkjtm\" (UID: \"c44f520d-f602-40e0-94d4-23277e56d67e\") " pod="openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm" Oct 30 12:52:09.438752 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.438717 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c44f520d-f602-40e0-94d4-23277e56d67e-secret-metrics-server-client-certs\") pod \"metrics-server-6d7f54bf5f-jkjtm\" (UID: \"c44f520d-f602-40e0-94d4-23277e56d67e\") " pod="openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm" Oct 30 12:52:09.439259 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.439233 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c44f520d-f602-40e0-94d4-23277e56d67e-secret-metrics-server-tls\") pod \"metrics-server-6d7f54bf5f-jkjtm\" (UID: \"c44f520d-f602-40e0-94d4-23277e56d67e\") " pod="openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm" Oct 30 12:52:09.461035 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.461014 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhkpc\" (UniqueName: \"kubernetes.io/projected/c44f520d-f602-40e0-94d4-23277e56d67e-kube-api-access-jhkpc\") pod \"metrics-server-6d7f54bf5f-jkjtm\" (UID: \"c44f520d-f602-40e0-94d4-23277e56d67e\") " pod="openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm" Oct 30 12:52:09.491807 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.491744 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:09.516569 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.516540 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-spsvf"] Oct 30 12:52:09.519349 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.519327 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-spsvf" Oct 30 12:52:09.536798 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.536777 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvdzg\" (UniqueName: \"kubernetes.io/projected/14c0845c-1947-44b0-ada5-4b68ff9974f2-kube-api-access-wvdzg\") pod \"monitoring-plugin-77f4f76594-spsvf\" (UID: \"14c0845c-1947-44b0-ada5-4b68ff9974f2\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-spsvf" Oct 30 12:52:09.536873 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.536823 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/14c0845c-1947-44b0-ada5-4b68ff9974f2-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-spsvf\" (UID: \"14c0845c-1947-44b0-ada5-4b68ff9974f2\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-spsvf" Oct 30 12:52:09.543286 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.543264 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-5p7wp\"" Oct 30 12:52:09.543286 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.543279 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 12:52:09.544350 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.544328 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-spsvf"] Oct 30 12:52:09.592356 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.592331 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm" Oct 30 12:52:09.637391 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.637333 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/14c0845c-1947-44b0-ada5-4b68ff9974f2-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-spsvf\" (UID: \"14c0845c-1947-44b0-ada5-4b68ff9974f2\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-spsvf" Oct 30 12:52:09.637530 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.637512 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvdzg\" (UniqueName: \"kubernetes.io/projected/14c0845c-1947-44b0-ada5-4b68ff9974f2-kube-api-access-wvdzg\") pod \"monitoring-plugin-77f4f76594-spsvf\" (UID: \"14c0845c-1947-44b0-ada5-4b68ff9974f2\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-spsvf" Oct 30 12:52:09.641030 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.640944 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/14c0845c-1947-44b0-ada5-4b68ff9974f2-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-spsvf\" (UID: \"14c0845c-1947-44b0-ada5-4b68ff9974f2\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-spsvf" Oct 30 12:52:09.668236 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.668204 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvdzg\" (UniqueName: \"kubernetes.io/projected/14c0845c-1947-44b0-ada5-4b68ff9974f2-kube-api-access-wvdzg\") pod \"monitoring-plugin-77f4f76594-spsvf\" (UID: \"14c0845c-1947-44b0-ada5-4b68ff9974f2\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-spsvf" Oct 30 12:52:09.675367 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.675017 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6","Type":"ContainerStarted","Data":"f9faebd70ebc3a8c927575ff0f95fccf09a0cfe531b51c4f1e15b0dcc1b893a0"} Oct 30 12:52:09.675367 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.675093 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6","Type":"ContainerStarted","Data":"0f2ffec3b4f4126d8652136503fd01a5b2b2777198c21196ae9fb1417ffbc09d"} Oct 30 12:52:09.675367 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.675135 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6","Type":"ContainerStarted","Data":"f286e0644ea0a9ec6d4f6a8fb58ccc6203530437c28865238d9646445df9d24b"} Oct 30 12:52:09.675367 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.675173 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6","Type":"ContainerStarted","Data":"152c54719bb4d0ba719cbc354f48222394e0c7a65551449a7f4aa9b18422c840"} Oct 30 12:52:09.675367 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.675247 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6","Type":"ContainerStarted","Data":"c9a7dacaac898cf7e5afc6344acdf7f321d449ebd7f94c4fa42c6ac24c83535c"} Oct 30 12:52:09.714643 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.714521 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:09.831335 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.831265 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-spsvf" Oct 30 12:52:09.852314 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.852245 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:09.912890 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:09.912863 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm"] Oct 30 12:52:09.917127 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:52:09.917093 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc44f520d_f602_40e0_94d4_23277e56d67e.slice/crio-07156ce7e70955a73bee391bd2ecbcc00edb79c18b55d20af84579c41d048044 WatchSource:0}: Error finding container 07156ce7e70955a73bee391bd2ecbcc00edb79c18b55d20af84579c41d048044: Status 404 returned error can't find the container with id 07156ce7e70955a73bee391bd2ecbcc00edb79c18b55d20af84579c41d048044 Oct 30 12:52:10.012968 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.012936 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-76d946b6b9-5lvf8"] Oct 30 12:52:10.015627 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.015587 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" Oct 30 12:52:10.036713 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.036692 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 12:52:10.036816 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.036695 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 12:52:10.036816 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.036727 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 12:52:10.036816 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.036699 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 12:52:10.036816 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.036769 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-w4qng\"" Oct 30 12:52:10.041659 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.041623 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b0992e81-44ba-41c3-8079-c801a2d40aa4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76d946b6b9-5lvf8\" (UID: \"b0992e81-44ba-41c3-8079-c801a2d40aa4\") " pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" Oct 30 12:52:10.041899 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.041701 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0992e81-44ba-41c3-8079-c801a2d40aa4-serving-certs-ca-bundle\") pod \"telemeter-client-76d946b6b9-5lvf8\" (UID: \"b0992e81-44ba-41c3-8079-c801a2d40aa4\") " pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" Oct 30 12:52:10.041899 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.041739 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b0992e81-44ba-41c3-8079-c801a2d40aa4-secret-telemeter-client\") pod \"telemeter-client-76d946b6b9-5lvf8\" (UID: \"b0992e81-44ba-41c3-8079-c801a2d40aa4\") " pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" Oct 30 12:52:10.041899 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.041771 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0992e81-44ba-41c3-8079-c801a2d40aa4-metrics-client-ca\") pod \"telemeter-client-76d946b6b9-5lvf8\" (UID: \"b0992e81-44ba-41c3-8079-c801a2d40aa4\") " pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" Oct 30 12:52:10.041899 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.041839 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b0992e81-44ba-41c3-8079-c801a2d40aa4-telemeter-client-tls\") pod \"telemeter-client-76d946b6b9-5lvf8\" (UID: \"b0992e81-44ba-41c3-8079-c801a2d40aa4\") " pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" Oct 30 12:52:10.041899 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.041884 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0992e81-44ba-41c3-8079-c801a2d40aa4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76d946b6b9-5lvf8\" (UID: \"b0992e81-44ba-41c3-8079-c801a2d40aa4\") " pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" Oct 30 12:52:10.042161 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.041942 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b0992e81-44ba-41c3-8079-c801a2d40aa4-federate-client-tls\") pod \"telemeter-client-76d946b6b9-5lvf8\" (UID: \"b0992e81-44ba-41c3-8079-c801a2d40aa4\") " pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" Oct 30 12:52:10.042161 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.041964 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lltjs\" (UniqueName: \"kubernetes.io/projected/b0992e81-44ba-41c3-8079-c801a2d40aa4-kube-api-access-lltjs\") pod \"telemeter-client-76d946b6b9-5lvf8\" (UID: \"b0992e81-44ba-41c3-8079-c801a2d40aa4\") " pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" Oct 30 12:52:10.049098 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.049073 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 12:52:10.050501 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.050483 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-76d946b6b9-5lvf8"] Oct 30 12:52:10.063865 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.063845 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 12:52:10.139514 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.139484 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-spsvf"] Oct 30 12:52:10.143416 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.143385 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0992e81-44ba-41c3-8079-c801a2d40aa4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76d946b6b9-5lvf8\" (UID: \"b0992e81-44ba-41c3-8079-c801a2d40aa4\") " pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" Oct 30 12:52:10.143528 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.143464 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b0992e81-44ba-41c3-8079-c801a2d40aa4-federate-client-tls\") pod \"telemeter-client-76d946b6b9-5lvf8\" (UID: \"b0992e81-44ba-41c3-8079-c801a2d40aa4\") " pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" Oct 30 12:52:10.143528 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:52:10.143477 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14c0845c_1947_44b0_ada5_4b68ff9974f2.slice/crio-00f70bf3b3b7e70a68aa7921812aeb917267fc7aa4ee2483a0e6a458e7e26f6d WatchSource:0}: Error finding container 00f70bf3b3b7e70a68aa7921812aeb917267fc7aa4ee2483a0e6a458e7e26f6d: Status 404 returned error can't find the container with id 00f70bf3b3b7e70a68aa7921812aeb917267fc7aa4ee2483a0e6a458e7e26f6d Oct 30 12:52:10.143528 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.143492 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lltjs\" (UniqueName: \"kubernetes.io/projected/b0992e81-44ba-41c3-8079-c801a2d40aa4-kube-api-access-lltjs\") pod \"telemeter-client-76d946b6b9-5lvf8\" (UID: \"b0992e81-44ba-41c3-8079-c801a2d40aa4\") " pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" Oct 30 12:52:10.143737 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.143543 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b0992e81-44ba-41c3-8079-c801a2d40aa4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76d946b6b9-5lvf8\" (UID: \"b0992e81-44ba-41c3-8079-c801a2d40aa4\") " pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" Oct 30 12:52:10.143737 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.143578 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0992e81-44ba-41c3-8079-c801a2d40aa4-serving-certs-ca-bundle\") pod \"telemeter-client-76d946b6b9-5lvf8\" (UID: \"b0992e81-44ba-41c3-8079-c801a2d40aa4\") " pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" Oct 30 12:52:10.143737 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.143605 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b0992e81-44ba-41c3-8079-c801a2d40aa4-secret-telemeter-client\") pod \"telemeter-client-76d946b6b9-5lvf8\" (UID: \"b0992e81-44ba-41c3-8079-c801a2d40aa4\") " pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" Oct 30 12:52:10.143737 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.143648 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0992e81-44ba-41c3-8079-c801a2d40aa4-metrics-client-ca\") pod \"telemeter-client-76d946b6b9-5lvf8\" (UID: \"b0992e81-44ba-41c3-8079-c801a2d40aa4\") " pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" Oct 30 12:52:10.143737 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.143705 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b0992e81-44ba-41c3-8079-c801a2d40aa4-telemeter-client-tls\") pod \"telemeter-client-76d946b6b9-5lvf8\" (UID: \"b0992e81-44ba-41c3-8079-c801a2d40aa4\") " pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" Oct 30 12:52:10.144511 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.144430 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0992e81-44ba-41c3-8079-c801a2d40aa4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76d946b6b9-5lvf8\" (UID: \"b0992e81-44ba-41c3-8079-c801a2d40aa4\") " pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" Oct 30 12:52:10.145315 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.145210 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0992e81-44ba-41c3-8079-c801a2d40aa4-metrics-client-ca\") pod \"telemeter-client-76d946b6b9-5lvf8\" (UID: \"b0992e81-44ba-41c3-8079-c801a2d40aa4\") " pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" Oct 30 12:52:10.145315 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.145256 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0992e81-44ba-41c3-8079-c801a2d40aa4-serving-certs-ca-bundle\") pod \"telemeter-client-76d946b6b9-5lvf8\" (UID: \"b0992e81-44ba-41c3-8079-c801a2d40aa4\") " pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" Oct 30 12:52:10.146788 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.146713 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b0992e81-44ba-41c3-8079-c801a2d40aa4-telemeter-client-tls\") pod \"telemeter-client-76d946b6b9-5lvf8\" (UID: \"b0992e81-44ba-41c3-8079-c801a2d40aa4\") " pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" Oct 30 12:52:10.146788 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.146746 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b0992e81-44ba-41c3-8079-c801a2d40aa4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76d946b6b9-5lvf8\" (UID: \"b0992e81-44ba-41c3-8079-c801a2d40aa4\") " pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" Oct 30 12:52:10.147634 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.147157 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b0992e81-44ba-41c3-8079-c801a2d40aa4-federate-client-tls\") pod \"telemeter-client-76d946b6b9-5lvf8\" (UID: \"b0992e81-44ba-41c3-8079-c801a2d40aa4\") " pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" Oct 30 12:52:10.147881 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.147861 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b0992e81-44ba-41c3-8079-c801a2d40aa4-secret-telemeter-client\") pod \"telemeter-client-76d946b6b9-5lvf8\" (UID: \"b0992e81-44ba-41c3-8079-c801a2d40aa4\") " pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" Oct 30 12:52:10.171655 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.171635 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lltjs\" (UniqueName: \"kubernetes.io/projected/b0992e81-44ba-41c3-8079-c801a2d40aa4-kube-api-access-lltjs\") pod \"telemeter-client-76d946b6b9-5lvf8\" (UID: \"b0992e81-44ba-41c3-8079-c801a2d40aa4\") " pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" Oct 30 12:52:10.330181 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.330150 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" Oct 30 12:52:10.622061 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.621988 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-v7skv" Oct 30 12:52:10.659234 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.659207 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-76d946b6b9-5lvf8"] Oct 30 12:52:10.680254 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:52:10.680223 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0992e81_44ba_41c3_8079_c801a2d40aa4.slice/crio-1e3b60100d2344794fcb00673508fd88209fb2fba72fe2ce48906b9a978dd3a7 WatchSource:0}: Error finding container 1e3b60100d2344794fcb00673508fd88209fb2fba72fe2ce48906b9a978dd3a7: Status 404 returned error can't find the container with id 1e3b60100d2344794fcb00673508fd88209fb2fba72fe2ce48906b9a978dd3a7 Oct 30 12:52:10.683131 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.683096 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-spsvf" event={"ID":"14c0845c-1947-44b0-ada5-4b68ff9974f2","Type":"ContainerStarted","Data":"00f70bf3b3b7e70a68aa7921812aeb917267fc7aa4ee2483a0e6a458e7e26f6d"} Oct 30 12:52:10.690265 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.690238 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6","Type":"ContainerStarted","Data":"459c34ebd1810e07df534942315e093a3b0368420e828a7f3f8b76b44110cfee"} Oct 30 12:52:10.693794 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.693767 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm" event={"ID":"c44f520d-f602-40e0-94d4-23277e56d67e","Type":"ContainerStarted","Data":"07156ce7e70955a73bee391bd2ecbcc00edb79c18b55d20af84579c41d048044"} Oct 30 12:52:10.696868 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.696843 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85cbd9968c-p6js7" event={"ID":"f89a7014-2966-43c7-a901-709c690b0003","Type":"ContainerStarted","Data":"4062791b86ea9ea99e6ffa00a7b797c6dfc7855c0fbcf5ab34e6ca6dec169367"} Oct 30 12:52:10.751375 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.750783 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.823575682 podStartE2EDuration="5.750765376s" totalImagesPullingTime="2.915150552s" totalInitContainerRuntime="12.039142ms" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:10.748396181 +0000 UTC m=+58.923867082" watchObservedRunningTime="2026-10-30 12:52:10.750765376 +0000 UTC m=+58.926236277" Oct 30 12:52:10.796511 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.796450 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-85cbd9968c-p6js7" podStartSLOduration=1.989258651 podStartE2EDuration="4.796431346s" totalImagesPullingTime="2.807172695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:10.795469588 +0000 UTC m=+58.970940489" watchObservedRunningTime="2026-10-30 12:52:10.796431346 +0000 UTC m=+58.971902247" Oct 30 12:52:10.897566 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:10.897491 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:11.019505 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.019473 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:11.473602 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.473577 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:11.594010 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.593934 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:11.608362 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.608332 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 12:52:11.611482 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.611463 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.633361 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.633111 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 12:52:11.633361 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.633226 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 12:52:11.633361 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.633122 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 12:52:11.648658 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.647717 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 12:52:11.648658 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.647846 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 12:52:11.648658 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.647854 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 12:52:11.658197 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.657863 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 12:52:11.658197 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.657920 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 12:52:11.658197 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.657920 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 12:52:11.658197 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.657999 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 12:52:11.658197 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.658046 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 12:52:11.658197 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.658053 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 12:52:11.658197 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.658114 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5s6kd\"" Oct 30 12:52:11.658197 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.658128 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 12:52:11.658197 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.658175 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5rv5r3ic93mpd\"" Oct 30 12:52:11.659976 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.659946 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.660073 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.659985 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d876w\" (UniqueName: \"kubernetes.io/projected/ecd3d4d5-a59a-4b0e-a981-956b10d08531-kube-api-access-d876w\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.660073 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.660045 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.660187 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.660074 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.660187 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.660094 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.660187 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.660120 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ecd3d4d5-a59a-4b0e-a981-956b10d08531-config-out\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.660187 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.660164 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.660373 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.660182 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.660373 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.660221 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-config\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.660373 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.660241 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.660373 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.660266 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.660373 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.660288 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.660373 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.660308 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-web-config\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.660373 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.660335 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ecd3d4d5-a59a-4b0e-a981-956b10d08531-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.660373 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.660351 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.660830 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.660384 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.660830 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.660408 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.660830 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.660431 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.660830 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.660461 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.660830 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.660479 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.663903 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.663815 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 12:52:11.668761 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.668743 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 12:52:11.704028 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.703999 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-spsvf" event={"ID":"14c0845c-1947-44b0-ada5-4b68ff9974f2","Type":"ContainerStarted","Data":"5296c8b555b4b0613170ad66ad77788e3f9066c84fb62c9a520cbeb7950a9b8b"} Oct 30 12:52:11.704523 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.704505 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-spsvf" Oct 30 12:52:11.708236 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.708177 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" event={"ID":"b0992e81-44ba-41c3-8079-c801a2d40aa4","Type":"ContainerStarted","Data":"1e3b60100d2344794fcb00673508fd88209fb2fba72fe2ce48906b9a978dd3a7"} Oct 30 12:52:11.710168 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.710101 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm" event={"ID":"c44f520d-f602-40e0-94d4-23277e56d67e","Type":"ContainerStarted","Data":"5945e6fb18d31c958503fb59c9d0cb8a80c119a6dfffe38a0dcc0bb3ef75f360"} Oct 30 12:52:11.710940 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.710908 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-spsvf" Oct 30 12:52:11.726590 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.726561 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:11.757588 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.757543 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-spsvf" podStartSLOduration=1.440510314 podStartE2EDuration="2.757530363s" totalImagesPullingTime="1.317020049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:11.755309158 +0000 UTC m=+59.930780059" watchObservedRunningTime="2026-10-30 12:52:11.757530363 +0000 UTC m=+59.933001260" Oct 30 12:52:11.761640 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.761101 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.761640 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.761152 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d876w\" (UniqueName: \"kubernetes.io/projected/ecd3d4d5-a59a-4b0e-a981-956b10d08531-kube-api-access-d876w\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.768638 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.766435 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.768638 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.766559 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.768638 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.766588 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.768638 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.766634 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ecd3d4d5-a59a-4b0e-a981-956b10d08531-config-out\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.768638 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.766710 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.768638 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.766740 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.768638 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.766837 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-config\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.768638 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.766862 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.768638 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.766891 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.768638 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.766942 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.768638 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.766967 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-web-config\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.768638 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.767048 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ecd3d4d5-a59a-4b0e-a981-956b10d08531-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.768638 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.767073 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.768638 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.767187 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.768638 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.767216 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.768638 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.767285 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.768638 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.767340 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.769410 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.767388 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.771350 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.770122 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.771350 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.770771 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.775242 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.772875 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.777554 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.777264 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.778761 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.778385 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.778761 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.778425 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.779501 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.779211 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.779501 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.779277 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.779501 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.779384 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-web-config\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.779501 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.779461 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.780829 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.780799 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.781159 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.781081 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.781447 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.781421 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ecd3d4d5-a59a-4b0e-a981-956b10d08531-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.781775 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.781734 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ecd3d4d5-a59a-4b0e-a981-956b10d08531-config-out\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.781975 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.781955 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-config\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.782035 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.781899 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.783211 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.783192 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.783707 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.783685 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.784830 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.784784 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.790018 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.789996 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d876w\" (UniqueName: \"kubernetes.io/projected/ecd3d4d5-a59a-4b0e-a981-956b10d08531-kube-api-access-d876w\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:11.866466 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.866378 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm" podStartSLOduration=1.462407172 podStartE2EDuration="2.866364559s" totalImagesPullingTime="1.403957387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:11.865800229 +0000 UTC m=+60.041271129" watchObservedRunningTime="2026-10-30 12:52:11.866364559 +0000 UTC m=+60.041835459" Oct 30 12:52:11.932174 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:11.932138 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:12.280375 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:12.280346 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 12:52:12.294490 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:52:12.294375 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecd3d4d5_a59a_4b0e_a981_956b10d08531.slice/crio-3410ea027ea71f3e80e819cc87d3a28cf98bce131de13e4a81f8b081e6458cae WatchSource:0}: Error finding container 3410ea027ea71f3e80e819cc87d3a28cf98bce131de13e4a81f8b081e6458cae: Status 404 returned error can't find the container with id 3410ea027ea71f3e80e819cc87d3a28cf98bce131de13e4a81f8b081e6458cae Oct 30 12:52:12.714505 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:12.714473 2582 generic.go:356] "Generic (PLEG): container finished" podID="ecd3d4d5-a59a-4b0e-a981-956b10d08531" containerID="2bdb163e8f97e8e5afd18e9046bd2c976fa179cc65c409ed26d34fa58f041ab4" exitCode=0 Oct 30 12:52:12.714942 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:12.714565 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ecd3d4d5-a59a-4b0e-a981-956b10d08531","Type":"ContainerDied","Data":"2bdb163e8f97e8e5afd18e9046bd2c976fa179cc65c409ed26d34fa58f041ab4"} Oct 30 12:52:12.714942 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:12.714703 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ecd3d4d5-a59a-4b0e-a981-956b10d08531","Type":"ContainerStarted","Data":"3410ea027ea71f3e80e819cc87d3a28cf98bce131de13e4a81f8b081e6458cae"} Oct 30 12:52:12.716744 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:12.716719 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" event={"ID":"b0992e81-44ba-41c3-8079-c801a2d40aa4","Type":"ContainerStarted","Data":"d9b937f7a2d26f5024a78187fd87be286a2a7a7dcf82343ec11522830cfe6d2b"} Oct 30 12:52:12.716831 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:12.716782 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" event={"ID":"b0992e81-44ba-41c3-8079-c801a2d40aa4","Type":"ContainerStarted","Data":"741f258ed5dc61e747eff1ad70d3dd5c0f61a1fb2592137753475541881d5219"} Oct 30 12:52:12.716831 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:12.716804 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" event={"ID":"b0992e81-44ba-41c3-8079-c801a2d40aa4","Type":"ContainerStarted","Data":"361e695d666223c7a440b7785d9ea96559017858200ea4306ee2dc26df1d05d0"} Oct 30 12:52:12.847716 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:12.847646 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" podStartSLOduration=2.01329472 podStartE2EDuration="3.847634368s" totalImagesPullingTime="1.834339648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:12.846862176 +0000 UTC m=+61.022333086" watchObservedRunningTime="2026-10-30 12:52:12.847634368 +0000 UTC m=+61.023105272" Oct 30 12:52:14.275144 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:14.275110 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:14.396286 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:14.396247 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:15.735226 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:15.735192 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ecd3d4d5-a59a-4b0e-a981-956b10d08531","Type":"ContainerStarted","Data":"389ef54daca67f771705e3e74fe397815377121bb962ae7740b822516e1ce5a3"} Oct 30 12:52:15.735602 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:15.735288 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ecd3d4d5-a59a-4b0e-a981-956b10d08531","Type":"ContainerStarted","Data":"f758417ec1675d0ce680c01e967da15988b5cc0396604b5b43b45c7a87d11498"} Oct 30 12:52:17.099508 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:17.099474 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85cbd9968c-p6js7" Oct 30 12:52:17.100056 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:17.099537 2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-85cbd9968c-p6js7" Oct 30 12:52:17.105750 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:17.105725 2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-85cbd9968c-p6js7" Oct 30 12:52:17.752911 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:17.752875 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ecd3d4d5-a59a-4b0e-a981-956b10d08531","Type":"ContainerStarted","Data":"733c9a46d88bf34e394c3f801c2a74a66a16cf1b882c2f5efbc9ca100a4714e6"} Oct 30 12:52:17.753057 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:17.752968 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ecd3d4d5-a59a-4b0e-a981-956b10d08531","Type":"ContainerStarted","Data":"3d067f86de45fb7da4d703f9d8ddfdf02e51e9db07b94ac0e02685e0a8f0e5f7"} Oct 30 12:52:17.753057 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:17.752994 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ecd3d4d5-a59a-4b0e-a981-956b10d08531","Type":"ContainerStarted","Data":"c626537d69a0eccf8091bdf1e44c648d000558fd2f939e9f7c4124f238c3219d"} Oct 30 12:52:17.753057 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:17.753019 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ecd3d4d5-a59a-4b0e-a981-956b10d08531","Type":"ContainerStarted","Data":"966d7771b9f840d69179f59a1557d5269698ef1b542ddcf4e8791e225816c6d4"} Oct 30 12:52:17.757309 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:17.757285 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-85cbd9968c-p6js7" Oct 30 12:52:17.833650 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:17.833573 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.197887526 podStartE2EDuration="6.833559981s" totalImagesPullingTime="4.615273994s" totalInitContainerRuntime="20.398461ms" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:17.832213068 +0000 UTC m=+66.007684005" watchObservedRunningTime="2026-10-30 12:52:17.833559981 +0000 UTC m=+66.009030881" Oct 30 12:52:18.129311 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.129232 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fcccc7b-a83c-4c52-a648-cb549e70b54d-metrics-certs\") pod \"network-metrics-daemon-tpz8b\" (UID: \"1fcccc7b-a83c-4c52-a648-cb549e70b54d\") " pod="openshift-multus/network-metrics-daemon-tpz8b" Oct 30 12:52:18.129311 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.129300 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50bec85b-5592-437d-aa6f-4899aeb494a5-original-pull-secret\") pod \"global-pull-secret-syncer-s7fsp\" (UID: \"50bec85b-5592-437d-aa6f-4899aeb494a5\") " pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:52:18.145197 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.145173 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 12:52:18.145275 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.145251 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 12:52:18.152422 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.152398 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50bec85b-5592-437d-aa6f-4899aeb494a5-original-pull-secret\") pod \"global-pull-secret-syncer-s7fsp\" (UID: \"50bec85b-5592-437d-aa6f-4899aeb494a5\") " pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:52:18.152513 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.152494 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fcccc7b-a83c-4c52-a648-cb549e70b54d-metrics-certs\") pod \"network-metrics-daemon-tpz8b\" (UID: \"1fcccc7b-a83c-4c52-a648-cb549e70b54d\") " pod="openshift-multus/network-metrics-daemon-tpz8b" Oct 30 12:52:18.229971 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.229946 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqvdp\" (UniqueName: \"kubernetes.io/projected/1bb48bfa-827b-4179-b9fa-418fc14f0f44-kube-api-access-dqvdp\") pod \"network-check-target-95n9l\" (UID: \"1bb48bfa-827b-4179-b9fa-418fc14f0f44\") " pod="openshift-network-diagnostics/network-check-target-95n9l" Oct 30 12:52:18.245482 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.245458 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 12:52:18.266103 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.266083 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 12:52:18.273560 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.273532 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqvdp\" (UniqueName: \"kubernetes.io/projected/1bb48bfa-827b-4179-b9fa-418fc14f0f44-kube-api-access-dqvdp\") pod \"network-check-target-95n9l\" (UID: \"1bb48bfa-827b-4179-b9fa-418fc14f0f44\") " pod="openshift-network-diagnostics/network-check-target-95n9l" Oct 30 12:52:18.318205 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.318172 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-68fxr\"" Oct 30 12:52:18.323659 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.323643 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95n9l" Oct 30 12:52:18.324192 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.324176 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-wqgtx\"" Oct 30 12:52:18.329329 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.329314 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:52:18.329548 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.329527 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hszmd\"" Oct 30 12:52:18.334509 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.334493 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpz8b" Oct 30 12:52:18.413882 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.412888 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-689bbd6cb4-lmgqn"] Oct 30 12:52:18.416657 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.416391 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:18.437946 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.437926 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-689bbd6cb4-lmgqn"] Oct 30 12:52:18.451877 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.451859 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 12:52:18.532410 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.532389 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01873344-73c8-47b2-b6ba-8e338ba18315-console-serving-cert\") pod \"console-689bbd6cb4-lmgqn\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:18.532496 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.532420 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w592\" (UniqueName: \"kubernetes.io/projected/01873344-73c8-47b2-b6ba-8e338ba18315-kube-api-access-2w592\") pod \"console-689bbd6cb4-lmgqn\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:18.532496 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.532447 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01873344-73c8-47b2-b6ba-8e338ba18315-trusted-ca-bundle\") pod \"console-689bbd6cb4-lmgqn\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:18.532578 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.532557 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01873344-73c8-47b2-b6ba-8e338ba18315-console-oauth-config\") pod \"console-689bbd6cb4-lmgqn\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:18.532631 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.532584 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01873344-73c8-47b2-b6ba-8e338ba18315-service-ca\") pod \"console-689bbd6cb4-lmgqn\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:18.532631 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.532599 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01873344-73c8-47b2-b6ba-8e338ba18315-tmp\") pod \"console-689bbd6cb4-lmgqn\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:18.532707 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.532642 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01873344-73c8-47b2-b6ba-8e338ba18315-oauth-serving-cert\") pod \"console-689bbd6cb4-lmgqn\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:18.532707 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.532671 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01873344-73c8-47b2-b6ba-8e338ba18315-console-config\") pod \"console-689bbd6cb4-lmgqn\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:18.633544 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.633514 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01873344-73c8-47b2-b6ba-8e338ba18315-trusted-ca-bundle\") pod \"console-689bbd6cb4-lmgqn\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:18.633705 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.633571 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01873344-73c8-47b2-b6ba-8e338ba18315-console-oauth-config\") pod \"console-689bbd6cb4-lmgqn\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:18.633705 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.633598 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01873344-73c8-47b2-b6ba-8e338ba18315-service-ca\") pod \"console-689bbd6cb4-lmgqn\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:18.633820 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.633724 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01873344-73c8-47b2-b6ba-8e338ba18315-tmp\") pod \"console-689bbd6cb4-lmgqn\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:18.633820 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.633765 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01873344-73c8-47b2-b6ba-8e338ba18315-oauth-serving-cert\") pod \"console-689bbd6cb4-lmgqn\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:18.633820 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.633806 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01873344-73c8-47b2-b6ba-8e338ba18315-console-config\") pod \"console-689bbd6cb4-lmgqn\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:18.633966 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.633887 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01873344-73c8-47b2-b6ba-8e338ba18315-console-serving-cert\") pod \"console-689bbd6cb4-lmgqn\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:18.633966 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.633911 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w592\" (UniqueName: \"kubernetes.io/projected/01873344-73c8-47b2-b6ba-8e338ba18315-kube-api-access-2w592\") pod \"console-689bbd6cb4-lmgqn\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:18.634057 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.634001 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-95n9l"] Oct 30 12:52:18.634153 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.634134 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01873344-73c8-47b2-b6ba-8e338ba18315-tmp\") pod \"console-689bbd6cb4-lmgqn\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:18.634338 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.634316 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01873344-73c8-47b2-b6ba-8e338ba18315-service-ca\") pod \"console-689bbd6cb4-lmgqn\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:18.634572 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.634550 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01873344-73c8-47b2-b6ba-8e338ba18315-console-config\") pod \"console-689bbd6cb4-lmgqn\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:18.634660 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.634630 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01873344-73c8-47b2-b6ba-8e338ba18315-trusted-ca-bundle\") pod \"console-689bbd6cb4-lmgqn\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:18.635191 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.635150 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01873344-73c8-47b2-b6ba-8e338ba18315-oauth-serving-cert\") pod \"console-689bbd6cb4-lmgqn\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:18.636377 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.636290 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01873344-73c8-47b2-b6ba-8e338ba18315-console-oauth-config\") pod \"console-689bbd6cb4-lmgqn\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:18.637042 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.637022 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-s7fsp"] Oct 30 12:52:18.638891 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:52:18.638864 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bb48bfa_827b_4179_b9fa_418fc14f0f44.slice/crio-3fed4127a265b40e1040260f5bbf5bcc0d93021fbf60ba8b2dac44897c94681f WatchSource:0}: Error finding container 3fed4127a265b40e1040260f5bbf5bcc0d93021fbf60ba8b2dac44897c94681f: Status 404 returned error can't find the container with id 3fed4127a265b40e1040260f5bbf5bcc0d93021fbf60ba8b2dac44897c94681f Oct 30 12:52:18.639279 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.639260 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01873344-73c8-47b2-b6ba-8e338ba18315-console-serving-cert\") pod \"console-689bbd6cb4-lmgqn\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:18.640832 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:52:18.640812 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50bec85b_5592_437d_aa6f_4899aeb494a5.slice/crio-a76e4cca2d9ac374f3d281c532a10ca542230fb3203096c2b2727d6e57f2f39a WatchSource:0}: Error finding container a76e4cca2d9ac374f3d281c532a10ca542230fb3203096c2b2727d6e57f2f39a: Status 404 returned error can't find the container with id a76e4cca2d9ac374f3d281c532a10ca542230fb3203096c2b2727d6e57f2f39a Oct 30 12:52:18.660464 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.660441 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w592\" (UniqueName: \"kubernetes.io/projected/01873344-73c8-47b2-b6ba-8e338ba18315-kube-api-access-2w592\") pod \"console-689bbd6cb4-lmgqn\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:18.667004 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.666982 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tpz8b"] Oct 30 12:52:18.669842 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:52:18.669823 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fcccc7b_a83c_4c52_a648_cb549e70b54d.slice/crio-8b0ab6dfec3ac6c49e388269cf07f3ee06fb4d7ee5f351155e4ccf3f9e88f43a WatchSource:0}: Error finding container 8b0ab6dfec3ac6c49e388269cf07f3ee06fb4d7ee5f351155e4ccf3f9e88f43a: Status 404 returned error can't find the container with id 8b0ab6dfec3ac6c49e388269cf07f3ee06fb4d7ee5f351155e4ccf3f9e88f43a Oct 30 12:52:18.694009 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.693989 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:18.727362 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.727341 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:18.757200 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.757177 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-s7fsp" event={"ID":"50bec85b-5592-437d-aa6f-4899aeb494a5","Type":"ContainerStarted","Data":"a76e4cca2d9ac374f3d281c532a10ca542230fb3203096c2b2727d6e57f2f39a"} Oct 30 12:52:18.758366 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.758344 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tpz8b" event={"ID":"1fcccc7b-a83c-4c52-a648-cb549e70b54d","Type":"ContainerStarted","Data":"8b0ab6dfec3ac6c49e388269cf07f3ee06fb4d7ee5f351155e4ccf3f9e88f43a"} Oct 30 12:52:18.759581 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.759561 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-95n9l" event={"ID":"1bb48bfa-827b-4179-b9fa-418fc14f0f44","Type":"ContainerStarted","Data":"3fed4127a265b40e1040260f5bbf5bcc0d93021fbf60ba8b2dac44897c94681f"} Oct 30 12:52:18.827400 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:18.827367 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:19.024850 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:19.024817 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-689bbd6cb4-lmgqn"] Oct 30 12:52:19.028022 ip-10-0-137-21 kubenswrapper[2582]: W1030 12:52:19.027992 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01873344_73c8_47b2_b6ba_8e338ba18315.slice/crio-0647e9ff213d9818f0c5cc4f86bd064d5d41a6318831f00fb5d98ca6c711629e WatchSource:0}: Error finding container 0647e9ff213d9818f0c5cc4f86bd064d5d41a6318831f00fb5d98ca6c711629e: Status 404 returned error can't find the container with id 0647e9ff213d9818f0c5cc4f86bd064d5d41a6318831f00fb5d98ca6c711629e Oct 30 12:52:19.432703 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:19.432587 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:19.551973 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:19.551893 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:19.767092 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:19.766968 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-689bbd6cb4-lmgqn" event={"ID":"01873344-73c8-47b2-b6ba-8e338ba18315","Type":"ContainerStarted","Data":"5a9387eb948f972dc6d4865e6a5a4f799b675aefee291ec250feccd574daea53"} Oct 30 12:52:19.767092 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:19.767062 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-689bbd6cb4-lmgqn" event={"ID":"01873344-73c8-47b2-b6ba-8e338ba18315","Type":"ContainerStarted","Data":"0647e9ff213d9818f0c5cc4f86bd064d5d41a6318831f00fb5d98ca6c711629e"} Oct 30 12:52:19.827142 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:19.826574 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-689bbd6cb4-lmgqn" podStartSLOduration=1.8265561909999999 podStartE2EDuration="1.826556191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:19.82508365 +0000 UTC m=+68.000554568" watchObservedRunningTime="2026-10-30 12:52:19.826556191 +0000 UTC m=+68.002027093" Oct 30 12:52:20.779635 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:20.778731 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tpz8b" event={"ID":"1fcccc7b-a83c-4c52-a648-cb549e70b54d","Type":"ContainerStarted","Data":"d01b2d511e10fa74b11c070ab83143e8c706b5f1df44d188fd55b2437b1982dd"} Oct 30 12:52:20.779635 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:20.778798 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tpz8b" event={"ID":"1fcccc7b-a83c-4c52-a648-cb549e70b54d","Type":"ContainerStarted","Data":"21247d5197ae31576702d71bd087bda288014c5858ec59cfa4fbf1de27334de6"} Oct 30 12:52:20.827358 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:20.826122 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tpz8b" podStartSLOduration=67.782422012 podStartE2EDuration="1m8.826100905s" totalImagesPullingTime="1.043678893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:20.824780816 +0000 UTC m=+69.000251719" watchObservedRunningTime="2026-10-30 12:52:20.826100905 +0000 UTC m=+69.001571808" Oct 30 12:52:21.780443 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:21.780408 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-95n9l" event={"ID":"1bb48bfa-827b-4179-b9fa-418fc14f0f44","Type":"ContainerStarted","Data":"8f027603980f8f7eeba2c92e9e8952f66da094b84fe4d80d766b2b30d3ee8343"} Oct 30 12:52:21.780859 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:21.780743 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-95n9l" Oct 30 12:52:21.826743 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:21.826690 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-95n9l" podStartSLOduration=66.774357264 podStartE2EDuration="1m9.826673576s" totalImagesPullingTime="3.052316312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:21.826022746 +0000 UTC m=+70.001493645" watchObservedRunningTime="2026-10-30 12:52:21.826673576 +0000 UTC m=+70.002144478" Oct 30 12:52:21.932405 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:21.932375 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:22.552439 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:22.552399 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:22.670938 ip-10-0-137-21 kubenswrapper[2582]: I1030 12:52:22.670898 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:23.454590 ip-10-0-137-21 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 12:52:23.470945 ip-10-0-137-21 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 12:52:23.471114 ip-10-0-137-21 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 12:52:23.471162 ip-10-0-137-21 systemd[1]: kubelet.service: Consumed 4.417s CPU time. Oct 30 12:52:23.482109 ip-10-0-137-21 systemd[1]: Starting Kubernetes Kubelet... Oct 30 12:52:23.561039 ip-10-0-137-21 kubenswrapper[5922]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 12:52:23.561039 ip-10-0-137-21 kubenswrapper[5922]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 12:52:23.561039 ip-10-0-137-21 kubenswrapper[5922]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 12:52:23.561039 ip-10-0-137-21 kubenswrapper[5922]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 12:52:23.563042 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.562971 5922 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:52:23.563042 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563042 5922 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:52:23.563042 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563046 5922 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:52:23.563136 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563049 5922 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:52:23.563136 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563052 5922 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:52:23.563136 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563055 5922 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:52:23.563136 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563057 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:52:23.563136 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563060 5922 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:52:23.563136 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563065 5922 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:52:23.563136 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563070 5922 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:52:23.563136 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563073 5922 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:52:23.563136 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563076 5922 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:52:23.563136 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563079 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:52:23.563136 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563086 5922 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:52:23.563136 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563089 5922 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:52:23.563136 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563092 5922 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:52:23.563136 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563095 5922 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:52:23.563136 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563097 5922 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:52:23.563136 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563100 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:52:23.563136 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563102 5922 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:52:23.563136 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563105 5922 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:52:23.563136 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563108 5922 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:52:23.563586 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563110 5922 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:52:23.563586 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563113 5922 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:52:23.563586 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563116 5922 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:52:23.563586 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563120 5922 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:52:23.563586 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563123 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:52:23.563586 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563126 5922 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:52:23.563586 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563128 5922 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:52:23.563586 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563131 5922 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:52:23.563586 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563134 5922 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:52:23.563586 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563137 5922 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:52:23.563586 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563140 5922 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:52:23.563586 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563142 5922 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:52:23.563586 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563145 5922 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:52:23.563586 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563148 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:52:23.563586 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563150 5922 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:52:23.563586 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563153 5922 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:52:23.563586 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563156 5922 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:52:23.563586 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563158 5922 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:52:23.563586 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563161 5922 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:52:23.563586 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563163 5922 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:52:23.564062 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563166 5922 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:52:23.564062 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563168 5922 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:52:23.564062 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563171 5922 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:52:23.564062 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563174 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:52:23.564062 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563176 5922 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:52:23.564062 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563179 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:52:23.564062 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563181 5922 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:52:23.564062 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563184 5922 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:52:23.564062 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563186 5922 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:52:23.564062 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563189 5922 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:52:23.564062 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563191 5922 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:52:23.564062 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563198 5922 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:52:23.564062 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563201 5922 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:52:23.564062 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563204 5922 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:52:23.564062 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563206 5922 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:52:23.564062 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563209 5922 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:52:23.564062 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563212 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:52:23.564062 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563215 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:52:23.564062 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563217 5922 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:52:23.564503 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563220 5922 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:52:23.564503 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563222 5922 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:52:23.564503 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563225 5922 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:52:23.564503 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563229 5922 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:52:23.564503 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563231 5922 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:52:23.564503 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563234 5922 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:52:23.564503 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563237 5922 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:52:23.564503 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563239 5922 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:52:23.564503 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563242 5922 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:52:23.564503 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563244 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:52:23.564503 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563247 5922 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:52:23.564503 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563249 5922 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:52:23.564503 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563252 5922 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:52:23.564503 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563255 5922 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:52:23.564503 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563257 5922 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:52:23.564503 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563260 5922 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:52:23.564503 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563263 5922 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:52:23.564503 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563266 5922 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:52:23.564503 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563269 5922 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:52:23.564969 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563271 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:52:23.564969 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563274 5922 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:52:23.564969 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563277 5922 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:52:23.564969 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563279 5922 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:52:23.564969 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563282 5922 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:52:23.564969 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563287 5922 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:52:23.564969 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563291 5922 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:52:23.564969 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563294 5922 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:52:23.564969 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563297 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:52:23.564969 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563300 5922 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:52:23.564969 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563303 5922 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:52:23.564969 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563306 5922 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:52:23.564969 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563308 5922 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:52:23.564969 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563311 5922 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:52:23.564969 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563313 5922 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:52:23.564969 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563316 5922 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:52:23.564969 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563318 5922 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:52:23.564969 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563322 5922 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:52:23.564969 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563324 5922 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:52:23.565449 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563327 5922 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:52:23.565449 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563330 5922 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:52:23.565449 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563332 5922 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:52:23.565449 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563335 5922 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:52:23.565449 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563337 5922 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:52:23.565449 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563340 5922 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:52:23.565449 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.563342 5922 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:52:23.565449 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564088 5922 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:52:23.565449 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564095 5922 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:52:23.565449 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564098 5922 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:52:23.565449 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564101 5922 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:52:23.565449 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564104 5922 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:52:23.565449 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564107 5922 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:52:23.565449 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564109 5922 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:52:23.565449 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564112 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:52:23.565449 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564115 5922 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:52:23.565449 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564118 5922 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:52:23.565449 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564120 5922 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:52:23.565449 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564125 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:52:23.565449 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564128 5922 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:52:23.565936 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564131 5922 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:52:23.565936 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564133 5922 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:52:23.565936 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564136 5922 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:52:23.565936 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564139 5922 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:52:23.565936 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564141 5922 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:52:23.565936 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564144 5922 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:52:23.565936 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564147 5922 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:52:23.565936 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564150 5922 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:52:23.565936 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564153 5922 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:52:23.565936 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564155 5922 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:52:23.565936 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564158 5922 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:52:23.565936 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564161 5922 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:52:23.565936 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564163 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:52:23.565936 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564166 5922 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:52:23.565936 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564168 5922 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:52:23.565936 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564171 5922 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:52:23.565936 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564173 5922 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:52:23.565936 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564176 5922 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:52:23.565936 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564179 5922 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:52:23.565936 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564181 5922 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:52:23.566401 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564184 5922 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:52:23.566401 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564187 5922 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:52:23.566401 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564189 5922 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:52:23.566401 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564193 5922 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:52:23.566401 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564196 5922 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:52:23.566401 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564198 5922 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:52:23.566401 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564201 5922 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:52:23.566401 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564204 5922 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:52:23.566401 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564207 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:52:23.566401 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564210 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:52:23.566401 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564214 5922 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:52:23.566401 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564217 5922 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:52:23.566401 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564219 5922 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:52:23.566401 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564222 5922 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:52:23.566401 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564225 5922 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:52:23.566401 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564227 5922 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:52:23.566401 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564230 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:52:23.566401 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564232 5922 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:52:23.566401 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564236 5922 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:52:23.566873 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564238 5922 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:52:23.566873 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564241 5922 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:52:23.566873 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564243 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:52:23.566873 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564246 5922 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:52:23.566873 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564249 5922 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:52:23.566873 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564251 5922 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:52:23.566873 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564254 5922 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:52:23.566873 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564256 5922 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:52:23.566873 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564259 5922 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:52:23.566873 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564261 5922 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:52:23.566873 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564264 5922 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:52:23.566873 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564266 5922 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:52:23.566873 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564269 5922 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:52:23.566873 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564271 5922 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:52:23.566873 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564274 5922 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:52:23.566873 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564276 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:52:23.566873 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564279 5922 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:52:23.566873 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564282 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:52:23.566873 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564285 5922 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:52:23.567317 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564288 5922 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:52:23.567317 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564290 5922 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:52:23.567317 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564295 5922 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:52:23.567317 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564298 5922 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:52:23.567317 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564303 5922 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:52:23.567317 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564307 5922 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:52:23.567317 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564310 5922 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:52:23.567317 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564312 5922 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:52:23.567317 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564315 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:52:23.567317 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564318 5922 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:52:23.567317 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564321 5922 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:52:23.567317 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564325 5922 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:52:23.567317 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564329 5922 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:52:23.567317 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564333 5922 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:52:23.567317 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564335 5922 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:52:23.567317 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564338 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:52:23.567317 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564342 5922 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:52:23.567317 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564344 5922 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:52:23.567317 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564347 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:52:23.567771 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564349 5922 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:52:23.567771 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564352 5922 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:52:23.567771 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564354 5922 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:52:23.567771 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564357 5922 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:52:23.567771 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564359 5922 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:52:23.567771 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564362 5922 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:52:23.567771 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564365 5922 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:52:23.567771 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564367 5922 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:52:23.567771 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564370 5922 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:52:23.567771 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564372 5922 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:52:23.567771 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564375 5922 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:52:23.567771 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564377 5922 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:52:23.567771 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564385 5922 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:52:23.567771 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564388 5922 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:52:23.567771 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564391 5922 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:52:23.567771 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564394 5922 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:52:23.567771 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564906 5922 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:52:23.567771 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564914 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:52:23.567771 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564917 5922 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:52:23.568209 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564920 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:52:23.568209 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564922 5922 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:52:23.568209 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564925 5922 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:52:23.568209 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564927 5922 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:52:23.568209 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564930 5922 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:52:23.568209 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564933 5922 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:52:23.568209 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564936 5922 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:52:23.568209 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564939 5922 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:52:23.568209 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564942 5922 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:52:23.568209 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564944 5922 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:52:23.568209 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564947 5922 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:52:23.568209 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564950 5922 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:52:23.568209 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564952 5922 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:52:23.568209 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564955 5922 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:52:23.568209 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564958 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:52:23.568209 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564960 5922 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:52:23.568209 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564963 5922 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:52:23.568209 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564966 5922 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:52:23.568209 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564968 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:52:23.568699 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564971 5922 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:52:23.568699 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564974 5922 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:52:23.568699 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564977 5922 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:52:23.568699 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564979 5922 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:52:23.568699 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564982 5922 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:52:23.568699 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564984 5922 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:52:23.568699 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564988 5922 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:52:23.568699 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564991 5922 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:52:23.568699 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564993 5922 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:52:23.568699 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564996 5922 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:52:23.568699 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.564999 5922 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:52:23.568699 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565002 5922 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:52:23.568699 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565005 5922 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:52:23.568699 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565008 5922 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:52:23.568699 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565010 5922 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:52:23.568699 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565013 5922 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:52:23.568699 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565015 5922 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:52:23.568699 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565018 5922 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:52:23.568699 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565020 5922 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:52:23.568699 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565023 5922 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:52:23.569183 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565025 5922 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:52:23.569183 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565030 5922 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:52:23.569183 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565033 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:52:23.569183 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565036 5922 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:52:23.569183 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565039 5922 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:52:23.569183 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565042 5922 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:52:23.569183 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565044 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:52:23.569183 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565047 5922 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:52:23.569183 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565049 5922 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:52:23.569183 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565052 5922 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:52:23.569183 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565055 5922 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:52:23.569183 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565058 5922 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:52:23.569183 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565061 5922 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:52:23.569183 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565063 5922 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:52:23.569183 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565066 5922 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:52:23.569183 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565068 5922 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:52:23.569183 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565071 5922 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:52:23.569183 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565073 5922 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:52:23.569183 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565076 5922 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:52:23.569661 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565079 5922 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:52:23.569661 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565081 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:52:23.569661 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565084 5922 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:52:23.569661 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565087 5922 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:52:23.569661 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565091 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:52:23.569661 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565094 5922 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:52:23.569661 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565096 5922 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:52:23.569661 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565099 5922 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:52:23.569661 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565102 5922 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:52:23.569661 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565104 5922 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:52:23.569661 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565107 5922 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:52:23.569661 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565109 5922 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:52:23.569661 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565112 5922 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:52:23.569661 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565115 5922 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:52:23.569661 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565117 5922 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:52:23.569661 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565120 5922 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:52:23.569661 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565122 5922 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:52:23.569661 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565125 5922 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:52:23.569661 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565128 5922 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:52:23.570106 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565130 5922 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:52:23.570106 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565133 5922 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:52:23.570106 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565135 5922 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:52:23.570106 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565138 5922 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:52:23.570106 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565145 5922 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:52:23.570106 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565147 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:52:23.570106 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565150 5922 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:52:23.570106 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565153 5922 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:52:23.570106 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565155 5922 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:52:23.570106 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565158 5922 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:52:23.570106 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565160 5922 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:52:23.570106 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565163 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:52:23.570106 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565166 5922 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:52:23.570106 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565170 5922 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:52:23.570106 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565174 5922 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:52:23.570106 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565177 5922 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:52:23.570106 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565180 5922 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:52:23.570106 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565186 5922 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:52:23.570106 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565188 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:52:23.570106 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565191 5922 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:52:23.570622 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565196 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:52:23.570622 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565198 5922 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:52:23.570622 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565201 5922 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:52:23.570622 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565204 5922 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:52:23.570622 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565206 5922 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:52:23.570622 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565209 5922 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:52:23.570622 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565300 5922 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 12:52:23.570622 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565308 5922 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 12:52:23.570622 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565322 5922 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 12:52:23.570622 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565326 5922 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 12:52:23.570622 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565331 5922 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 12:52:23.570622 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565334 5922 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 12:52:23.570622 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565338 5922 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 12:52:23.570622 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565342 5922 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 12:52:23.570622 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565345 5922 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 12:52:23.570622 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565349 5922 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 12:52:23.570622 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565354 5922 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 12:52:23.570622 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565357 5922 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 12:52:23.570622 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565361 5922 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 12:52:23.570622 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565364 5922 flags.go:64] FLAG: --cgroup-root="" Oct 30 12:52:23.570622 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565366 5922 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 12:52:23.570622 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565369 5922 flags.go:64] FLAG: --client-ca-file="" Oct 30 12:52:23.570622 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565373 5922 flags.go:64] FLAG: --cloud-provider="external" Oct 30 12:52:23.571158 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565376 5922 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 12:52:23.571158 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565381 5922 flags.go:64] FLAG: --cluster-domain="" Oct 30 12:52:23.571158 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565385 5922 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 12:52:23.571158 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565388 5922 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 12:52:23.571158 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565391 5922 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 12:52:23.571158 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565395 5922 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 12:52:23.571158 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565400 5922 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 12:52:23.571158 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565403 5922 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 12:52:23.571158 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565406 5922 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 12:52:23.571158 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565409 5922 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 12:52:23.571158 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565412 5922 flags.go:64] FLAG: --contention-profiling="false" Oct 30 12:52:23.571158 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565415 5922 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 12:52:23.571158 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565418 5922 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 12:52:23.571158 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565421 5922 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 12:52:23.571158 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565424 5922 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 12:52:23.571158 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565428 5922 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 12:52:23.571158 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565432 5922 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 12:52:23.571158 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565434 5922 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 12:52:23.571158 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565437 5922 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 12:52:23.571158 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565440 5922 flags.go:64] FLAG: --enable-server="true" Oct 30 12:52:23.571158 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565443 5922 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 12:52:23.571158 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565447 5922 flags.go:64] FLAG: --event-burst="100" Oct 30 12:52:23.571158 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565450 5922 flags.go:64] FLAG: --event-qps="50" Oct 30 12:52:23.571158 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565454 5922 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 12:52:23.571741 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565457 5922 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 12:52:23.571741 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565462 5922 flags.go:64] FLAG: --eviction-hard="" Oct 30 12:52:23.571741 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565466 5922 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 12:52:23.571741 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565469 5922 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 12:52:23.571741 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565472 5922 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 12:52:23.571741 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565475 5922 flags.go:64] FLAG: --eviction-soft="" Oct 30 12:52:23.571741 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565478 5922 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 12:52:23.571741 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565481 5922 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 12:52:23.571741 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565484 5922 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 12:52:23.571741 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565487 5922 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 12:52:23.571741 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565490 5922 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 12:52:23.571741 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565493 5922 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 12:52:23.571741 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565496 5922 flags.go:64] FLAG: --feature-gates="" Oct 30 12:52:23.571741 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565500 5922 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 12:52:23.571741 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565504 5922 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 12:52:23.571741 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565507 5922 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 12:52:23.571741 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565510 5922 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 12:52:23.571741 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565513 5922 flags.go:64] FLAG: --healthz-port="10248" Oct 30 12:52:23.571741 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565516 5922 flags.go:64] FLAG: --help="false" Oct 30 12:52:23.571741 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565520 5922 flags.go:64] FLAG: --hostname-override="ip-10-0-137-21.ec2.internal" Oct 30 12:52:23.571741 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565523 5922 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 12:52:23.571741 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565526 5922 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 12:52:23.571741 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565529 5922 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 12:52:23.572262 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565532 5922 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 12:52:23.572262 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565535 5922 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 12:52:23.572262 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565538 5922 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 12:52:23.572262 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565541 5922 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 12:52:23.572262 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565544 5922 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 12:52:23.572262 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565547 5922 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 12:52:23.572262 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565549 5922 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 12:52:23.572262 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565552 5922 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 12:52:23.572262 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565555 5922 flags.go:64] FLAG: --kube-reserved="" Oct 30 12:52:23.572262 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565558 5922 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 12:52:23.572262 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565562 5922 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 12:52:23.572262 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565565 5922 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 12:52:23.572262 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565567 5922 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 12:52:23.572262 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565570 5922 flags.go:64] FLAG: --lock-file="" Oct 30 12:52:23.572262 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565573 5922 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 12:52:23.572262 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565576 5922 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 12:52:23.572262 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565579 5922 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 12:52:23.572262 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565584 5922 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 12:52:23.572262 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565587 5922 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 12:52:23.572262 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565590 5922 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 12:52:23.572262 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565593 5922 flags.go:64] FLAG: --logging-format="text" Oct 30 12:52:23.572262 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565596 5922 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 12:52:23.572262 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565599 5922 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 12:52:23.572262 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565603 5922 flags.go:64] FLAG: --manifest-url="" Oct 30 12:52:23.572883 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565606 5922 flags.go:64] FLAG: --manifest-url-header="" Oct 30 12:52:23.572883 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565623 5922 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 12:52:23.572883 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565626 5922 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 12:52:23.572883 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565630 5922 flags.go:64] FLAG: --max-pods="110" Oct 30 12:52:23.572883 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565633 5922 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 12:52:23.572883 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565636 5922 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 12:52:23.572883 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565639 5922 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 12:52:23.572883 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565642 5922 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 12:52:23.572883 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565645 5922 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 12:52:23.572883 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565648 5922 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 12:52:23.572883 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565651 5922 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 12:52:23.572883 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565658 5922 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 12:52:23.572883 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565661 5922 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 12:52:23.572883 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565664 5922 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 12:52:23.572883 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565667 5922 flags.go:64] FLAG: --pod-cidr="" Oct 30 12:52:23.572883 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565670 5922 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 12:52:23.572883 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565673 5922 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 12:52:23.572883 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565676 5922 flags.go:64] FLAG: --pods-per-core="0" Oct 30 12:52:23.572883 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565679 5922 flags.go:64] FLAG: --port="10250" Oct 30 12:52:23.572883 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565682 5922 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 12:52:23.572883 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565685 5922 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-057450241d331bc81" Oct 30 12:52:23.572883 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565688 5922 flags.go:64] FLAG: --qos-reserved="" Oct 30 12:52:23.572883 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565691 5922 flags.go:64] FLAG: --read-only-port="10255" Oct 30 12:52:23.572883 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565694 5922 flags.go:64] FLAG: --register-node="true" Oct 30 12:52:23.572883 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565697 5922 flags.go:64] FLAG: --register-with-taints="" Oct 30 12:52:23.572883 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565700 5922 flags.go:64] FLAG: --registry-burst="10" Oct 30 12:52:23.573733 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565703 5922 flags.go:64] FLAG: --registry-qps="5" Oct 30 12:52:23.573733 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565706 5922 flags.go:64] FLAG: --reserved-cpus="" Oct 30 12:52:23.573733 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565709 5922 flags.go:64] FLAG: --reserved-memory="" Oct 30 12:52:23.573733 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565713 5922 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 12:52:23.573733 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565716 5922 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 12:52:23.573733 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565719 5922 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 12:52:23.573733 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565722 5922 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 12:52:23.573733 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565726 5922 flags.go:64] FLAG: --runonce="false" Oct 30 12:52:23.573733 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565729 5922 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 12:52:23.573733 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565732 5922 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 12:52:23.573733 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565735 5922 flags.go:64] FLAG: --seccomp-default="false" Oct 30 12:52:23.573733 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565737 5922 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 12:52:23.573733 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565740 5922 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 12:52:23.573733 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565743 5922 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 12:52:23.573733 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565746 5922 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 12:52:23.573733 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565749 5922 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 12:52:23.573733 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565752 5922 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 12:52:23.573733 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565755 5922 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 12:52:23.573733 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565758 5922 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 12:52:23.573733 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565761 5922 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 12:52:23.573733 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565764 5922 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 12:52:23.573733 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565767 5922 flags.go:64] FLAG: --system-cgroups="" Oct 30 12:52:23.573733 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565770 5922 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 12:52:23.573733 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565775 5922 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 12:52:23.573733 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565778 5922 flags.go:64] FLAG: --tls-cert-file="" Oct 30 12:52:23.574735 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565781 5922 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 12:52:23.574735 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565785 5922 flags.go:64] FLAG: --tls-min-version="" Oct 30 12:52:23.574735 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565788 5922 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 12:52:23.574735 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565791 5922 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 12:52:23.574735 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565794 5922 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 12:52:23.574735 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565796 5922 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 12:52:23.574735 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565799 5922 flags.go:64] FLAG: --v="2" Oct 30 12:52:23.574735 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565803 5922 flags.go:64] FLAG: --version="false" Oct 30 12:52:23.574735 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565807 5922 flags.go:64] FLAG: --vmodule="" Oct 30 12:52:23.574735 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565811 5922 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 12:52:23.574735 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.565814 5922 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 12:52:23.574735 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565937 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:52:23.574735 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565942 5922 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:52:23.574735 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565946 5922 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:52:23.574735 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565950 5922 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:52:23.574735 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565953 5922 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:52:23.574735 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565955 5922 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:52:23.574735 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565958 5922 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:52:23.574735 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565961 5922 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:52:23.574735 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565963 5922 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:52:23.574735 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565966 5922 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:52:23.574735 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565968 5922 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:52:23.575298 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565971 5922 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:52:23.575298 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565974 5922 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:52:23.575298 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565976 5922 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:52:23.575298 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565979 5922 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:52:23.575298 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565981 5922 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:52:23.575298 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565984 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:52:23.575298 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565986 5922 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:52:23.575298 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565991 5922 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:52:23.575298 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565993 5922 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:52:23.575298 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565996 5922 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:52:23.575298 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.565998 5922 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:52:23.575298 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566001 5922 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:52:23.575298 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566004 5922 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:52:23.575298 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566006 5922 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:52:23.575298 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566009 5922 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:52:23.575298 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566012 5922 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:52:23.575298 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566014 5922 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:52:23.575298 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566017 5922 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:52:23.575298 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566019 5922 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:52:23.575298 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566022 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:52:23.575795 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566024 5922 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:52:23.575795 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566027 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:52:23.575795 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566030 5922 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:52:23.575795 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566032 5922 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:52:23.575795 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566036 5922 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:52:23.575795 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566039 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:52:23.575795 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566042 5922 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:52:23.575795 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566045 5922 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:52:23.575795 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566047 5922 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:52:23.575795 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566050 5922 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:52:23.575795 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566053 5922 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:52:23.575795 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566055 5922 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:52:23.575795 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566058 5922 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:52:23.575795 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566060 5922 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:52:23.575795 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566063 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:52:23.575795 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566065 5922 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:52:23.575795 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566068 5922 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:52:23.575795 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566070 5922 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:52:23.575795 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566073 5922 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:52:23.576339 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566077 5922 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:52:23.576339 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566079 5922 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:52:23.576339 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566082 5922 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:52:23.576339 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566085 5922 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:52:23.576339 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566087 5922 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:52:23.576339 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566090 5922 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:52:23.576339 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566093 5922 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:52:23.576339 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566095 5922 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:52:23.576339 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566098 5922 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:52:23.576339 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566100 5922 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:52:23.576339 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566103 5922 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:52:23.576339 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566105 5922 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:52:23.576339 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566108 5922 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:52:23.576339 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566110 5922 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:52:23.576339 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566113 5922 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:52:23.576339 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566116 5922 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:52:23.576339 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566118 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:52:23.576339 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566122 5922 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:52:23.576339 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566125 5922 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:52:23.576339 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566128 5922 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:52:23.576930 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566131 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:52:23.576930 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566133 5922 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:52:23.576930 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566136 5922 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:52:23.576930 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566138 5922 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:52:23.576930 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566141 5922 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:52:23.576930 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566143 5922 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:52:23.576930 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566146 5922 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:52:23.576930 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566148 5922 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:52:23.576930 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566151 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:52:23.576930 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566154 5922 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:52:23.576930 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566156 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:52:23.576930 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566159 5922 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:52:23.576930 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566163 5922 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:52:23.576930 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566166 5922 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:52:23.576930 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566169 5922 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:52:23.576930 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566173 5922 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:52:23.576930 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566176 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:52:23.576930 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566178 5922 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:52:23.576930 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566181 5922 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:52:23.577489 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566183 5922 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:52:23.577489 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566186 5922 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:52:23.577489 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566189 5922 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:52:23.577489 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566192 5922 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:52:23.577489 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566194 5922 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:52:23.577489 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566196 5922 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:52:23.577489 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566199 5922 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:52:23.577489 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566202 5922 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:52:23.577489 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566204 5922 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:52:23.577489 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566207 5922 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:52:23.577489 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566210 5922 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:52:23.577489 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566213 5922 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:52:23.577489 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566216 5922 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:52:23.577489 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566219 5922 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:52:23.577489 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566221 5922 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:52:23.577489 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566224 5922 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:52:23.577489 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566226 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:52:23.578371 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.566251 5922 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 12:52:23.578371 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566289 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:52:23.578371 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566294 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:52:23.578371 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566298 5922 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:52:23.578371 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566302 5922 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:52:23.578371 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566305 5922 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:52:23.578371 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566307 5922 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:52:23.578371 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566310 5922 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:52:23.578371 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566312 5922 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:52:23.578371 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566317 5922 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:52:23.578371 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566320 5922 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:52:23.578371 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566323 5922 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:52:23.578371 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566325 5922 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:52:23.578371 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566328 5922 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:52:23.578371 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566331 5922 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:52:23.579040 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566333 5922 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:52:23.579040 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566336 5922 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:52:23.579040 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566338 5922 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:52:23.579040 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566341 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:52:23.579040 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566344 5922 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:52:23.579040 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566346 5922 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:52:23.579040 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566349 5922 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:52:23.579040 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566351 5922 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:52:23.579040 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566354 5922 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:52:23.579040 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566356 5922 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:52:23.579040 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566360 5922 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:52:23.579040 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566362 5922 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:52:23.579040 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566365 5922 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:52:23.579040 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566368 5922 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:52:23.579040 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566370 5922 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:52:23.579040 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566373 5922 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:52:23.579040 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566376 5922 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:52:23.579040 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566378 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:52:23.579040 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566381 5922 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:52:23.579040 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566384 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:52:23.579679 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566386 5922 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:52:23.579679 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566389 5922 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:52:23.579679 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566391 5922 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:52:23.579679 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566394 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:52:23.579679 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566396 5922 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:52:23.579679 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566399 5922 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:52:23.579679 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566403 5922 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:52:23.579679 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566406 5922 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:52:23.579679 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566408 5922 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:52:23.579679 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566411 5922 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:52:23.579679 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566414 5922 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:52:23.579679 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566416 5922 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:52:23.579679 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566419 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:52:23.579679 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566421 5922 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:52:23.579679 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566424 5922 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:52:23.579679 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566426 5922 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:52:23.579679 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566429 5922 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:52:23.579679 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566432 5922 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:52:23.579679 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566434 5922 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:52:23.580204 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566437 5922 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:52:23.580204 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566440 5922 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:52:23.580204 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566442 5922 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:52:23.580204 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566446 5922 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:52:23.580204 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566448 5922 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:52:23.580204 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566451 5922 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:52:23.580204 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566454 5922 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:52:23.580204 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566456 5922 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:52:23.580204 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566459 5922 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:52:23.580204 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566462 5922 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:52:23.580204 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566464 5922 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:52:23.580204 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566467 5922 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:52:23.580204 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566470 5922 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:52:23.580204 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566472 5922 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:52:23.580204 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566475 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:52:23.580204 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566477 5922 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:52:23.580204 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566480 5922 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:52:23.580204 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566482 5922 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:52:23.580204 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566485 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:52:23.580204 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566491 5922 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:52:23.580742 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566493 5922 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:52:23.580742 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566496 5922 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:52:23.580742 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566498 5922 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:52:23.580742 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566501 5922 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:52:23.580742 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566503 5922 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:52:23.580742 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566506 5922 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:52:23.580742 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566509 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:52:23.580742 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566511 5922 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:52:23.580742 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566514 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:52:23.580742 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566517 5922 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:52:23.580742 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566519 5922 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:52:23.580742 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566522 5922 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:52:23.580742 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566525 5922 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:52:23.580742 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566529 5922 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:52:23.580742 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566532 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:52:23.580742 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566536 5922 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:52:23.580742 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566540 5922 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:52:23.580742 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566543 5922 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:52:23.580742 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566545 5922 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:52:23.581190 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566548 5922 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:52:23.581190 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566551 5922 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:52:23.581190 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566553 5922 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:52:23.581190 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566556 5922 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:52:23.581190 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566559 5922 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:52:23.581190 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566561 5922 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:52:23.581190 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566564 5922 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:52:23.581190 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566566 5922 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:52:23.581190 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566569 5922 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:52:23.581190 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566572 5922 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:52:23.581190 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566575 5922 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:52:23.581190 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566578 5922 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:52:23.581190 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566583 5922 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:52:23.581190 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.566585 5922 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:52:23.581190 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.570095 5922 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 12:52:23.581190 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.570109 5922 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 12:52:23.581190 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.570126 5922 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 12:52:23.581190 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.570131 5922 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 12:52:23.581190 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570190 5922 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:52:23.581190 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570195 5922 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:52:23.581698 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570199 5922 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:52:23.581698 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570202 5922 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:52:23.581698 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570205 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:52:23.581698 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570207 5922 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:52:23.581698 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570211 5922 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:52:23.581698 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570213 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:52:23.581698 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570216 5922 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:52:23.581698 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570219 5922 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:52:23.581698 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570222 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:52:23.581698 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570226 5922 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:52:23.581698 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570228 5922 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:52:23.581698 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570231 5922 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:52:23.581698 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570233 5922 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:52:23.581698 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570236 5922 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:52:23.581698 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570243 5922 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:52:23.581698 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570246 5922 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:52:23.581698 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570249 5922 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:52:23.581698 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570251 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:52:23.581698 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570254 5922 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:52:23.582203 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570257 5922 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:52:23.582203 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570259 5922 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:52:23.582203 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570262 5922 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:52:23.582203 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570265 5922 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:52:23.582203 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570267 5922 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:52:23.582203 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570270 5922 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:52:23.582203 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570273 5922 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:52:23.582203 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570275 5922 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:52:23.582203 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570278 5922 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:52:23.582203 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570280 5922 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:52:23.582203 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570283 5922 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:52:23.582203 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570285 5922 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:52:23.582203 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570288 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:52:23.582203 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570290 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:52:23.582203 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570293 5922 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:52:23.582203 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570295 5922 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:52:23.582203 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570298 5922 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:52:23.582203 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570301 5922 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:52:23.582203 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570303 5922 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:52:23.582717 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570306 5922 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:52:23.582717 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570308 5922 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:52:23.582717 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570311 5922 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:52:23.582717 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570314 5922 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:52:23.582717 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570316 5922 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:52:23.582717 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570319 5922 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:52:23.582717 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570321 5922 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:52:23.582717 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570325 5922 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:52:23.582717 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570329 5922 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:52:23.582717 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570336 5922 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:52:23.582717 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570339 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:52:23.582717 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570342 5922 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:52:23.582717 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570345 5922 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:52:23.582717 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570347 5922 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:52:23.582717 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570350 5922 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:52:23.582717 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570353 5922 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:52:23.582717 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570355 5922 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:52:23.582717 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570358 5922 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:52:23.582717 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570361 5922 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:52:23.582717 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570363 5922 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:52:23.583243 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570365 5922 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:52:23.583243 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570368 5922 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:52:23.583243 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570371 5922 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:52:23.583243 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570373 5922 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:52:23.583243 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570376 5922 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:52:23.583243 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570378 5922 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:52:23.583243 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570381 5922 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:52:23.583243 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570383 5922 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:52:23.583243 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570386 5922 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:52:23.583243 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570388 5922 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:52:23.583243 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570391 5922 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:52:23.583243 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570393 5922 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:52:23.583243 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570398 5922 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:52:23.583243 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570402 5922 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:52:23.583243 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570405 5922 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:52:23.583243 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570408 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:52:23.583243 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570410 5922 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:52:23.583243 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570413 5922 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:52:23.583243 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570416 5922 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:52:23.583743 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570418 5922 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:52:23.583743 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570421 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:52:23.583743 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570423 5922 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:52:23.583743 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570431 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:52:23.583743 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570434 5922 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:52:23.583743 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570436 5922 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:52:23.583743 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570438 5922 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:52:23.583743 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570441 5922 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:52:23.583743 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570443 5922 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:52:23.583743 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570446 5922 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:52:23.583743 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570448 5922 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:52:23.583743 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570451 5922 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:52:23.583743 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570454 5922 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:52:23.583743 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570456 5922 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:52:23.583743 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570458 5922 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:52:23.583743 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570461 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:52:23.583743 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570463 5922 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:52:23.583743 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570466 5922 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:52:23.583743 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570468 5922 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:52:23.583743 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570471 5922 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:52:23.584239 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570474 5922 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:52:23.584239 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570476 5922 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:52:23.584239 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570479 5922 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:52:23.584239 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570481 5922 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:52:23.584239 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570484 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:52:23.584239 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570486 5922 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:52:23.584239 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570489 5922 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:52:23.584239 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.570516 5922 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 12:52:23.584239 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570709 5922 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:52:23.584239 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570714 5922 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:52:23.584239 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570717 5922 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:52:23.584239 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570720 5922 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:52:23.584239 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570723 5922 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:52:23.584239 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570725 5922 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:52:23.584239 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570728 5922 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:52:23.584239 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570731 5922 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:52:23.584629 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570740 5922 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:52:23.584629 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570743 5922 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:52:23.584629 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570745 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:52:23.584629 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570748 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:52:23.584629 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570751 5922 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:52:23.584629 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570753 5922 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:52:23.584629 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570756 5922 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:52:23.584629 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570758 5922 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:52:23.584629 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570761 5922 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:52:23.584629 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570764 5922 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:52:23.584629 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570767 5922 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:52:23.584629 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570769 5922 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:52:23.584629 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570772 5922 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:52:23.584629 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570775 5922 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:52:23.584629 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570777 5922 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:52:23.584629 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570780 5922 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:52:23.584629 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570782 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:52:23.584629 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570785 5922 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:52:23.584629 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570788 5922 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:52:23.584629 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570790 5922 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:52:23.585116 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570794 5922 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:52:23.585116 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570800 5922 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:52:23.585116 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570804 5922 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:52:23.585116 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570807 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:52:23.585116 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570810 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:52:23.585116 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570812 5922 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:52:23.585116 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570815 5922 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:52:23.585116 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570818 5922 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:52:23.585116 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570820 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:52:23.585116 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570823 5922 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:52:23.585116 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570825 5922 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:52:23.585116 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570828 5922 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:52:23.585116 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570830 5922 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:52:23.585116 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570837 5922 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:52:23.585116 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570840 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:52:23.585116 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570843 5922 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:52:23.585116 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570845 5922 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:52:23.585116 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570848 5922 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:52:23.585116 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570851 5922 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:52:23.585607 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570853 5922 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:52:23.585607 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570857 5922 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:52:23.585607 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570860 5922 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:52:23.585607 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570863 5922 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:52:23.585607 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570865 5922 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:52:23.585607 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570868 5922 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:52:23.585607 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570870 5922 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:52:23.585607 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570873 5922 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:52:23.585607 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570876 5922 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:52:23.585607 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570878 5922 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:52:23.585607 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570881 5922 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:52:23.585607 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570888 5922 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:52:23.585607 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570891 5922 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:52:23.585607 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570894 5922 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:52:23.585607 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570898 5922 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:52:23.585607 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570901 5922 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:52:23.585607 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570903 5922 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:52:23.585607 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570906 5922 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:52:23.585607 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570908 5922 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:52:23.585607 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570911 5922 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:52:23.586112 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570913 5922 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:52:23.586112 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570916 5922 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:52:23.586112 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570918 5922 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:52:23.586112 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570921 5922 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:52:23.586112 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570923 5922 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:52:23.586112 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570926 5922 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:52:23.586112 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570928 5922 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:52:23.586112 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570936 5922 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:52:23.586112 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570939 5922 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:52:23.586112 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570941 5922 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:52:23.586112 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570944 5922 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:52:23.586112 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570946 5922 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:52:23.586112 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570949 5922 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:52:23.586112 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570953 5922 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:52:23.586112 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570955 5922 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:52:23.586112 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570958 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:52:23.586112 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570961 5922 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:52:23.586112 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570963 5922 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:52:23.586112 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570966 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:52:23.586620 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570969 5922 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:52:23.586620 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570971 5922 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:52:23.586620 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570974 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:52:23.586620 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570976 5922 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:52:23.586620 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570979 5922 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:52:23.586620 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570981 5922 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:52:23.586620 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570985 5922 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:52:23.586620 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570991 5922 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:52:23.586620 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570993 5922 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:52:23.586620 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570996 5922 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:52:23.586620 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.570999 5922 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:52:23.586620 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571001 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:52:23.586620 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571004 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:52:23.586620 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571006 5922 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:52:23.586620 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571009 5922 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:52:23.586620 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571012 5922 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:52:23.586620 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571014 5922 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:52:23.586620 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571017 5922 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:52:23.586620 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571019 5922 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:52:23.587078 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571022 5922 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:52:23.587078 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.571044 5922 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 12:52:23.587078 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571095 5922 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:52:23.587078 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571099 5922 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:52:23.587078 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571102 5922 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:52:23.587078 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571105 5922 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:52:23.587078 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571108 5922 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:52:23.587078 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571110 5922 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:52:23.587078 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571114 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:52:23.587078 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571117 5922 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:52:23.587078 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571119 5922 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:52:23.587078 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571122 5922 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:52:23.587078 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571126 5922 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:52:23.587078 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571130 5922 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:52:23.587078 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571133 5922 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:52:23.587439 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571136 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:52:23.587439 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571139 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:52:23.587439 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571141 5922 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:52:23.587439 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571144 5922 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:52:23.587439 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571147 5922 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:52:23.587439 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571151 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:52:23.587439 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571154 5922 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:52:23.587439 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571156 5922 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:52:23.587439 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571159 5922 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:52:23.587439 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571162 5922 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:52:23.587439 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571165 5922 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:52:23.587439 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571167 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:52:23.587439 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571170 5922 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:52:23.587439 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571173 5922 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:52:23.587439 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571176 5922 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:52:23.587439 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571178 5922 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:52:23.587439 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571181 5922 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:52:23.587439 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571183 5922 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:52:23.587439 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571186 5922 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:52:23.587946 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571188 5922 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:52:23.587946 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571196 5922 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:52:23.587946 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571198 5922 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:52:23.587946 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571201 5922 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:52:23.587946 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571204 5922 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:52:23.587946 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571206 5922 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:52:23.587946 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571210 5922 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:52:23.587946 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571213 5922 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:52:23.587946 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571216 5922 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:52:23.587946 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571218 5922 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:52:23.587946 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571222 5922 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:52:23.587946 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571224 5922 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:52:23.587946 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571227 5922 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:52:23.587946 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571229 5922 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:52:23.587946 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571232 5922 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:52:23.587946 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571234 5922 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:52:23.587946 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571237 5922 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:52:23.587946 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571240 5922 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:52:23.587946 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571243 5922 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:52:23.587946 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571246 5922 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:52:23.588446 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571249 5922 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:52:23.588446 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571251 5922 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:52:23.588446 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571254 5922 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:52:23.588446 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571256 5922 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:52:23.588446 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571259 5922 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:52:23.588446 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571261 5922 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:52:23.588446 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571264 5922 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:52:23.588446 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571266 5922 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:52:23.588446 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571269 5922 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:52:23.588446 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571271 5922 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:52:23.588446 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571274 5922 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:52:23.588446 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571276 5922 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:52:23.588446 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571279 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:52:23.588446 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571281 5922 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:52:23.588446 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571289 5922 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:52:23.588446 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571291 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:52:23.588446 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571294 5922 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:52:23.588446 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571297 5922 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:52:23.588446 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571301 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:52:23.588446 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571304 5922 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:52:23.588947 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571307 5922 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:52:23.588947 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571309 5922 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:52:23.588947 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571313 5922 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:52:23.588947 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571317 5922 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:52:23.588947 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571321 5922 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:52:23.588947 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571325 5922 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:52:23.588947 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571328 5922 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:52:23.588947 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571331 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:52:23.588947 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571334 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:52:23.588947 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571337 5922 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:52:23.588947 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571340 5922 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:52:23.588947 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571343 5922 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:52:23.588947 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571346 5922 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:52:23.588947 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571349 5922 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:52:23.588947 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571352 5922 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:52:23.588947 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571354 5922 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:52:23.588947 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571357 5922 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:52:23.588947 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571360 5922 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:52:23.588947 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571362 5922 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:52:23.589400 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571365 5922 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:52:23.589400 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571367 5922 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:52:23.589400 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571370 5922 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:52:23.589400 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571372 5922 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:52:23.589400 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571375 5922 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:52:23.589400 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571377 5922 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:52:23.589400 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571380 5922 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:52:23.589400 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571383 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:52:23.589400 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571390 5922 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:52:23.589400 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571393 5922 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:52:23.589400 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571395 5922 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:52:23.589400 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571400 5922 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:52:23.589400 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571402 5922 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:52:23.589400 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571405 5922 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:52:23.589400 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:23.571407 5922 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:52:23.589400 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.571459 5922 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 12:52:23.589400 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.572404 5922 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 12:52:23.589400 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.572452 5922 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 12:52:23.589400 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.572660 5922 server.go:1014] "Starting client certificate rotation" Oct 30 12:52:23.589903 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.572837 5922 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 12:52:23.589903 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.572905 5922 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 12:46:12 +0000 UTC" deadline="2028-05-02 18:25:28.268928383 +0000 UTC" Oct 30 12:52:23.589903 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.572972 5922 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13205h33m4.695964652s" Oct 30 12:52:23.589903 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.573213 5922 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 12:52:23.589903 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.575653 5922 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 12:52:23.589903 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.576782 5922 log.go:25] "Validated CRI v1 runtime API" Oct 30 12:52:23.589903 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.579220 5922 log.go:25] "Validated CRI v1 image API" Oct 30 12:52:23.589903 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.579910 5922 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 12:52:23.589903 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.582060 5922 fs.go:135] Filesystem UUIDs: map[5bcae973-a38e-4b89-9185-ba40144ffa06:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e8189750-30b2-4de3-9a9b-c16d3fe68d8c:/dev/nvme0n1p3] Oct 30 12:52:23.590152 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.582081 5922 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/00f70bf3b3b7e70a68aa7921812aeb917267fc7aa4ee2483a0e6a458e7e26f6d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/00f70bf3b3b7e70a68aa7921812aeb917267fc7aa4ee2483a0e6a458e7e26f6d/userdata/shm major:0 minor:363 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0647e9ff213d9818f0c5cc4f86bd064d5d41a6318831f00fb5d98ca6c711629e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0647e9ff213d9818f0c5cc4f86bd064d5d41a6318831f00fb5d98ca6c711629e/userdata/shm major:0 minor:482 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/07156ce7e70955a73bee391bd2ecbcc00edb79c18b55d20af84579c41d048044/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/07156ce7e70955a73bee391bd2ecbcc00edb79c18b55d20af84579c41d048044/userdata/shm major:0 minor:359 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/076576ce8f705e376e3e20d5ba028afd9046651ed42cefcc0e1400fc95ef000a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/076576ce8f705e376e3e20d5ba028afd9046651ed42cefcc0e1400fc95ef000a/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0fb70e7381decaff50290b92ee2c0505239b354e82ff99bf84e63be1e937c006/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0fb70e7381decaff50290b92ee2c0505239b354e82ff99bf84e63be1e937c006/userdata/shm major:0 minor:278 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1697328fc8c5672d44ef75b4e614af0feacc2e137b5e29cc32a0554ac92829eb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1697328fc8c5672d44ef75b4e614af0feacc2e137b5e29cc32a0554ac92829eb/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1e3b60100d2344794fcb00673508fd88209fb2fba72fe2ce48906b9a978dd3a7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1e3b60100d2344794fcb00673508fd88209fb2fba72fe2ce48906b9a978dd3a7/userdata/shm major:0 minor:381 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2d03adb62aeda4a8cc61bb3dca86865ed63bd63ff6e3b7d346b3c6d438fcd903/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2d03adb62aeda4a8cc61bb3dca86865ed63bd63ff6e3b7d346b3c6d438fcd903/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3410ea027ea71f3e80e819cc87d3a28cf98bce131de13e4a81f8b081e6458cae/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3410ea027ea71f3e80e819cc87d3a28cf98bce131de13e4a81f8b081e6458cae/userdata/shm major:0 minor:413 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3fed4127a265b40e1040260f5bbf5bcc0d93021fbf60ba8b2dac44897c94681f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3fed4127a265b40e1040260f5bbf5bcc0d93021fbf60ba8b2dac44897c94681f/userdata/shm major:0 minor:467 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4407c6e829fd9619dbbaeb700be6308ccf26f6335725731cf1d5ddd53c2c6ad3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4407c6e829fd9619dbbaeb700be6308ccf26f6335725731cf1d5ddd53c2c6ad3/userdata/shm major:0 minor:249 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/52495134362f8b43db53d257786d659819f2329d5a13f341d1b9c0cc4a20619b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/52495134362f8b43db53d257786d659819f2329d5a13f341d1b9c0cc4a20619b/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/58287815249aca0628fe9cfd989113fcd3966286481e51121bf107cd4599d14f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/58287815249aca0628fe9cfd989113fcd3966286481e51121bf107cd4599d14f/userdata/shm major:0 minor:292 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/64493cb387412ea7ebe03ce195389352c6b410c126810317e7a7cef26cb9b066/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/64493cb387412ea7ebe03ce195389352c6b410c126810317e7a7cef26cb9b066/userdata/shm major:0 minor:273 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6bc842612ded17a701978fb17fb8dace799d5e75552b5c8b0104dd3c213b4857/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6bc842612ded17a701978fb17fb8dace799d5e75552b5c8b0104dd3c213b4857/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6ff02a1f79b7714a3c55113244bdf4c133dd72421d6929105f6298f05ccb7636/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6ff02a1f79b7714a3c55113244bdf4c133dd72421d6929105f6298f05ccb7636/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/815dcea6522fbada33cae727dbd21ebe8040c2dced78a4d7324e4652362ad187/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/815dcea6522fbada33cae727dbd21ebe8040c2dced78a4d7324e4652362ad187/userdata/shm major:0 minor:315 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8b0ab6dfec3ac6c49e388269cf07f3ee06fb4d7ee5f351155e4ccf3f9e88f43a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8b0ab6dfec3ac6c49e388269cf07f3ee06fb4d7ee5f351155e4ccf3f9e88f43a/userdata/shm major:0 minor:471 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8b99f37e6db72b8da01b867c94cb74058eb342a3d1d037fbcbd4dcf00672bdbf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8b99f37e6db72b8da01b867c94cb74058eb342a3d1d037fbcbd4dcf00672bdbf/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/92483d9ab9d6b9c70a1cf111febac24212e30ce539406891994098559142f86b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/92483d9ab9d6b9c70a1cf111febac24212e30ce539406891994098559142f86b/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a76e4cca2d9ac374f3d281c532a10ca542230fb3203096c2b2727d6e57f2f39a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a76e4cca2d9ac374f3d281c532a10ca542230fb3203096c2b2727d6e57f2f39a/userdata/shm major:0 minor:469 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ae4a53261c8f2d08d8af9130166c20d999ab7bf0ed529f7d154cbc0ae514e6ed/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ae4a53261c8f2d08d8af9130166c20d999ab7bf0ed529f7d154cbc0ae514e6ed/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b72d684ebf8e4040794f375efa08178308d583211935ee32ab65d99a3a74d8e6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b72d684ebf8e4040794f375efa08178308d583211935ee32ab65d99a3a74d8e6/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c8edd7fed57efb3f2fe9eb4b5689338b4c619b94606f3223fac094499fd99a54/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c8edd7fed57efb3f2fe9eb4b5689338b4c619b94606f3223fac094499fd99a54/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cfaf6eb0e32f16cf0289059ab69c1e2d5690928fa9d7a16ea1a6cecf114b053c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cfaf6eb0e32f16cf0289059ab69c1e2d5690928fa9d7a16ea1a6cecf114b053c/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f2508189ec0d29ac883e4c6404a709ef657cdf80440bf5200b0fc427fa4f210d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f2508189ec0d29ac883e4c6404a709ef657cdf80440bf5200b0fc427fa4f210d/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01873344-73c8-47b2-b6ba-8e338ba18315/volumes/kubernetes.io~projected/kube-api-access-2w592:{mountpoint:/var/lib/kubelet/pods/01873344-73c8-47b2-b6ba-8e338ba18315/volumes/kubernetes.io~projected/kube-api-access-2w592 major:0 minor:479 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01873344-73c8-47b2-b6ba-8e338ba18315/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/01873344-73c8-47b2-b6ba-8e338ba18315/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:473 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01873344-73c8-47b2-b6ba-8e338ba18315/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/01873344-73c8-47b2-b6ba-8e338ba18315/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:476 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11a0b508-2f9c-4be6-b5c7-b3d74766e801/volumes/kubernetes.io~projected/kube-api-access-tkldf:{mountpoint:/var/lib/kubelet/pods/11a0b508-2f9c-4be6-b5c7-b3d74766e801/volumes/kubernetes.io~projected/kube-api-access-tkldf major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14c0845c-1947-44b0-ada5-4b68ff9974f2/volumes/kubernetes.io~projected/kube-api-access-wvdzg:{mountpoint:/var/lib/kubelet/pods/14c0845c-1947-44b0-ada5-4b68ff9974f2/volumes/kubernetes.io~projected/kube-api-access-wvdzg major:0 minor:362 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14c0845c-1947-44b0-ada5-4b68ff9974f2/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/14c0845c-1947-44b0-ada5-4b68ff9974f2/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1bb48bfa-827b-4179-b9fa-418fc14f0f44/volumes/kubernetes.io~projected/kube-api-access-dqvdp:{mountpoint:/var/lib/kubelet/pods/1bb48bfa-827b-4179-b9fa-418fc14f0f44/volumes/kubernetes.io~projected/kube-api-access-dqvdp major:0 minor:466 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1fcccc7b-a83c-4c52-a648-cb549e70b54d/volumes/kubernetes.io~projected/kube-api-access-xxwlx:{mountpoint:/var/lib/kubelet/pods/1fcccc7b-a83c-4c52-a648-cb549e70b54d/volumes/kubernetes.io~projected/kube-api-access-xxwlx major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1fcccc7b-a83c-4c52-a648-cb549e70b54d/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/1fcccc7b-a83c-4c52-a648-cb549e70b54d/volumes/kubernetes.io~secret/metrics-certs major:0 minor:465 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f38af1e-cd92-4cd8-a643-151b10b8da12/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/3f38af1e-cd92-4cd8-a643-151b10b8da12/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4dca982f-f509-454b-a057-d7fd9b3f34bd/volumes/kubernetes.io~projected/kube-api-access-5m944:{mountpoint:/var/lib/kubelet/pods/4dca982f-f509-454b-a057-d7fd9b3f34bd/volumes/kubernetes.io~projected/kube-api-access-5m944 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4dca982f-f509-454b-a057-d7fd9b3f34bd/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/4dca982f-f509-454b-a057-d7fd9b3f34bd/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50bec85b-5592-437d-aa6f-4899aeb494a5/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/50bec85b-5592-437d-aa6f-4899aeb494a5/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:461 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51a542f5-796e-46b1-95bd-31751d50b52b/volumes/kubernetes.io~projected/kube-api-access-tvf5p:{mountpoint:/var/lib/kubelet/pods/51a542f5-796e-46b1-95bd-31751d50b52b/volumes/kubernetes.io~projected/kube-api-access-tvf5p major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51a542f5-796e-46b1-95bd-31751d50b52b/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/51a542f5-796e-46b1-95bd-31751d50b52b/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56928a4d-7923-46e6-97bb-6690514f89a4/volumes/kubernetes.io~projected/kube-api-access-9v6nm:{mountpoint:/var/lib/kubelet/pods/56928a4d-7923-46e6-97bb-6690514f89a4/volumes/kubernetes.io~projected/kube-api-access-9v6nm major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56928a4d-7923-46e6-97bb-6690514f89a4/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/56928a4d-7923-46e6-97bb-6690514f89a4/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56928a4d-7923-46e6-97bb-6690514f89a4/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/56928a4d-7923-46e6-97bb-6690514f89a4/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:245 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66172ef8-aaad-4875-b3b3-bd81b9e9ba59/volumes/kubernetes.io~projected/kube-api-access-wvxmx:{mountpoint:/var/lib/kubelet/pods/66172ef8-aaad-4875-b3b3-bd81b9e9ba59/volumes/kubernetes.io~projected/kube-api-access-wvxmx major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fd0fdb6-c254-446d-8832-5a2fb284f03d/volumes/kubernetes.io~projected/kube-api-access-27dm6:{mountpoint:/var/lib/kubelet/pods/8fd0fdb6-c254-446d-8832-5a2fb284f03d/volumes/kubernetes.io~projected/kube-api-access-27dm6 major:0 minor:271 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fd0fdb6-c254-446d-8832-5a2fb284f03d/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/8fd0fdb6-c254-446d-8832-5a2fb284f03d/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fd0fdb6-c254-446d-8832-5a2fb284f03d/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/8fd0fdb6-c254-446d-8832-5a2fb284f03d/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab451161-af25-47e2-860e-f7d5186d4491/volumes/kubernetes.io~projected/kube-api-access-gbr4h:{mountpoint:/var/lib/kubelet/pods/ab451161-af25-47e2-860e-f7d5186d4491/volumes/kubernetes.io~projected/kube-api-access-gbr4h major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab451161-af25-47e2-860e-f7d5186d4491/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/ab451161-af25-47e2-860e-f7d5186d4491/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/accde534-15e4-42cf-94df-c0aa085f0603/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/accde534-15e4-42cf-94df-c0aa085f0603/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/accde534-15e4-42cf-94df-c0aa085f0603/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/accde534-15e4-42cf-94df-c0aa085f0603/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/accde534-15e4-42cf-94df-c0aa085f0603/volumes/kubernetes.io~projected/kube-api-access-wksmn:{mountpoint:/var/lib/kubelet/pods/accde534-15e4-42cf-94df-c0aa085f0603/volumes/kubernetes.io~projected/kube-api-access-wksmn major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0992e81-44ba-41c3-8079-c801a2d40aa4/volumes/kubernetes.io~projected/kube-api-access-lltjs:{mountpoint:/var/lib/kubelet/pods/b0992e81-44ba-41c3-8079-c801a2d40aa4/volumes/kubernetes.io~projected/kube-api-access-lltjs major:0 minor:378 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0992e81-44ba-41c3-8079-c801a2d40aa4/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/b0992e81-44ba-41c3-8079-c801a2d40aa4/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:376 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0992e81-44ba-41c3-8079-c801a2d40aa4/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/b0992e81-44ba-41c3-8079-c801a2d40aa4/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:377 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0992e81-44ba-41c3-8079-c801a2d40aa4/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/b0992e81-44ba-41c3-8079-c801a2d40aa4/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:374 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0992e81-44ba-41c3-8079-c801a2d40aa4/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/b0992e81-44ba-41c3-8079-c801a2d40aa4/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:375 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bea99b12-e97c-4c0e-9360-07e542a4294d/volumes/kubernetes.io~projected/kube-api-access-m8fhd:{mountpoint:/var/lib/kubelet/pods/bea99b12-e97c-4c0e-9360-07e542a4294d/volumes/kubernetes.io~projected/kube-api-access-m8fhd major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6/volume-subpaths/web-config/alertmanager/9 major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6/volume-subpaths/web-config/config-reloader/6 major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6/volumes/kubernetes.io~empty-dir/config-out major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6/volumes/kubernetes.io~projected/kube-api-access-82x4j:{mountpoint:/var/lib/kubelet/pods/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6/volumes/kubernetes.io~projected/kube-api-access-82x4j major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6/volumes/kubernetes.io~projected/tls-assets major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6/volumes/kubernetes.io~secret/config-volume major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6/volumes/kubernetes.io~secret/web-config major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c44f520d-f602-40e0-94d4-23277e56d67e/volumes/kubernetes.io~projected/kube-api-access-jhkpc:{mountpoint:/var/lib/kubelet/pods/c44f520d-f602-40e0-94d4-23277e56d67e/volumes/kubernetes.io~projected/kube-api-access-jhkpc major:0 minor:358 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c44f520d-f602-40e0-94d4-23277e56d67e/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/c44f520d-f602-40e0-94d4-23277e56d67e/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:352 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c44f520d-f602-40e0-94d4-23277e56d67e/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/c44f520d-f602-40e0-94d4-23277e56d67e/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c44f520d-f602-40e0-94d4-23277e56d67e/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/c44f520d-f602-40e0-94d4-23277e56d67e/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:357 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9b03988-7080-4352-ba1e-08ee68ba3df8/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/d9b03988-7080-4352-ba1e-08ee68ba3df8/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9b03988-7080-4352-ba1e-08ee68ba3df8/volumes/kubernetes.io~projected/kube-api-access-qp4zz:{mountpoint:/var/lib/kubelet/pods/d9b03988-7080-4352-ba1e-08ee68ba3df8/volumes/kubernetes.io~projected/kube-api-access-qp4zz major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9b03988-7080-4352-ba1e-08ee68ba3df8/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/d9b03988-7080-4352-ba1e-08ee68ba3df8/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c/volumes/kubernetes.io~projected/kube-api-access-x9x52:{mountpoint:/var/lib/kubelet/pods/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c/volumes/kubernetes.io~projected/kube-api-access-x9x52 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd3d929e-3197-4794-9b31-794427a55693/volumes/kubernetes.io~projected/kube-api-access-m8xd7:{mountpoint:/var/lib/kubelet/pods/dd3d929e-3197-4794-9b31-794427a55693/volumes/kubernetes.io~projected/kube-api-access-m8xd7 major:0 minor:272 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd3d929e-3197-4794-9b31-794427a55693/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/dd3d929e-3197-4794-9b31-794427a55693/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd3d929e-3197-4794-9b31-794427a55693/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/dd3d929e-3197-4794-9b31-794427a55693/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volume-subpaths/web-config/config-reloader/2 major:0 minor:404 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:404 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volume-subpaths/web-config/prometheus/15 major:0 minor:404 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volumes/kubernetes.io~empty-dir/config-out major:0 minor:409 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volumes/kubernetes.io~projected/kube-api-access-d876w:{mountpoint:/var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volumes/kubernetes.io~projected/kube-api-access-d876w major:0 minor:412 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volumes/kubernetes.io~projected/tls-assets major:0 minor:402 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volumes/kubernetes.io~secret/config major:0 minor:408 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:410 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:406 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:407 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:405 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:397 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:403 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:411 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volumes/kubernetes.io~secret/web-config major:0 minor:404 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee23e57d-7b77-4dba-95a5-84ad59133981/volumes/kubernetes.io~projected/kube-api-access-hwddx:{mountpoint:/var/lib/kubelet/pods/ee23e57d-7b77-4dba-95a5-84ad59133981/volumes/kubernetes.io~projected/kube-api-access-hwddx major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f89a7014-2966-43c7-a901-709c690b0003/volumes/kubernetes.io~projected/kube-api-access-ssl6b:{mountpoint:/var/lib/kubelet/pods/f89a7014-2966-43c7-a901-709c690b0003/volumes/kubernetes.io~projected/kube-api-access-ssl6b major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f89a7014-2966-43c7-a901-709c690b0003/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/f89a7014-2966-43c7-a901-709c690b0003/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f89a7014-2966-43c7-a901-709c690b0003/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/f89a7014-2966-43c7-a901-709c690b0003/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe19a15d-5112-4b76-a226-4b8462e2fb33/volumes/kubernetes.io~projected/kube-api-access-qpgrm:{mountpoint:/var/lib/kubelet/pods/fe19a15d-5112-4b76-a226-4b8462e2fb33/volumes/kubernetes.io~projected/kube-api-access-qpgrm major:0 minor:65 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/80cc070d6dfed4512c1ef9168df6b7f5e845edc4a39d3bae62497712cc055293/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/9a0edafdf71ffbc4407e95b0787820f559a2e5b1a4e2801266658ac5acc33b7c/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/04fe95537570f39d444e26a7a491c6c60f26096df129c8edec156be50f42dcc5/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/e9e964a951d1fb619a8113df8d583f4e7acee018c1b40da218c46caaecf15dfb/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/5ee979ae6b0962016594921df4f5d48ff098095298b85545f12d888ce6dd85eb/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/2da0b7f8978604538ea3cad32ba217752a3c2b41d438204ffd90fa20ed0c3614/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/fe8459e0aad4674c77ca7a7ac5ba48a8badf45a8891f25d4637bd56c4784a2cf/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/1e29db695e690c805384811abe38c99bf61e666de8b6701e563d7b48e62082a0/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/83a97767c6fb9d5b9352fd63986ddf03871608e6f5864e83b069c475f6d58971/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-133:{mountpoint:/var/lib/containers/storage/overlay/516ae12a3e37f2441e6e2031a53c6f26e6641038fe4853f2db8275241ae150be/merged major:0 minor:133 fsType:overlay blockSize:0} overlay_0-145:{mountpoint:/var/lib/containers/storage/overlay/d84de7b6e95caadde629b6bfbb3088fbfc5bbf6c52c189e158b2baad98eb3a0e/merged major:0 minor:145 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/1ff356538032c606543de6722fc239759a33b4b9e829c5221daa7257b739ea4f/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-148:{mountpoint:/var/lib/containers/storage/overlay/d2e02d8bab703b030a899944066e9ee7ef507abf6dfffe1dd670890a63ea0068/merged major:0 minor:148 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/8760a1554adc44c0d77e9b50f59d720509a3ba604cd80a8e1145e75ad2ebb336/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/ad7c5b78561c28e94dc276cb3fa483211fe26405787f0ec3eb5a9f14a845ed3a/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/a2691dc9315a8437cfc1303d75dd04ccbf993aaad4f669805dfc61d4339c1f01/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/30de788f34dfb0dc73141d28208291cfd3b1231bb6618196cbf7a7901810d3bb/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/0987ed91507874848b4bf1630813d911b6476af4f7df6cc057246a614ffe94d4/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/6087639e689f7dcb0ddb208968f35fb990b62c59b6924545a72d80d693ee097d/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/77c1456a7c1ac831e59b9ef8b7f6fce782bae8bf91eff33586e634265de73faa/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/9ba78509f9a8a635325f0ef5562410ff758a1bfdca5c7e78e89ac45760fbaeff/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/989408023a4af8f8c260a4717cea0bb34403a6e06da3051925ac6ca1a43a5d59/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/4363c9855071024e557ab962d8756d2e8a09553722c0383e285d34e29059ec40/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/5a3ec26ea2b1433bb9f8d0ea3e27bf6ff894778a36702d852ed806ee1097b56e/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/35330894a5ce972ed0ded84b220b007333009b0e5a6c4225514beb01d7c2cc11/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/93d4adb2858e10aea4df4eb00fe031962137a62d69a69e07ecd6cc65e5e7ee88/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/071029731ce2ed98b75e8820710d4a9fd33b215b94015277ab7da5bc2c1d914d/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/cc8ddcaf6451a4bb6e206482f3e7997195649348d6aed531410df2bbe55729c4/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/fbf177cb50f8cfe85fd1c1fc0572b20861093019c074f87531da7b0a6436dbf6/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/d7ca46529d31d57f78af402e50862409b37a97e4c1afc3c2604f29937830f399/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/4dade3925f70b10108df0f6d002b97d5b33c63fb1c3fc8407dd07feb6e4fa84c/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-257:{mountpoint:/var/lib/containers/storage/overlay/7e265534e1f5c4b0e8aff7648dcb59c012c49cfc71f9f1098386e5f96a01b503/merged major:0 minor:257 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/d93617ddc19897ab4424afe1cdc0986514a5e06ce0ae31dde2d452ceb95ef915/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/864eed821290d85dbf89bc70599ccd99db87712af577ba9940d25ea9b406b40e/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/5bb50fbedc941b8eb6c3d2f9df61e9e50a6fa040ef05f9b6c39b46f6a9228d8d/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/eecaa83382c99243989537c46eb807df35e7c634664f7d1fbbc0083f28863384/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/e39d919c93c4d820ca2fec31880b4a82d854c8a56cddd62ec56d2b5cd2954956/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/7fb8abdbac6057a3a1aabd55f105a38197f33e849fda44be8868a65039a9752e/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/2c9701537909543f45e91aac4f05e7968c8b4995b61a1aebf7877a465298c10c/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/80932b9f49ed490183ffa0cbb29033951128194c499502f5369534d633fee98a/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/c1b60b7b5f7ed10a108ee6c41c95a4866ae09c0a6a20d4d3fce24c5ae3c13068/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/9ae342864a7bcc7b631c43a64bf8608e2a9357692787bc0e572fc45c3c0e1b14/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/d51a7e20fb428a95c32ff172e5a7c8f8bf0b255c9e70334050d4f7e1071cd879/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-331:{mountpoint:/var/lib/containers/storage/overlay/4ba291ce71cb0d43864b30c5d79d12f92f70675990ac340a56c04f924408c108/merged major:0 minor:331 fsType:overlay blockSize:0} overlay_0-337:{mountpoint:/var/lib/containers/storage/overlay/dfb22b20e3424ac653fa173b48a1980fba960e09cc6b45696bc008999904cda2/merged major:0 minor:337 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/8ac69f29f24882a28ab7599cb16aff155e4928dfebafb87482d10abe36cf413f/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-347:{mountpoint:/var/lib/containers/storage/overlay/5efc6d128601d22144e3fde61dd8c269eab991cd06153e4eece106fbecb1db95/merged major:0 minor:347 fsType:overlay blockSize:0} overlay_0-365:{mountpoint:/var/lib/containers/storage/overlay/14f7049709952fda46150a941e105df949f097891e38fd687b81f257c0006496/merged major:0 minor:365 fsType:overlay blockSize:0} overlay_0-367:{mountpoint:/var/lib/containers/storage/overlay/11202eac978780100f86f60db77bde84110b0bcff3c11b1b325c380f10672542/merged major:0 minor:367 fsType:overlay blockSize:0} overlay_0-369:{mountpoint:/var/lib/containers/storage/overlay/f9d387b8df75125b1a92637828dd1c7c479af1b4fff2692b2e6f01f696c7ef14/merged major:0 minor:369 fsType:overlay blockSize:0} overlay_0-379:{mountpoint:/var/lib/containers/storage/overlay/1f7b9b4784ce0ec788d67f4415aa100c7ce3e76a9d4ab82404d18a64bc0df298/merged major:0 minor:379 fsType:overlay blockSize:0} overlay_0-387:{mountpoint:/var/lib/containers/storage/overlay/0afefbb4579312330f71b66af8a934a3fbe3dfda566f2399705fbf0aca2f58d6/merged major:0 minor:387 fsType:overlay blockSize:0} overlay_0-389:{mountpoint:/var/lib/containers/storage/overlay/5610df566d8e9de05e701aba55f827b2248b34120e7028ea803f77f021af2d09/merged major:0 minor:389 fsType:overlay blockSize:0} overlay_0-391:{mountpoint:/var/lib/containers/storage/overlay/5ecf0541240027866d96cec16f58be2c0cc528e65b75f92bf2ddd71dac16c467/merged major:0 minor:391 fsType:overlay blockSize:0} overlay_0-415:{mountpoint:/var/lib/containers/storage/overlay/2754d3167b0063876aeaa1fdc5478bb3cd6bef3da06a56367490ae7f871a0205/merged major:0 minor:415 fsType:overlay blockSize:0} overlay_0-417:{mountpoint:/var/lib/containers/storage/overlay/60a21d5bdf6578f0f60542f1a6caae197686a8732dc54ffd2bb2521d8a2b0d1f/merged major:0 minor:417 fsType:overlay blockSize:0} overlay_0-419:{mountpoint:/var/lib/containers/storage/overlay/0b77e370e561f1abedb3f32538a74c30d4fdeb57f217825092be2ada3b7a963b/merged major:0 minor:419 fsType:overlay blockSize:0} overlay_0-425:{mountpoint:/var/lib/containers/storage/overlay/a7ae47035198cda9fedfa0f448e5ac96c77e438ef6bf06bcbcffb74c8bcca8fc/merged major:0 minor:425 fsType:overlay blockSize:0} overlay_0-430:{mountpoint:/var/lib/containers/storage/overlay/92cd0ea939d4046d695f4cddd5e48d1cad951ac626c742d0644a6212d3e7a182/merged major:0 minor:430 fsType:overlay blockSize:0} overlay_0-435:{mountpoint:/var/lib/containers/storage/overlay/489e66325078e6bfb60ddfc4ce5b845a44e6ca9b77cf63c0951633b486e620e3/merged major:0 minor:435 fsType:overlay blockSize:0} overlay_0-441:{mountpoint:/var/lib/containers/storage/overlay/c2d85b93582889c00590b6db45754b9ff3e3af4f0432e62f21bf32ccf54a1823/merged major:0 minor:441 fsType:overlay blockSize:0} overlay_0-446:{mountpoint:/var/lib/containers/storage/overlay/ca186be84ca222e73640b2941aee551966e6cf993923c133044d1ac3c9cbaf97/merged major:0 minor:446 fsType:overlay blockSize:0} overlay_0-451:{mountpoint:/var/lib/containers/storage/overlay/ccb548dae16ed30b4c7a9cef4647330d497344e6fcde24469f02f04041f3b214/merged major:0 minor:451 fsType:overlay blockSize:0} overlay_0-456:{mountpoint:/var/lib/containers/storage/overlay/69f2dc08fda24190dbab84464239173ae030589fd4662d90817218f579779640/merged major:0 minor:456 fsType:overlay blockSize:0} overlay_0-474:{mountpoint:/var/lib/containers/storage/overlay/c58874c1d0653f48d48d6ce2122d77cb6f1450495e751631e813317dfcf674f6/merged major:0 minor:474 fsType:overlay blockSize:0} overlay_0-477:{mountpoint:/var/lib/containers/storage/overlay/521d157f795b466a537d34d12af14fc5d4a9dce97f70325c8d02e1e0f316f7a3/merged major:0 minor:477 fsType:overlay blockSize:0} overlay_0-480:{mountpoint:/var/lib/containers/storage/overlay/3de626aa4d2ca6bc40d9960173efed3c1abe6713f3243eee191247daa665d03a/merged major:0 minor:480 fsType:overlay blockSize:0} overlay_0-484:{mountpoint:/var/lib/containers/storage/overlay/1b2f8ab49fbb9dae7b94f5143127c7114c577acfd1cb04f6cd97a05239161d34/merged major:0 minor:484 fsType:overlay blockSize:0} overlay_0-486:{mountpoint:/var/lib/containers/storage/overlay/28514050be8721df3fd06f7fe9840365db1a79f6f32fc0fc7474105decddff71/merged major:0 minor:486 fsType:overlay blockSize:0} overlay_0-488:{mountpoint:/var/lib/containers/storage/overlay/826f836048552f0262861b01d3d4444df8075f4cf009c89bc6e387770a391fd4/merged major:0 minor:488 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/12bbf93f13b00b114793c68964d66c242942d6ffde25b2df96ffe66ae3fe587f/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-494:{mountpoint:/var/lib/containers/storage/overlay/c71090ced6fee4a94644847005354d8415af575129d636bceeb508b1545929a6/merged major:0 minor:494 fsType:overlay blockSize:0} overlay_0-500:{mountpoint:/var/lib/containers/storage/overlay/58b708541a2684929d8dde9a816d498d588cb775fe94edc797c30270964240ba/merged major:0 minor:500 fsType:overlay blockSize:0} overlay_0-505:{mountpoint:/var/lib/containers/storage/overlay/73885de39ccdae7109501b1927bd310204bbfd1d8697359c606bfaef52acf0ab/merged major:0 minor:505 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/da8c580b0d050cedac8c9800d6dfaa07d9877868a48d01096d8f708ccf6fb4d5/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/45b7cfdab427adcf11ae9066c772c2b67305e940746b31c9a9b05e7e8ef00a25/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/6bd05b712fa5dd8b7e426f537e5888cd3474a21320851b4ace503ac22d00e43a/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/bcf8daadb89017d947096996c35df25769ab282b4114ae8b896be0a27f9195e3/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/95fea27cd04d40b3aa42ca696d1b914beee47a1f98d04e802640f409e1acde5e/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/3051602633d273c1cbe7c327bdbef8be7aca387a0301e2eb22694a3b949bafe2/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/b5f0814cb1be70bf1414ac75d2b675c5349c0b1b08394bcaeaab7309c1dee403/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/5d284aa30abe39ebeb0a54394c8e9fe8644b21ffa2c0192d94581ddb22d69889/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/98703f8dd1faac51028bc9718e4bd9f86707c9173b35dcc8001e0b8402d66b54/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 12:52:23.591258 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.590851 5922 manager.go:217] Machine: {Timestamp:2026-10-30 12:52:23.590382733 +0000 UTC m=+0.059798468 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3109423 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20dd1909b8e7a8a21ca82f86d2faeb SystemUUID:ec20dd19-09b8-e7a8-a21c-a82f86d2faeb BootID:eb44e9f4-016b-4b3c-8320-cea1e9014cf9 Filesystems:[{Device:/var/lib/kubelet/pods/d9b03988-7080-4352-ba1e-08ee68ba3df8/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dd3d929e-3197-4794-9b31-794427a55693/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:270 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b0992e81-44ba-41c3-8079-c801a2d40aa4/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:376 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:409 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volumes/kubernetes.io~projected/kube-api-access-d876w DeviceMajor:0 DeviceMinor:412 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-435 DeviceMajor:0 DeviceMinor:435 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c8edd7fed57efb3f2fe9eb4b5689338b4c619b94606f3223fac094499fd99a54/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1fcccc7b-a83c-4c52-a648-cb549e70b54d/volumes/kubernetes.io~projected/kube-api-access-xxwlx DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/14c0845c-1947-44b0-ada5-4b68ff9974f2/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:361 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-367 DeviceMajor:0 DeviceMinor:367 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/accde534-15e4-42cf-94df-c0aa085f0603/volumes/kubernetes.io~projected/kube-api-access-wksmn DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a76e4cca2d9ac374f3d281c532a10ca542230fb3203096c2b2727d6e57f2f39a/userdata/shm DeviceMajor:0 DeviceMinor:469 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/00f70bf3b3b7e70a68aa7921812aeb917267fc7aa4ee2483a0e6a458e7e26f6d/userdata/shm DeviceMajor:0 DeviceMinor:363 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/11a0b508-2f9c-4be6-b5c7-b3d74766e801/volumes/kubernetes.io~projected/kube-api-access-tkldf DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8fd0fdb6-c254-446d-8832-5a2fb284f03d/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:277 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f89a7014-2966-43c7-a901-709c690b0003/volumes/kubernetes.io~projected/kube-api-access-ssl6b DeviceMajor:0 DeviceMinor:314 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:285 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-331 DeviceMajor:0 DeviceMinor:331 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:405 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dd3d929e-3197-4794-9b31-794427a55693/volumes/kubernetes.io~projected/kube-api-access-m8xd7 DeviceMajor:0 DeviceMinor:272 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:286 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c44f520d-f602-40e0-94d4-23277e56d67e/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:357 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:404 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-456 DeviceMajor:0 DeviceMinor:456 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8b0ab6dfec3ac6c49e388269cf07f3ee06fb4d7ee5f351155e4ccf3f9e88f43a/userdata/shm DeviceMajor:0 DeviceMinor:471 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-474 DeviceMajor:0 DeviceMinor:474 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/58287815249aca0628fe9cfd989113fcd3966286481e51121bf107cd4599d14f/userdata/shm DeviceMajor:0 DeviceMinor:292 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-347 DeviceMajor:0 DeviceMinor:347 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/07156ce7e70955a73bee391bd2ecbcc00edb79c18b55d20af84579c41d048044/userdata/shm DeviceMajor:0 DeviceMinor:359 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-446 DeviceMajor:0 DeviceMinor:446 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0647e9ff213d9818f0c5cc4f86bd064d5d41a6318831f00fb5d98ca6c711629e/userdata/shm DeviceMajor:0 DeviceMinor:482 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b72d684ebf8e4040794f375efa08178308d583211935ee32ab65d99a3a74d8e6/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2d03adb62aeda4a8cc61bb3dca86865ed63bd63ff6e3b7d346b3c6d438fcd903/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/51a542f5-796e-46b1-95bd-31751d50b52b/volumes/kubernetes.io~projected/kube-api-access-tvf5p DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6ff02a1f79b7714a3c55113244bdf4c133dd72421d6929105f6298f05ccb7636/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4dca982f-f509-454b-a057-d7fd9b3f34bd/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/50bec85b-5592-437d-aa6f-4899aeb494a5/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:461 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/66172ef8-aaad-4875-b3b3-bd81b9e9ba59/volumes/kubernetes.io~projected/kube-api-access-wvxmx DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d9b03988-7080-4352-ba1e-08ee68ba3df8/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:287 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/accde534-15e4-42cf-94df-c0aa085f0603/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-133 DeviceMajor:0 DeviceMinor:133 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/56928a4d-7923-46e6-97bb-6690514f89a4/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:241 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:284 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:282 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:289 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:285 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-379 DeviceMajor:0 DeviceMinor:379 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:402 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1bb48bfa-827b-4179-b9fa-418fc14f0f44/volumes/kubernetes.io~projected/kube-api-access-dqvdp DeviceMajor:0 DeviceMinor:466 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-484 DeviceMajor:0 DeviceMinor:484 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-505 DeviceMajor:0 DeviceMinor:505 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:285 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-337 DeviceMajor:0 DeviceMinor:337 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b0992e81-44ba-41c3-8079-c801a2d40aa4/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:377 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1e3b60100d2344794fcb00673508fd88209fb2fba72fe2ce48906b9a978dd3a7/userdata/shm DeviceMajor:0 DeviceMinor:381 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:407 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4407c6e829fd9619dbbaeb700be6308ccf26f6335725731cf1d5ddd53c2c6ad3/userdata/shm DeviceMajor:0 DeviceMinor:249 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:397 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:410 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/01873344-73c8-47b2-b6ba-8e338ba18315/volumes/kubernetes.io~projected/kube-api-access-2w592 DeviceMajor:0 DeviceMinor:479 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cfaf6eb0e32f16cf0289059ab69c1e2d5690928fa9d7a16ea1a6cecf114b053c/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bea99b12-e97c-4c0e-9360-07e542a4294d/volumes/kubernetes.io~projected/kube-api-access-m8fhd DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:283 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/51a542f5-796e-46b1-95bd-31751d50b52b/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/92483d9ab9d6b9c70a1cf111febac24212e30ce539406891994098559142f86b/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/815dcea6522fbada33cae727dbd21ebe8040c2dced78a4d7324e4652362ad187/userdata/shm DeviceMajor:0 DeviceMinor:315 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:411 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3fed4127a265b40e1040260f5bbf5bcc0d93021fbf60ba8b2dac44897c94681f/userdata/shm DeviceMajor:0 DeviceMinor:467 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-494 DeviceMajor:0 DeviceMinor:494 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/ee23e57d-7b77-4dba-95a5-84ad59133981/volumes/kubernetes.io~projected/kube-api-access-hwddx DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f2508189ec0d29ac883e4c6404a709ef657cdf80440bf5200b0fc427fa4f210d/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/56928a4d-7923-46e6-97bb-6690514f89a4/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:245 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8fd0fdb6-c254-446d-8832-5a2fb284f03d/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:265 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/64493cb387412ea7ebe03ce195389352c6b410c126810317e7a7cef26cb9b066/userdata/shm DeviceMajor:0 DeviceMinor:273 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1fcccc7b-a83c-4c52-a648-cb549e70b54d/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:465 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1697328fc8c5672d44ef75b4e614af0feacc2e137b5e29cc32a0554ac92829eb/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f89a7014-2966-43c7-a901-709c690b0003/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:312 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-387 DeviceMajor:0 DeviceMinor:387 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:404 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-148 DeviceMajor:0 DeviceMinor:148 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ab451161-af25-47e2-860e-f7d5186d4491/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c44f520d-f602-40e0-94d4-23277e56d67e/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:356 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3410ea027ea71f3e80e819cc87d3a28cf98bce131de13e4a81f8b081e6458cae/userdata/shm DeviceMajor:0 DeviceMinor:413 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-417 DeviceMajor:0 DeviceMinor:417 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/accde534-15e4-42cf-94df-c0aa085f0603/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b0992e81-44ba-41c3-8079-c801a2d40aa4/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:374 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:404 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-425 DeviceMajor:0 DeviceMinor:425 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:404 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/56928a4d-7923-46e6-97bb-6690514f89a4/volumes/kubernetes.io~projected/kube-api-access-9v6nm DeviceMajor:0 DeviceMinor:246 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dd3d929e-3197-4794-9b31-794427a55693/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:269 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-369 DeviceMajor:0 DeviceMinor:369 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fe19a15d-5112-4b76-a226-4b8462e2fb33/volumes/kubernetes.io~projected/kube-api-access-qpgrm DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/076576ce8f705e376e3e20d5ba028afd9046651ed42cefcc0e1400fc95ef000a/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0fb70e7381decaff50290b92ee2c0505239b354e82ff99bf84e63be1e937c006/userdata/shm DeviceMajor:0 DeviceMinor:278 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-391 DeviceMajor:0 DeviceMinor:391 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:403 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-430 DeviceMajor:0 DeviceMinor:430 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-477 DeviceMajor:0 DeviceMinor:477 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-500 DeviceMajor:0 DeviceMinor:500 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:285 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/01873344-73c8-47b2-b6ba-8e338ba18315/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:473 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3f38af1e-cd92-4cd8-a643-151b10b8da12/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d9b03988-7080-4352-ba1e-08ee68ba3df8/volumes/kubernetes.io~projected/kube-api-access-qp4zz DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8b99f37e6db72b8da01b867c94cb74058eb342a3d1d037fbcbd4dcf00672bdbf/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ab451161-af25-47e2-860e-f7d5186d4491/volumes/kubernetes.io~projected/kube-api-access-gbr4h DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c44f520d-f602-40e0-94d4-23277e56d67e/volumes/kubernetes.io~projected/kube-api-access-jhkpc DeviceMajor:0 DeviceMinor:358 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-389 DeviceMajor:0 DeviceMinor:389 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:288 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c/volumes/kubernetes.io~projected/kube-api-access-x9x52 DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-451 DeviceMajor:0 DeviceMinor:451 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/52495134362f8b43db53d257786d659819f2329d5a13f341d1b9c0cc4a20619b/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-145 DeviceMajor:0 DeviceMinor:145 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b0992e81-44ba-41c3-8079-c801a2d40aa4/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:375 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:408 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6/volumes/kubernetes.io~projected/kube-api-access-82x4j DeviceMajor:0 DeviceMinor:291 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-365 DeviceMajor:0 DeviceMinor:365 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b0992e81-44ba-41c3-8079-c801a2d40aa4/volumes/kubernetes.io~projected/kube-api-access-lltjs DeviceMajor:0 DeviceMinor:378 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-488 DeviceMajor:0 DeviceMinor:488 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4dca982f-f509-454b-a057-d7fd9b3f34bd/volumes/kubernetes.io~projected/kube-api-access-5m944 DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:284 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:406 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-419 DeviceMajor:0 DeviceMinor:419 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-441 DeviceMajor:0 DeviceMinor:441 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-486 DeviceMajor:0 DeviceMinor:486 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6bc842612ded17a701978fb17fb8dace799d5e75552b5c8b0104dd3c213b4857/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f89a7014-2966-43c7-a901-709c690b0003/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:313 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8fd0fdb6-c254-446d-8832-5a2fb284f03d/volumes/kubernetes.io~projected/kube-api-access-27dm6 DeviceMajor:0 DeviceMinor:271 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c44f520d-f602-40e0-94d4-23277e56d67e/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:352 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-415 DeviceMajor:0 DeviceMinor:415 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/01873344-73c8-47b2-b6ba-8e338ba18315/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:476 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ae4a53261c8f2d08d8af9130166c20d999ab7bf0ed529f7d154cbc0ae514e6ed/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-257 DeviceMajor:0 DeviceMinor:257 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:290 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/14c0845c-1947-44b0-ada5-4b68ff9974f2/volumes/kubernetes.io~projected/kube-api-access-wvdzg DeviceMajor:0 DeviceMinor:362 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-480 DeviceMajor:0 DeviceMinor:480 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:00f70bf3b3b7e70 MacAddress:ae:a8:f9:77:35:fd Speed:10000 Mtu:8901} {Name:0647e9ff213d981 MacAddress:7e:af:12:2c:ab:ba Speed:10000 Mtu:8901} {Name:07156ce7e70955a MacAddress:52:50:e5:3c:23:0d Speed:10000 Mtu:8901} {Name:1e3b60100d23447 MacAddress:4e:58:f4:20:f9:65 Speed:10000 Mtu:8901} {Name:3410ea027ea71f3 MacAddress:a6:9c:39:04:0a:42 Speed:10000 Mtu:8901} {Name:3fed4127a265b40 MacAddress:82:ba:bf:6c:8f:a7 Speed:10000 Mtu:8901} {Name:4407c6e829fd961 MacAddress:62:a5:62:90:3c:0a Speed:10000 Mtu:8901} {Name:58287815249aca0 MacAddress:4a:7a:d0:ea:a0:cf Speed:10000 Mtu:8901} {Name:64493cb387412ea MacAddress:2e:48:58:74:2c:d8 Speed:10000 Mtu:8901} {Name:6bc842612ded17a MacAddress:36:33:ec:de:94:8c Speed:10000 Mtu:8901} {Name:815dcea6522fbad MacAddress:5a:16:ca:9d:64:43 Speed:10000 Mtu:8901} {Name:8b0ab6dfec3ac6c MacAddress:82:cd:47:c9:c9:66 Speed:10000 Mtu:8901} {Name:92483d9ab9d6b9c MacAddress:a2:b4:2b:e3:50:35 Speed:10000 Mtu:8901} {Name:a76e4cca2d9ac37 MacAddress:a6:68:f7:6a:16:12 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f2:23:ad:9d Speed:0 Mtu:9001} {Name:br-int MacAddress:0a:b4:d7:a1:c3:6b Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:f2:23:ad:9d Speed:0 Mtu:9001} {Name:f2508189ec0d29a MacAddress:9a:94:76:d7:bf:da Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:aa:9e:34:ef:ba:39 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:06:35:fe:23:01:82 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 12:52:23.591508 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.591262 5922 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 12:52:23.591508 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.591301 5922 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 12:52:23.591581 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.591557 5922 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 12:52:23.591725 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.591582 5922 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-21.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 12:52:23.591767 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.591727 5922 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 12:52:23.591767 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.591736 5922 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 12:52:23.591767 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.591749 5922 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 12:52:23.591886 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.591768 5922 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 12:52:23.591886 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.591834 5922 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 12:52:23.591992 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.591982 5922 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 12:52:23.592088 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.592079 5922 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 12:52:23.592119 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.592114 5922 kubelet.go:493] "Attempting to sync node with API server" Oct 30 12:52:23.592147 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.592124 5922 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 12:52:23.592147 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.592135 5922 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 12:52:23.592147 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.592144 5922 kubelet.go:405] "Adding apiserver pod source" Oct 30 12:52:23.592226 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.592152 5922 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 12:52:23.594062 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.593565 5922 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 12:52:23.594062 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.593590 5922 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 12:52:23.594424 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.594402 5922 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 12:52:23.597082 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.597063 5922 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 12:52:23.597082 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.597078 5922 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 12:52:23.597213 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.597188 5922 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 12:52:23.597407 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.597391 5922 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 12:52:23.597470 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.597432 5922 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 12:52:23.597565 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.597554 5922 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 12:52:23.597644 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.597574 5922 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 12:52:23.597644 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.597585 5922 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 12:52:23.597644 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.597594 5922 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 12:52:23.597644 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.597603 5922 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 12:52:23.597644 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.597633 5922 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 12:52:23.597644 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.597644 5922 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 12:52:23.597921 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.597652 5922 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 12:52:23.597921 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.597664 5922 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 12:52:23.597921 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.597673 5922 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 12:52:23.597921 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.597692 5922 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 12:52:23.597921 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.597707 5922 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 12:52:23.597921 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.597740 5922 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 12:52:23.597921 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.597750 5922 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 12:52:23.601169 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.601152 5922 server.go:1263] "Started kubelet" Oct 30 12:52:23.601245 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.601210 5922 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 12:52:23.601300 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.601248 5922 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 12:52:23.601342 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.601327 5922 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 12:52:23.601650 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.601630 5922 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 12:52:23.601654 ip-10-0-137-21 systemd[1]: Started Kubernetes Kubelet. Oct 30 12:52:23.602383 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.602366 5922 server.go:317] "Adding debug handlers to kubelet server" Oct 30 12:52:23.606862 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.606670 5922 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 12:52:23.606862 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.606686 5922 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 12:52:23.606862 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.606847 5922 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 12:52:23.606862 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.606860 5922 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 12:52:23.607239 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.606984 5922 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 12:52:23.607239 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.606775 5922 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:46:14 +0000 UTC" deadline="2028-03-26 01:13:35.858239528 +0000 UTC" Oct 30 12:52:23.607239 ip-10-0-137-21 kubenswrapper[5922]: E1030 12:52:23.607108 5922 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-21.ec2.internal\" not found" Oct 30 12:52:23.607239 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.607134 5922 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12300h21m12.251110226s" Oct 30 12:52:23.608398 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.608303 5922 factory.go:55] Registering systemd factory Oct 30 12:52:23.608398 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.608324 5922 factory.go:223] Registration of the systemd container factory successfully Oct 30 12:52:23.608827 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.608802 5922 factory.go:153] Registering CRI-O factory Oct 30 12:52:23.608827 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.608820 5922 factory.go:223] Registration of the crio container factory successfully Oct 30 12:52:23.608966 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.608873 5922 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 12:52:23.608966 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.608894 5922 factory.go:103] Registering Raw factory Oct 30 12:52:23.608966 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.608913 5922 manager.go:1199] Started watching for new ooms in manager Oct 30 12:52:23.609269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.609256 5922 manager.go:319] Starting recovery of all containers Oct 30 12:52:23.614098 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614060 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f89a7014-2966-43c7-a901-709c690b0003" volumeName="kubernetes.io/projected/f89a7014-2966-43c7-a901-709c690b0003-kube-api-access-ssl6b" seLinuxMountContext="" Oct 30 12:52:23.614197 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614101 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fcccc7b-a83c-4c52-a648-cb549e70b54d" volumeName="kubernetes.io/secret/1fcccc7b-a83c-4c52-a648-cb549e70b54d-metrics-certs" seLinuxMountContext="" Oct 30 12:52:23.614197 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614114 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11a0b508-2f9c-4be6-b5c7-b3d74766e801" volumeName="kubernetes.io/projected/11a0b508-2f9c-4be6-b5c7-b3d74766e801-kube-api-access-tkldf" seLinuxMountContext="" Oct 30 12:52:23.614197 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614126 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56928a4d-7923-46e6-97bb-6690514f89a4" volumeName="kubernetes.io/projected/56928a4d-7923-46e6-97bb-6690514f89a4-kube-api-access-9v6nm" seLinuxMountContext="" Oct 30 12:52:23.614197 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614136 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" volumeName="kubernetes.io/projected/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-tls-assets" seLinuxMountContext="" Oct 30 12:52:23.614197 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614147 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9b03988-7080-4352-ba1e-08ee68ba3df8" volumeName="kubernetes.io/secret/d9b03988-7080-4352-ba1e-08ee68ba3df8-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 12:52:23.614197 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614160 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecd3d4d5-a59a-4b0e-a981-956b10d08531" volumeName="kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 12:52:23.614197 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614171 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4dca982f-f509-454b-a057-d7fd9b3f34bd" volumeName="kubernetes.io/secret/4dca982f-f509-454b-a057-d7fd9b3f34bd-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 12:52:23.614520 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614190 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd3d929e-3197-4794-9b31-794427a55693" volumeName="kubernetes.io/secret/dd3d929e-3197-4794-9b31-794427a55693-kube-state-metrics-tls" seLinuxMountContext="" Oct 30 12:52:23.614520 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614226 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecd3d4d5-a59a-4b0e-a981-956b10d08531" volumeName="kubernetes.io/empty-dir/ecd3d4d5-a59a-4b0e-a981-956b10d08531-config-out" seLinuxMountContext="" Oct 30 12:52:23.614520 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614240 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecd3d4d5-a59a-4b0e-a981-956b10d08531" volumeName="kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-config" seLinuxMountContext="" Oct 30 12:52:23.614520 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614252 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f38af1e-cd92-4cd8-a643-151b10b8da12" volumeName="kubernetes.io/configmap/3f38af1e-cd92-4cd8-a643-151b10b8da12-konnectivity-ca" seLinuxMountContext="" Oct 30 12:52:23.614520 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614264 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4dca982f-f509-454b-a057-d7fd9b3f34bd" volumeName="kubernetes.io/empty-dir/4dca982f-f509-454b-a057-d7fd9b3f34bd-data-volume" seLinuxMountContext="" Oct 30 12:52:23.614520 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614276 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fd0fdb6-c254-446d-8832-5a2fb284f03d" volumeName="kubernetes.io/empty-dir/8fd0fdb6-c254-446d-8832-5a2fb284f03d-node-exporter-textfile" seLinuxMountContext="" Oct 30 12:52:23.614520 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614291 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" volumeName="kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 30 12:52:23.614520 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614300 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecd3d4d5-a59a-4b0e-a981-956b10d08531" volumeName="kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 30 12:52:23.614520 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614307 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe19a15d-5112-4b76-a226-4b8462e2fb33" volumeName="kubernetes.io/projected/fe19a15d-5112-4b76-a226-4b8462e2fb33-kube-api-access-qpgrm" seLinuxMountContext="" Oct 30 12:52:23.614520 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614313 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fd0fdb6-c254-446d-8832-5a2fb284f03d" volumeName="kubernetes.io/secret/8fd0fdb6-c254-446d-8832-5a2fb284f03d-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 12:52:23.614520 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614332 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" volumeName="kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 30 12:52:23.614520 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614342 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" volumeName="kubernetes.io/configmap/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 30 12:52:23.614520 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614353 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c44f520d-f602-40e0-94d4-23277e56d67e" volumeName="kubernetes.io/secret/c44f520d-f602-40e0-94d4-23277e56d67e-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 30 12:52:23.614520 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614364 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f89a7014-2966-43c7-a901-709c690b0003" volumeName="kubernetes.io/empty-dir/f89a7014-2966-43c7-a901-709c690b0003-tmp" seLinuxMountContext="" Oct 30 12:52:23.614520 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614376 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14c0845c-1947-44b0-ada5-4b68ff9974f2" volumeName="kubernetes.io/projected/14c0845c-1947-44b0-ada5-4b68ff9974f2-kube-api-access-wvdzg" seLinuxMountContext="" Oct 30 12:52:23.614520 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614388 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fcccc7b-a83c-4c52-a648-cb549e70b54d" volumeName="kubernetes.io/projected/1fcccc7b-a83c-4c52-a648-cb549e70b54d-kube-api-access-xxwlx" seLinuxMountContext="" Oct 30 12:52:23.614520 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614406 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dad46b5c-e3e3-4efa-b8f1-b13de224ef8c" volumeName="kubernetes.io/configmap/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-multus-daemon-config" seLinuxMountContext="" Oct 30 12:52:23.614520 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614418 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd3d929e-3197-4794-9b31-794427a55693" volumeName="kubernetes.io/configmap/dd3d929e-3197-4794-9b31-794427a55693-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 30 12:52:23.614520 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614430 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecd3d4d5-a59a-4b0e-a981-956b10d08531" volumeName="kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 12:52:23.614520 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614442 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0992e81-44ba-41c3-8079-c801a2d40aa4" volumeName="kubernetes.io/projected/b0992e81-44ba-41c3-8079-c801a2d40aa4-kube-api-access-lltjs" seLinuxMountContext="" Oct 30 12:52:23.614520 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614458 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c44f520d-f602-40e0-94d4-23277e56d67e" volumeName="kubernetes.io/configmap/c44f520d-f602-40e0-94d4-23277e56d67e-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 12:52:23.614520 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614469 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50bec85b-5592-437d-aa6f-4899aeb494a5" volumeName="kubernetes.io/secret/50bec85b-5592-437d-aa6f-4899aeb494a5-original-pull-secret" seLinuxMountContext="" Oct 30 12:52:23.614520 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614482 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56928a4d-7923-46e6-97bb-6690514f89a4" volumeName="kubernetes.io/secret/56928a4d-7923-46e6-97bb-6690514f89a4-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 12:52:23.614520 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614494 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01873344-73c8-47b2-b6ba-8e338ba18315" volumeName="kubernetes.io/secret/01873344-73c8-47b2-b6ba-8e338ba18315-console-oauth-config" seLinuxMountContext="" Oct 30 12:52:23.614520 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614508 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51a542f5-796e-46b1-95bd-31751d50b52b" volumeName="kubernetes.io/empty-dir/51a542f5-796e-46b1-95bd-31751d50b52b-tmp-dir" seLinuxMountContext="" Oct 30 12:52:23.614520 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614521 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fd0fdb6-c254-446d-8832-5a2fb284f03d" volumeName="kubernetes.io/configmap/8fd0fdb6-c254-446d-8832-5a2fb284f03d-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 12:52:23.614520 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614534 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" volumeName="kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614548 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecd3d4d5-a59a-4b0e-a981-956b10d08531" volumeName="kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-configmap-metrics-client-ca" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614562 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0992e81-44ba-41c3-8079-c801a2d40aa4" volumeName="kubernetes.io/secret/b0992e81-44ba-41c3-8079-c801a2d40aa4-telemeter-client-tls" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614574 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd3d929e-3197-4794-9b31-794427a55693" volumeName="kubernetes.io/empty-dir/dd3d929e-3197-4794-9b31-794427a55693-volume-directive-shadow" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614586 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01873344-73c8-47b2-b6ba-8e338ba18315" volumeName="kubernetes.io/configmap/01873344-73c8-47b2-b6ba-8e338ba18315-console-config" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614600 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0992e81-44ba-41c3-8079-c801a2d40aa4" volumeName="kubernetes.io/secret/b0992e81-44ba-41c3-8079-c801a2d40aa4-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614627 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dad46b5c-e3e3-4efa-b8f1-b13de224ef8c" volumeName="kubernetes.io/projected/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-kube-api-access-x9x52" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614640 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd3d929e-3197-4794-9b31-794427a55693" volumeName="kubernetes.io/configmap/dd3d929e-3197-4794-9b31-794427a55693-metrics-client-ca" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614655 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecd3d4d5-a59a-4b0e-a981-956b10d08531" volumeName="kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614667 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01873344-73c8-47b2-b6ba-8e338ba18315" volumeName="kubernetes.io/configmap/01873344-73c8-47b2-b6ba-8e338ba18315-service-ca" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614679 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" volumeName="kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-config-volume" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614692 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" volumeName="kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-web-config" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614704 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecd3d4d5-a59a-4b0e-a981-956b10d08531" volumeName="kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614716 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51a542f5-796e-46b1-95bd-31751d50b52b" volumeName="kubernetes.io/secret/51a542f5-796e-46b1-95bd-31751d50b52b-metrics-tls" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614728 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66172ef8-aaad-4875-b3b3-bd81b9e9ba59" volumeName="kubernetes.io/configmap/66172ef8-aaad-4875-b3b3-bd81b9e9ba59-serviceca" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614740 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" volumeName="kubernetes.io/empty-dir/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-config-out" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614751 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c44f520d-f602-40e0-94d4-23277e56d67e" volumeName="kubernetes.io/secret/c44f520d-f602-40e0-94d4-23277e56d67e-client-ca-bundle" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614763 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c44f520d-f602-40e0-94d4-23277e56d67e" volumeName="kubernetes.io/secret/c44f520d-f602-40e0-94d4-23277e56d67e-secret-metrics-server-tls" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614775 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9b03988-7080-4352-ba1e-08ee68ba3df8" volumeName="kubernetes.io/configmap/d9b03988-7080-4352-ba1e-08ee68ba3df8-ovnkube-config" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614788 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" volumeName="kubernetes.io/projected/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-kube-api-access-82x4j" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614801 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9b03988-7080-4352-ba1e-08ee68ba3df8" volumeName="kubernetes.io/configmap/d9b03988-7080-4352-ba1e-08ee68ba3df8-ovnkube-script-lib" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614812 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dad46b5c-e3e3-4efa-b8f1-b13de224ef8c" volumeName="kubernetes.io/configmap/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-cni-binary-copy" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614833 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f89a7014-2966-43c7-a901-709c690b0003" volumeName="kubernetes.io/secret/f89a7014-2966-43c7-a901-709c690b0003-console-oauth-config" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614845 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51a542f5-796e-46b1-95bd-31751d50b52b" volumeName="kubernetes.io/configmap/51a542f5-796e-46b1-95bd-31751d50b52b-config-volume" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614856 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66172ef8-aaad-4875-b3b3-bd81b9e9ba59" volumeName="kubernetes.io/projected/66172ef8-aaad-4875-b3b3-bd81b9e9ba59-kube-api-access-wvxmx" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614868 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c44f520d-f602-40e0-94d4-23277e56d67e" volumeName="kubernetes.io/projected/c44f520d-f602-40e0-94d4-23277e56d67e-kube-api-access-jhkpc" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614880 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd3d929e-3197-4794-9b31-794427a55693" volumeName="kubernetes.io/secret/dd3d929e-3197-4794-9b31-794427a55693-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614891 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecd3d4d5-a59a-4b0e-a981-956b10d08531" volumeName="kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614904 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01873344-73c8-47b2-b6ba-8e338ba18315" volumeName="kubernetes.io/configmap/01873344-73c8-47b2-b6ba-8e338ba18315-trusted-ca-bundle" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614916 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1bb48bfa-827b-4179-b9fa-418fc14f0f44" volumeName="kubernetes.io/projected/1bb48bfa-827b-4179-b9fa-418fc14f0f44-kube-api-access-dqvdp" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614929 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f38af1e-cd92-4cd8-a643-151b10b8da12" volumeName="kubernetes.io/secret/3f38af1e-cd92-4cd8-a643-151b10b8da12-agent-certs" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614940 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="accde534-15e4-42cf-94df-c0aa085f0603" volumeName="kubernetes.io/empty-dir/accde534-15e4-42cf-94df-c0aa085f0603-tmp" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614969 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c44f520d-f602-40e0-94d4-23277e56d67e" volumeName="kubernetes.io/empty-dir/c44f520d-f602-40e0-94d4-23277e56d67e-audit-log" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.614991 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9b03988-7080-4352-ba1e-08ee68ba3df8" volumeName="kubernetes.io/projected/d9b03988-7080-4352-ba1e-08ee68ba3df8-kube-api-access-qp4zz" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615003 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecd3d4d5-a59a-4b0e-a981-956b10d08531" volumeName="kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615015 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4dca982f-f509-454b-a057-d7fd9b3f34bd" volumeName="kubernetes.io/projected/4dca982f-f509-454b-a057-d7fd9b3f34bd-kube-api-access-5m944" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615028 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0992e81-44ba-41c3-8079-c801a2d40aa4" volumeName="kubernetes.io/configmap/b0992e81-44ba-41c3-8079-c801a2d40aa4-telemeter-trusted-ca-bundle" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615038 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecd3d4d5-a59a-4b0e-a981-956b10d08531" volumeName="kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615051 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe19a15d-5112-4b76-a226-4b8462e2fb33" volumeName="kubernetes.io/empty-dir/fe19a15d-5112-4b76-a226-4b8462e2fb33-tmp-dir" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615063 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4dca982f-f509-454b-a057-d7fd9b3f34bd" volumeName="kubernetes.io/configmap/4dca982f-f509-454b-a057-d7fd9b3f34bd-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615075 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fd0fdb6-c254-446d-8832-5a2fb284f03d" volumeName="kubernetes.io/configmap/8fd0fdb6-c254-446d-8832-5a2fb284f03d-metrics-client-ca" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615086 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecd3d4d5-a59a-4b0e-a981-956b10d08531" volumeName="kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615099 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f89a7014-2966-43c7-a901-709c690b0003" volumeName="kubernetes.io/configmap/f89a7014-2966-43c7-a901-709c690b0003-service-ca" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615109 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0992e81-44ba-41c3-8079-c801a2d40aa4" volumeName="kubernetes.io/secret/b0992e81-44ba-41c3-8079-c801a2d40aa4-federate-client-tls" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615121 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" volumeName="kubernetes.io/configmap/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-metrics-client-ca" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615135 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01873344-73c8-47b2-b6ba-8e338ba18315" volumeName="kubernetes.io/empty-dir/01873344-73c8-47b2-b6ba-8e338ba18315-tmp" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615147 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11a0b508-2f9c-4be6-b5c7-b3d74766e801" volumeName="kubernetes.io/configmap/11a0b508-2f9c-4be6-b5c7-b3d74766e801-iptables-alerter-script" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615158 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fd0fdb6-c254-446d-8832-5a2fb284f03d" volumeName="kubernetes.io/secret/8fd0fdb6-c254-446d-8832-5a2fb284f03d-node-exporter-tls" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615169 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab451161-af25-47e2-860e-f7d5186d4491" volumeName="kubernetes.io/projected/ab451161-af25-47e2-860e-f7d5186d4491-kube-api-access-gbr4h" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615179 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" volumeName="kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615192 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecd3d4d5-a59a-4b0e-a981-956b10d08531" volumeName="kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615203 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecd3d4d5-a59a-4b0e-a981-956b10d08531" volumeName="kubernetes.io/projected/ecd3d4d5-a59a-4b0e-a981-956b10d08531-kube-api-access-d876w" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615215 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f89a7014-2966-43c7-a901-709c690b0003" volumeName="kubernetes.io/configmap/f89a7014-2966-43c7-a901-709c690b0003-oauth-serving-cert" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615226 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f89a7014-2966-43c7-a901-709c690b0003" volumeName="kubernetes.io/configmap/f89a7014-2966-43c7-a901-709c690b0003-console-config" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615237 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" volumeName="kubernetes.io/empty-dir/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-alertmanager-main-db" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615248 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd3d929e-3197-4794-9b31-794427a55693" volumeName="kubernetes.io/projected/dd3d929e-3197-4794-9b31-794427a55693-kube-api-access-m8xd7" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615261 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecd3d4d5-a59a-4b0e-a981-956b10d08531" volumeName="kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-grpc-tls" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615271 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee23e57d-7b77-4dba-95a5-84ad59133981" volumeName="kubernetes.io/configmap/ee23e57d-7b77-4dba-95a5-84ad59133981-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615279 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee23e57d-7b77-4dba-95a5-84ad59133981" volumeName="kubernetes.io/configmap/ee23e57d-7b77-4dba-95a5-84ad59133981-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615287 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee23e57d-7b77-4dba-95a5-84ad59133981" volumeName="kubernetes.io/projected/ee23e57d-7b77-4dba-95a5-84ad59133981-kube-api-access-hwddx" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615295 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecd3d4d5-a59a-4b0e-a981-956b10d08531" volumeName="kubernetes.io/projected/ecd3d4d5-a59a-4b0e-a981-956b10d08531-tls-assets" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615319 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecd3d4d5-a59a-4b0e-a981-956b10d08531" volumeName="kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-metrics-client-certs" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615328 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f89a7014-2966-43c7-a901-709c690b0003" volumeName="kubernetes.io/secret/f89a7014-2966-43c7-a901-709c690b0003-console-serving-cert" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615336 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56928a4d-7923-46e6-97bb-6690514f89a4" volumeName="kubernetes.io/configmap/56928a4d-7923-46e6-97bb-6690514f89a4-metrics-client-ca" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615344 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fd0fdb6-c254-446d-8832-5a2fb284f03d" volumeName="kubernetes.io/projected/8fd0fdb6-c254-446d-8832-5a2fb284f03d-kube-api-access-27dm6" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615351 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab451161-af25-47e2-860e-f7d5186d4491" volumeName="kubernetes.io/secret/ab451161-af25-47e2-860e-f7d5186d4491-cert" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615359 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecd3d4d5-a59a-4b0e-a981-956b10d08531" volumeName="kubernetes.io/empty-dir/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-k8s-db" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615366 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee23e57d-7b77-4dba-95a5-84ad59133981" volumeName="kubernetes.io/configmap/ee23e57d-7b77-4dba-95a5-84ad59133981-cni-binary-copy" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615375 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51a542f5-796e-46b1-95bd-31751d50b52b" volumeName="kubernetes.io/projected/51a542f5-796e-46b1-95bd-31751d50b52b-kube-api-access-tvf5p" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615384 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56928a4d-7923-46e6-97bb-6690514f89a4" volumeName="kubernetes.io/secret/56928a4d-7923-46e6-97bb-6690514f89a4-prometheus-operator-tls" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615395 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" volumeName="kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-cluster-tls-config" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615402 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecd3d4d5-a59a-4b0e-a981-956b10d08531" volumeName="kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-web-config" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615410 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01873344-73c8-47b2-b6ba-8e338ba18315" volumeName="kubernetes.io/projected/01873344-73c8-47b2-b6ba-8e338ba18315-kube-api-access-2w592" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615417 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0992e81-44ba-41c3-8079-c801a2d40aa4" volumeName="kubernetes.io/secret/b0992e81-44ba-41c3-8079-c801a2d40aa4-secret-telemeter-client" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615435 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c44f520d-f602-40e0-94d4-23277e56d67e" volumeName="kubernetes.io/configmap/c44f520d-f602-40e0-94d4-23277e56d67e-metrics-server-audit-profiles" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615443 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0992e81-44ba-41c3-8079-c801a2d40aa4" volumeName="kubernetes.io/configmap/b0992e81-44ba-41c3-8079-c801a2d40aa4-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615451 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01873344-73c8-47b2-b6ba-8e338ba18315" volumeName="kubernetes.io/configmap/01873344-73c8-47b2-b6ba-8e338ba18315-oauth-serving-cert" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615463 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14c0845c-1947-44b0-ada5-4b68ff9974f2" volumeName="kubernetes.io/secret/14c0845c-1947-44b0-ada5-4b68ff9974f2-monitoring-plugin-cert" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615492 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0992e81-44ba-41c3-8079-c801a2d40aa4" volumeName="kubernetes.io/configmap/b0992e81-44ba-41c3-8079-c801a2d40aa4-metrics-client-ca" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615501 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9b03988-7080-4352-ba1e-08ee68ba3df8" volumeName="kubernetes.io/configmap/d9b03988-7080-4352-ba1e-08ee68ba3df8-env-overrides" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615509 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecd3d4d5-a59a-4b0e-a981-956b10d08531" volumeName="kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615516 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01873344-73c8-47b2-b6ba-8e338ba18315" volumeName="kubernetes.io/secret/01873344-73c8-47b2-b6ba-8e338ba18315-console-serving-cert" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615523 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="accde534-15e4-42cf-94df-c0aa085f0603" volumeName="kubernetes.io/empty-dir/accde534-15e4-42cf-94df-c0aa085f0603-etc-tuned" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615531 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="accde534-15e4-42cf-94df-c0aa085f0603" volumeName="kubernetes.io/projected/accde534-15e4-42cf-94df-c0aa085f0603-kube-api-access-wksmn" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615539 5922 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bea99b12-e97c-4c0e-9360-07e542a4294d" volumeName="kubernetes.io/projected/bea99b12-e97c-4c0e-9360-07e542a4294d-kube-api-access-m8fhd" seLinuxMountContext="" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615550 5922 reconstruct.go:97] "Volume reconstruction finished" Oct 30 12:52:23.615778 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.615558 5922 reconciler.go:29] "Reconciler: start to sync state" Oct 30 12:52:23.635266 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.635234 5922 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 12:52:23.636698 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.636678 5922 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 12:52:23.636698 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.636701 5922 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 12:52:23.636839 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.636723 5922 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 12:52:23.636839 ip-10-0-137-21 kubenswrapper[5922]: E1030 12:52:23.636763 5922 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 12:52:23.653623 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.653580 5922 generic.go:356] "Generic (PLEG): container finished" podID="c7d15293c60c5848ccd82e7130b45838" containerID="3cdd4b7edc10db5ddf3920923ece442e41dbb7163b81a835bf9d485f1fbd0f0c" exitCode=0 Oct 30 12:52:23.658325 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.658304 5922 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 12:52:23.659854 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.659833 5922 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:52:23.659961 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.659887 5922 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:52:23.660820 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.660769 5922 generic.go:356] "Generic (PLEG): container finished" podID="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" containerID="6665af12009c0962bf45c992c8dbd0324d4b4e5094dd12d19baa5bd145b23ecc" exitCode=0 Oct 30 12:52:23.671990 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.671969 5922 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:52:23.675187 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.675163 5922 generic.go:356] "Generic (PLEG): container finished" podID="8fd0fdb6-c254-446d-8832-5a2fb284f03d" containerID="776eeda575dbe4788b49d97bc8d8d4b89d1d6a5b07824becaed6548ce820310b" exitCode=0 Oct 30 12:52:23.682125 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.682109 5922 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:52:23.684945 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.684912 5922 generic.go:356] "Generic (PLEG): container finished" podID="ee23e57d-7b77-4dba-95a5-84ad59133981" containerID="ea8df7b68cacfe318f960db70cd745cf8fcb6033960ddb98df1df107cf254848" exitCode=0 Oct 30 12:52:23.684945 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.684939 5922 generic.go:356] "Generic (PLEG): container finished" podID="ee23e57d-7b77-4dba-95a5-84ad59133981" containerID="f6a15e7651ec6983a0084122fd46a5921cce20e0f2e5f34e2bf5c48b84f591f8" exitCode=0 Oct 30 12:52:23.685091 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.684952 5922 generic.go:356] "Generic (PLEG): container finished" podID="ee23e57d-7b77-4dba-95a5-84ad59133981" containerID="a49e71e67cfd0c016fdeeb9daa5c8a2547728d3527ae2f5b3c6ad715168a895d" exitCode=0 Oct 30 12:52:23.685091 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.684961 5922 generic.go:356] "Generic (PLEG): container finished" podID="ee23e57d-7b77-4dba-95a5-84ad59133981" containerID="424f941b15e3c0aabdcee253632f615ca93f53d7abb333388daada2accd2a1aa" exitCode=0 Oct 30 12:52:23.685091 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.684969 5922 generic.go:356] "Generic (PLEG): container finished" podID="ee23e57d-7b77-4dba-95a5-84ad59133981" containerID="1e586f0383e556ec9d0ab1d50915441e40fa894f7a987b5c1c65d9a52af31cc1" exitCode=0 Oct 30 12:52:23.685091 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.684977 5922 generic.go:356] "Generic (PLEG): container finished" podID="ee23e57d-7b77-4dba-95a5-84ad59133981" containerID="e6077a088ebe4035b199c3a7d23af9d47cf57a32ab3c7a37135ca5a33196bd00" exitCode=0 Oct 30 12:52:23.692074 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.692053 5922 generic.go:356] "Generic (PLEG): container finished" podID="ecd3d4d5-a59a-4b0e-a981-956b10d08531" containerID="2bdb163e8f97e8e5afd18e9046bd2c976fa179cc65c409ed26d34fa58f041ab4" exitCode=0 Oct 30 12:52:23.725556 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.725510 5922 manager.go:324] Recovery completed Oct 30 12:52:23.737126 ip-10-0-137-21 kubenswrapper[5922]: E1030 12:52:23.737101 5922 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 12:52:23.748119 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.748099 5922 cpu_manager.go:226] "Starting" policy="none" Oct 30 12:52:23.748119 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.748115 5922 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 12:52:23.748253 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.748133 5922 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 12:52:23.748348 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.748274 5922 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 12:52:23.748348 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.748288 5922 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 12:52:23.748348 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.748313 5922 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 12:52:23.748348 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.748322 5922 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 12:52:23.748348 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.748329 5922 policy_none.go:50] "Start" Oct 30 12:52:23.748348 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.748341 5922 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 12:52:23.748645 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.748355 5922 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 12:52:23.748645 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.748542 5922 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 12:52:23.748645 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.748553 5922 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 12:52:23.748645 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.748561 5922 policy_none.go:44] "Start" Oct 30 12:52:23.754485 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.754459 5922 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 12:52:23.757031 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.757014 5922 manager.go:341] "Starting Device Plugin manager" Oct 30 12:52:23.757566 ip-10-0-137-21 kubenswrapper[5922]: E1030 12:52:23.757052 5922 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 12:52:23.757605 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.757571 5922 server.go:87] "Starting device plugin registration server" Oct 30 12:52:23.757842 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.757832 5922 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 12:52:23.757873 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.757843 5922 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 12:52:23.757941 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.757925 5922 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 12:52:23.758052 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.758017 5922 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 12:52:23.758052 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.758063 5922 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 12:52:23.758199 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.758070 5922 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 12:52:23.758199 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.758105 5922 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T12:52:23.758047855Z","UUID":"48137603-87b1-4d5a-9409-5205b4335dea","Handler":null,"Name":"","Endpoint":""} Oct 30 12:52:23.759793 ip-10-0-137-21 kubenswrapper[5922]: E1030 12:52:23.759774 5922 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 12:52:23.760160 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.760142 5922 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 12:52:23.760231 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.760168 5922 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 12:52:23.858348 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.858319 5922 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:52:23.860115 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.860096 5922 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-21.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:52:23.860191 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.860123 5922 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-21.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:52:23.860191 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.860132 5922 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-21.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:52:23.862817 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.860479 5922 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-21.ec2.internal" Oct 30 12:52:23.897993 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.897974 5922 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-137-21.ec2.internal" Oct 30 12:52:23.898062 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.898032 5922 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-21.ec2.internal" Oct 30 12:52:23.938073 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.938031 5922 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-21.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-21.ec2.internal"] Oct 30 12:52:23.938658 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.938590 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-21.ec2.internal" event={"ID":"c7d15293c60c5848ccd82e7130b45838","Type":"ContainerStarted","Data":"710b176a6be7fb03f6faa77f6d2b4d3eb0b08e2a53531865a612ade2f6b8631a"} Oct 30 12:52:23.938775 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.938665 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-21.ec2.internal" event={"ID":"c7d15293c60c5848ccd82e7130b45838","Type":"ContainerDied","Data":"3cdd4b7edc10db5ddf3920923ece442e41dbb7163b81a835bf9d485f1fbd0f0c"} Oct 30 12:52:23.938775 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.938671 5922 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-21.ec2.internal" Oct 30 12:52:23.938775 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.938681 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-21.ec2.internal" event={"ID":"c7d15293c60c5848ccd82e7130b45838","Type":"ContainerStarted","Data":"cfaf6eb0e32f16cf0289059ab69c1e2d5690928fa9d7a16ea1a6cecf114b053c"} Oct 30 12:52:23.938923 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.938819 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-21.ec2.internal" event={"ID":"100438b9fc00ed25c10ae1227a1de159","Type":"ContainerStarted","Data":"32809343fc4d53e65653c04b2a4bdb750b6595098b05ab4b1020bb6d4cfc7deb"} Oct 30 12:52:23.938923 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.938834 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-21.ec2.internal" event={"ID":"100438b9fc00ed25c10ae1227a1de159","Type":"ContainerStarted","Data":"c8edd7fed57efb3f2fe9eb4b5689338b4c619b94606f3223fac094499fd99a54"} Oct 30 12:52:23.938923 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.938876 5922 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-21.ec2.internal" Oct 30 12:52:23.966832 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.966813 5922 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 12:52:23.966908 ip-10-0-137-21 kubenswrapper[5922]: E1030 12:52:23.966848 5922 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-21.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-21.ec2.internal" Oct 30 12:52:23.968001 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:23.967989 5922 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 12:52:23.968041 ip-10-0-137-21 kubenswrapper[5922]: E1030 12:52:23.968020 5922 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-21.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-21.ec2.internal" Oct 30 12:52:24.059302 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.059252 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7d15293c60c5848ccd82e7130b45838-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-21.ec2.internal\" (UID: \"c7d15293c60c5848ccd82e7130b45838\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-21.ec2.internal" Oct 30 12:52:24.059302 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.059276 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/100438b9fc00ed25c10ae1227a1de159-config\") pod \"kube-apiserver-proxy-ip-10-0-137-21.ec2.internal\" (UID: \"100438b9fc00ed25c10ae1227a1de159\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-21.ec2.internal" Oct 30 12:52:24.059302 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.059294 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c7d15293c60c5848ccd82e7130b45838-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-21.ec2.internal\" (UID: \"c7d15293c60c5848ccd82e7130b45838\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-21.ec2.internal" Oct 30 12:52:24.159750 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.159723 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/100438b9fc00ed25c10ae1227a1de159-config\") pod \"kube-apiserver-proxy-ip-10-0-137-21.ec2.internal\" (UID: \"100438b9fc00ed25c10ae1227a1de159\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-21.ec2.internal" Oct 30 12:52:24.159834 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.159758 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c7d15293c60c5848ccd82e7130b45838-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-21.ec2.internal\" (UID: \"c7d15293c60c5848ccd82e7130b45838\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-21.ec2.internal" Oct 30 12:52:24.159834 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.159776 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7d15293c60c5848ccd82e7130b45838-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-21.ec2.internal\" (UID: \"c7d15293c60c5848ccd82e7130b45838\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-21.ec2.internal" Oct 30 12:52:24.159834 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.159823 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7d15293c60c5848ccd82e7130b45838-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-21.ec2.internal\" (UID: \"c7d15293c60c5848ccd82e7130b45838\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-21.ec2.internal" Oct 30 12:52:24.159834 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.159827 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/100438b9fc00ed25c10ae1227a1de159-config\") pod \"kube-apiserver-proxy-ip-10-0-137-21.ec2.internal\" (UID: \"100438b9fc00ed25c10ae1227a1de159\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-21.ec2.internal" Oct 30 12:52:24.159951 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.159840 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c7d15293c60c5848ccd82e7130b45838-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-21.ec2.internal\" (UID: \"c7d15293c60c5848ccd82e7130b45838\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-21.ec2.internal" Oct 30 12:52:24.592401 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.592379 5922 apiserver.go:52] "Watching apiserver" Oct 30 12:52:24.645650 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.645605 5922 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 12:52:24.646349 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.646319 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-cfgs4","openshift-monitoring/prometheus-operator-9cd89b74c-xxb2q","openshift-multus/network-metrics-daemon-tpz8b","openshift-network-operator/iptables-alerter-zvw7s","openshift-monitoring/monitoring-plugin-77f4f76594-spsvf","openshift-monitoring/node-exporter-c4jgj","openshift-monitoring/prometheus-k8s-0","openshift-ovn-kubernetes/ovnkube-node-nwrx6","openshift-monitoring/alertmanager-main-0","kube-system/global-pull-secret-syncer-s7fsp","openshift-console/console-689bbd6cb4-lmgqn","openshift-monitoring/telemeter-client-76d946b6b9-5lvf8","openshift-multus/multus-sw8rj","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66","openshift-dns/node-resolver-9szkr","openshift-cluster-node-tuning-operator/tuned-k76bj","openshift-dns/dns-default-v7skv","openshift-monitoring/kube-state-metrics-5dc8d7d89b-4tq96","openshift-network-diagnostics/network-check-target-95n9l","kube-system/konnectivity-agent-2nqhp","openshift-console/console-85cbd9968c-p6js7","openshift-multus/multus-additional-cni-plugins-75bqd","openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm","openshift-ingress-canary/ingress-canary-854v4","kube-system/kube-apiserver-proxy-ip-10-0-137-21.ec2.internal","openshift-image-registry/node-ca-bcbg8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-21.ec2.internal"] Oct 30 12:52:24.663272 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.663247 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-68fxr\"" Oct 30 12:52:24.663376 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.663335 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 12:52:24.663376 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.663343 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 12:52:24.663376 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.663345 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-v4xk8\"" Oct 30 12:52:24.663376 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.663360 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-87w8v\"" Oct 30 12:52:24.663376 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.663381 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hszmd\"" Oct 30 12:52:24.663772 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.663755 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-48745\"" Oct 30 12:52:24.664093 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.664078 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 12:52:24.664175 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.664081 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 12:52:24.664175 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.664081 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 12:52:24.664175 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.664108 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 12:52:24.664175 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.664159 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 12:52:24.664175 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.664166 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 12:52:24.664439 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.664425 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7j54b\"" Oct 30 12:52:24.665210 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.665191 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 12:52:24.665210 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.665201 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 12:52:24.665210 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.665209 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 12:52:24.665339 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.665212 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 12:52:24.665339 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.665211 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 12:52:24.665339 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.665210 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:52:24.665431 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.665382 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 12:52:24.665431 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.665415 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 12:52:24.665484 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.665434 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 12:52:24.672870 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.672852 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 12:52:24.672924 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.672874 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 12:52:24.673086 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.673073 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 12:52:24.673086 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.673081 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 12:52:24.673831 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.673818 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 12:52:24.674022 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.674003 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 12:52:24.674293 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.674279 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 12:52:24.674577 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.674565 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 12:52:24.674825 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.674803 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 12:52:24.674891 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.674840 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-bhhxq\"" Oct 30 12:52:24.674928 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.674887 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qjjfj\"" Oct 30 12:52:24.675104 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.675090 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-wqgtx\"" Oct 30 12:52:24.675147 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.675126 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 12:52:24.675340 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.675329 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 12:52:24.676329 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.676318 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 12:52:24.676901 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.676888 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 12:52:24.677493 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.677473 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 12:52:24.677493 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.677489 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 12:52:24.677582 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.677494 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-m2lrf\"" Oct 30 12:52:24.677582 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.677500 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 12:52:24.677582 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.677493 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 12:52:24.677582 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.677577 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-vlkp5\"" Oct 30 12:52:24.677748 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.677665 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 12:52:24.677748 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.677695 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 12:52:24.677748 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.677728 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ktwvb\"" Oct 30 12:52:24.677887 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.677747 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 12:52:24.677887 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.677747 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 12:52:24.677887 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.677766 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 12:52:24.677887 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.677785 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 12:52:24.677887 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.677798 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 12:52:24.678039 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.677892 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 12:52:24.678039 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.677905 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 12:52:24.678039 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.678029 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 12:52:24.678267 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.678250 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 12:52:24.678550 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.678524 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:52:24.687489 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.687466 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 12:52:24.687628 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.687506 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 12:52:24.687628 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.687519 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 12:52:24.687628 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.687550 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 12:52:24.687889 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.687700 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 12:52:24.687889 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.687769 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 12:52:24.687889 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.687816 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 12:52:24.687889 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.687862 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 12:52:24.687889 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.687866 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 12:52:24.688875 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.688854 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 12:52:24.688982 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.688886 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 12:52:24.688982 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.688900 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 12:52:24.688982 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.688908 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 12:52:24.688982 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.688930 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 12:52:24.688982 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.688962 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 12:52:24.688982 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.688974 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 12:52:24.689235 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.689008 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 12:52:24.689235 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.689015 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 12:52:24.689235 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.689121 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 12:52:24.689401 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.689388 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 12:52:24.689447 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.689403 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 12:52:24.689447 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.689433 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 12:52:24.689513 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.689446 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 12:52:24.689985 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.689966 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 12:52:24.690415 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.689996 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 12:52:24.690415 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.690053 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 12:52:24.690415 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.690085 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 12:52:24.690415 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.690104 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 12:52:24.690415 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.690120 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 12:52:24.690415 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.690154 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-sfhnn\"" Oct 30 12:52:24.690415 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.690159 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9ppdm\"" Oct 30 12:52:24.690415 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.690185 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 12:52:24.690415 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.690201 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 12:52:24.690415 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.690207 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 12:52:24.690415 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.690212 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rwc4t\"" Oct 30 12:52:24.690415 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.690239 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 12:52:24.690415 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.690244 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-khs4n\"" Oct 30 12:52:24.690415 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.690205 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 12:52:24.690415 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.690243 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-5p7wp\"" Oct 30 12:52:24.690415 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.690271 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 12:52:24.690415 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.690297 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 12:52:24.690415 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.690298 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7dvvl\"" Oct 30 12:52:24.690415 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.690389 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-45hdr\"" Oct 30 12:52:24.691558 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.690469 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 12:52:24.691558 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.690502 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xj7mb\"" Oct 30 12:52:24.691558 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.690502 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-7j6fw\"" Oct 30 12:52:24.691558 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.690532 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 12:52:24.691558 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.690544 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 12:52:24.691558 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.690534 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5rv5r3ic93mpd\"" Oct 30 12:52:24.691558 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.690702 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2t7nptcd6h1f5\"" Oct 30 12:52:24.691558 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.690770 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 12:52:24.691558 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.690861 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-w4qng\"" Oct 30 12:52:24.691558 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.691106 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-66wcp\"" Oct 30 12:52:24.691558 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.691287 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 12:52:24.691558 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.691402 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5s6kd\"" Oct 30 12:52:24.692020 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.691764 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 12:52:24.692020 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.691896 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 12:52:24.694665 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.694644 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 12:52:24.697691 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.695345 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 12:52:24.699516 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.699494 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 12:52:24.699738 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.699717 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 12:52:24.709098 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.709081 5922 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-21.ec2.internal" Oct 30 12:52:24.709259 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.709243 5922 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-21.ec2.internal" Oct 30 12:52:24.710087 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.710068 5922 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 12:52:24.731424 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.731399 5922 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 12:52:24.731521 ip-10-0-137-21 kubenswrapper[5922]: E1030 12:52:24.731455 5922 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-21.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-21.ec2.internal" Oct 30 12:52:24.732421 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.732406 5922 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 12:52:24.732507 ip-10-0-137-21 kubenswrapper[5922]: E1030 12:52:24.732439 5922 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-21.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-21.ec2.internal" Oct 30 12:52:24.761456 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.761438 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01873344-73c8-47b2-b6ba-8e338ba18315-oauth-serving-cert\") pod \"console-689bbd6cb4-lmgqn\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:24.761542 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.761465 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-etc-kubernetes\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.761542 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.761484 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe19a15d-5112-4b76-a226-4b8462e2fb33-tmp-dir\") pod \"node-resolver-9szkr\" (UID: \"fe19a15d-5112-4b76-a226-4b8462e2fb33\") " pod="openshift-dns/node-resolver-9szkr" Oct 30 12:52:24.761542 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.761499 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m944\" (UniqueName: \"kubernetes.io/projected/4dca982f-f509-454b-a057-d7fd9b3f34bd-kube-api-access-5m944\") pod \"insights-runtime-extractor-cfgs4\" (UID: \"4dca982f-f509-454b-a057-d7fd9b3f34bd\") " pod="openshift-insights/insights-runtime-extractor-cfgs4" Oct 30 12:52:24.761542 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.761519 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-run-openvswitch\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.761691 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.761574 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp4zz\" (UniqueName: \"kubernetes.io/projected/d9b03988-7080-4352-ba1e-08ee68ba3df8-kube-api-access-qp4zz\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.761691 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.761594 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe19a15d-5112-4b76-a226-4b8462e2fb33-tmp-dir\") pod \"node-resolver-9szkr\" (UID: \"fe19a15d-5112-4b76-a226-4b8462e2fb33\") " pod="openshift-dns/node-resolver-9szkr" Oct 30 12:52:24.761691 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.761645 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab451161-af25-47e2-860e-f7d5186d4491-cert\") pod \"ingress-canary-854v4\" (UID: \"ab451161-af25-47e2-860e-f7d5186d4491\") " pod="openshift-ingress-canary/ingress-canary-854v4" Oct 30 12:52:24.761771 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.761697 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-system-cni-dir\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.761771 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.761716 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssl6b\" (UniqueName: \"kubernetes.io/projected/f89a7014-2966-43c7-a901-709c690b0003-kube-api-access-ssl6b\") pod \"console-85cbd9968c-p6js7\" (UID: \"f89a7014-2966-43c7-a901-709c690b0003\") " pod="openshift-console/console-85cbd9968c-p6js7" Oct 30 12:52:24.761771 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.761752 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0992e81-44ba-41c3-8079-c801a2d40aa4-serving-certs-ca-bundle\") pod \"telemeter-client-76d946b6b9-5lvf8\" (UID: \"b0992e81-44ba-41c3-8079-c801a2d40aa4\") " pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" Oct 30 12:52:24.761866 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.761772 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ecd3d4d5-a59a-4b0e-a981-956b10d08531-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.761866 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.761837 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01873344-73c8-47b2-b6ba-8e338ba18315-oauth-serving-cert\") pod \"console-689bbd6cb4-lmgqn\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:24.761926 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.761887 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.762021 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.762005 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ecd3d4d5-a59a-4b0e-a981-956b10d08531-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.762078 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.762036 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8fd0fdb6-c254-446d-8832-5a2fb284f03d-node-exporter-wtmp\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:24.762126 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.762078 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bea99b12-e97c-4c0e-9360-07e542a4294d-sys-fs\") pod \"aws-ebs-csi-driver-node-dsh66\" (UID: \"bea99b12-e97c-4c0e-9360-07e542a4294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" Oct 30 12:52:24.762126 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.762095 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab451161-af25-47e2-860e-f7d5186d4491-cert\") pod \"ingress-canary-854v4\" (UID: \"ab451161-af25-47e2-860e-f7d5186d4491\") " pod="openshift-ingress-canary/ingress-canary-854v4" Oct 30 12:52:24.762126 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.762115 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/56928a4d-7923-46e6-97bb-6690514f89a4-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-xxb2q\" (UID: \"56928a4d-7923-46e6-97bb-6690514f89a4\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xxb2q" Oct 30 12:52:24.762260 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.762208 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-web-config\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.762260 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.762246 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd3d929e-3197-4794-9b31-794427a55693-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-4tq96\" (UID: \"dd3d929e-3197-4794-9b31-794427a55693\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4tq96" Oct 30 12:52:24.762329 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.762279 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0992e81-44ba-41c3-8079-c801a2d40aa4-serving-certs-ca-bundle\") pod \"telemeter-client-76d946b6b9-5lvf8\" (UID: \"b0992e81-44ba-41c3-8079-c801a2d40aa4\") " pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" Oct 30 12:52:24.762364 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.762338 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhkpc\" (UniqueName: \"kubernetes.io/projected/c44f520d-f602-40e0-94d4-23277e56d67e-kube-api-access-jhkpc\") pod \"metrics-server-6d7f54bf5f-jkjtm\" (UID: \"c44f520d-f602-40e0-94d4-23277e56d67e\") " pod="openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm" Oct 30 12:52:24.762364 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.762355 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-multus-daemon-config\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.762457 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.762403 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/56928a4d-7923-46e6-97bb-6690514f89a4-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-xxb2q\" (UID: \"56928a4d-7923-46e6-97bb-6690514f89a4\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xxb2q" Oct 30 12:52:24.762457 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.762408 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-web-config\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.762457 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.762422 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-etc-modprobe-d\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:52:24.762592 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.762480 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b0992e81-44ba-41c3-8079-c801a2d40aa4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76d946b6b9-5lvf8\" (UID: \"b0992e81-44ba-41c3-8079-c801a2d40aa4\") " pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" Oct 30 12:52:24.762592 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.762532 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d9b03988-7080-4352-ba1e-08ee68ba3df8-ovnkube-config\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.762592 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.762562 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8xd7\" (UniqueName: \"kubernetes.io/projected/dd3d929e-3197-4794-9b31-794427a55693-kube-api-access-m8xd7\") pod \"kube-state-metrics-5dc8d7d89b-4tq96\" (UID: \"dd3d929e-3197-4794-9b31-794427a55693\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4tq96" Oct 30 12:52:24.762753 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.762593 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f89a7014-2966-43c7-a901-709c690b0003-service-ca\") pod \"console-85cbd9968c-p6js7\" (UID: \"f89a7014-2966-43c7-a901-709c690b0003\") " pod="openshift-console/console-85cbd9968c-p6js7" Oct 30 12:52:24.762753 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.762607 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-multus-daemon-config\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.762753 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.762634 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8fd0fdb6-c254-446d-8832-5a2fb284f03d-node-exporter-tls\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:24.762753 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.762639 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd3d929e-3197-4794-9b31-794427a55693-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-4tq96\" (UID: \"dd3d929e-3197-4794-9b31-794427a55693\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4tq96" Oct 30 12:52:24.762753 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.762664 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bea99b12-e97c-4c0e-9360-07e542a4294d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dsh66\" (UID: \"bea99b12-e97c-4c0e-9360-07e542a4294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" Oct 30 12:52:24.762753 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.762685 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b0992e81-44ba-41c3-8079-c801a2d40aa4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76d946b6b9-5lvf8\" (UID: \"b0992e81-44ba-41c3-8079-c801a2d40aa4\") " pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" Oct 30 12:52:24.762753 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.762727 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-host-kubelet\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.763005 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.762757 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-host-slash\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.763005 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.762768 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d9b03988-7080-4352-ba1e-08ee68ba3df8-ovnkube-config\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.763005 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.762824 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f89a7014-2966-43c7-a901-709c690b0003-service-ca\") pod \"console-85cbd9968c-p6js7\" (UID: \"f89a7014-2966-43c7-a901-709c690b0003\") " pod="openshift-console/console-85cbd9968c-p6js7" Oct 30 12:52:24.763005 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.762825 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.763005 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.762876 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee23e57d-7b77-4dba-95a5-84ad59133981-cni-binary-copy\") pod \"multus-additional-cni-plugins-75bqd\" (UID: \"ee23e57d-7b77-4dba-95a5-84ad59133981\") " pod="openshift-multus/multus-additional-cni-plugins-75bqd" Oct 30 12:52:24.763005 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.762928 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w592\" (UniqueName: \"kubernetes.io/projected/01873344-73c8-47b2-b6ba-8e338ba18315-kube-api-access-2w592\") pod \"console-689bbd6cb4-lmgqn\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:24.763005 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.762938 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8fd0fdb6-c254-446d-8832-5a2fb284f03d-node-exporter-tls\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:24.763005 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.762984 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lltjs\" (UniqueName: \"kubernetes.io/projected/b0992e81-44ba-41c3-8079-c801a2d40aa4-kube-api-access-lltjs\") pod \"telemeter-client-76d946b6b9-5lvf8\" (UID: \"b0992e81-44ba-41c3-8079-c801a2d40aa4\") " pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" Oct 30 12:52:24.763213 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.763070 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee23e57d-7b77-4dba-95a5-84ad59133981-cni-binary-copy\") pod \"multus-additional-cni-plugins-75bqd\" (UID: \"ee23e57d-7b77-4dba-95a5-84ad59133981\") " pod="openshift-multus/multus-additional-cni-plugins-75bqd" Oct 30 12:52:24.763213 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.763091 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.763213 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.763203 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.763308 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.763222 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/14c0845c-1947-44b0-ada5-4b68ff9974f2-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-spsvf\" (UID: \"14c0845c-1947-44b0-ada5-4b68ff9974f2\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-spsvf" Oct 30 12:52:24.763308 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.763266 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwddx\" (UniqueName: \"kubernetes.io/projected/ee23e57d-7b77-4dba-95a5-84ad59133981-kube-api-access-hwddx\") pod \"multus-additional-cni-plugins-75bqd\" (UID: \"ee23e57d-7b77-4dba-95a5-84ad59133981\") " pod="openshift-multus/multus-additional-cni-plugins-75bqd" Oct 30 12:52:24.763382 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.763367 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.763431 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.763393 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8fd0fdb6-c254-446d-8832-5a2fb284f03d-root\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:24.763431 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.763413 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.763431 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.763419 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/14c0845c-1947-44b0-ada5-4b68ff9974f2-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-spsvf\" (UID: \"14c0845c-1947-44b0-ada5-4b68ff9974f2\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-spsvf" Oct 30 12:52:24.763560 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.763424 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8fd0fdb6-c254-446d-8832-5a2fb284f03d-node-exporter-textfile\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:24.763560 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.763487 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c44f520d-f602-40e0-94d4-23277e56d67e-audit-log\") pod \"metrics-server-6d7f54bf5f-jkjtm\" (UID: \"c44f520d-f602-40e0-94d4-23277e56d67e\") " pod="openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm" Oct 30 12:52:24.763560 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.763493 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8fd0fdb6-c254-446d-8832-5a2fb284f03d-node-exporter-textfile\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:24.763560 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.763493 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.763560 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.763539 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c44f520d-f602-40e0-94d4-23277e56d67e-audit-log\") pod \"metrics-server-6d7f54bf5f-jkjtm\" (UID: \"c44f520d-f602-40e0-94d4-23277e56d67e\") " pod="openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm" Oct 30 12:52:24.763560 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.763542 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01873344-73c8-47b2-b6ba-8e338ba18315-console-oauth-config\") pod \"console-689bbd6cb4-lmgqn\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:24.763802 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.763575 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqvdp\" (UniqueName: \"kubernetes.io/projected/1bb48bfa-827b-4179-b9fa-418fc14f0f44-kube-api-access-dqvdp\") pod \"network-check-target-95n9l\" (UID: \"1bb48bfa-827b-4179-b9fa-418fc14f0f44\") " pod="openshift-network-diagnostics/network-check-target-95n9l" Oct 30 12:52:24.763802 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.763590 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/accde534-15e4-42cf-94df-c0aa085f0603-etc-tuned\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:52:24.763802 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.763628 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51a542f5-796e-46b1-95bd-31751d50b52b-tmp-dir\") pod \"dns-default-v7skv\" (UID: \"51a542f5-796e-46b1-95bd-31751d50b52b\") " pod="openshift-dns/dns-default-v7skv" Oct 30 12:52:24.763802 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.763690 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.763802 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.763698 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/accde534-15e4-42cf-94df-c0aa085f0603-etc-tuned\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:52:24.763802 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.763718 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51a542f5-796e-46b1-95bd-31751d50b52b-tmp-dir\") pod \"dns-default-v7skv\" (UID: \"51a542f5-796e-46b1-95bd-31751d50b52b\") " pod="openshift-dns/dns-default-v7skv" Oct 30 12:52:24.763802 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.763719 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fe19a15d-5112-4b76-a226-4b8462e2fb33-hosts-file\") pod \"node-resolver-9szkr\" (UID: \"fe19a15d-5112-4b76-a226-4b8462e2fb33\") " pod="openshift-dns/node-resolver-9szkr" Oct 30 12:52:24.763802 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.763756 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.763802 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.763756 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fd0fdb6-c254-446d-8832-5a2fb284f03d-metrics-client-ca\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:24.763802 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.763789 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3f38af1e-cd92-4cd8-a643-151b10b8da12-konnectivity-ca\") pod \"konnectivity-agent-2nqhp\" (UID: \"3f38af1e-cd92-4cd8-a643-151b10b8da12\") " pod="kube-system/konnectivity-agent-2nqhp" Oct 30 12:52:24.763802 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.763805 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0992e81-44ba-41c3-8079-c801a2d40aa4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76d946b6b9-5lvf8\" (UID: \"b0992e81-44ba-41c3-8079-c801a2d40aa4\") " pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" Oct 30 12:52:24.764265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.763806 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01873344-73c8-47b2-b6ba-8e338ba18315-console-oauth-config\") pod \"console-689bbd6cb4-lmgqn\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:24.764265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.763820 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0992e81-44ba-41c3-8079-c801a2d40aa4-metrics-client-ca\") pod \"telemeter-client-76d946b6b9-5lvf8\" (UID: \"b0992e81-44ba-41c3-8079-c801a2d40aa4\") " pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" Oct 30 12:52:24.764265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.763843 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.764265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.763860 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8fd0fdb6-c254-446d-8832-5a2fb284f03d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:24.764265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.763876 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/56928a4d-7923-46e6-97bb-6690514f89a4-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-xxb2q\" (UID: \"56928a4d-7923-46e6-97bb-6690514f89a4\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xxb2q" Oct 30 12:52:24.764265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.763893 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd3d929e-3197-4794-9b31-794427a55693-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-4tq96\" (UID: \"dd3d929e-3197-4794-9b31-794427a55693\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4tq96" Oct 30 12:52:24.764265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.763905 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fd0fdb6-c254-446d-8832-5a2fb284f03d-metrics-client-ca\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:24.764265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.763924 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01873344-73c8-47b2-b6ba-8e338ba18315-console-config\") pod \"console-689bbd6cb4-lmgqn\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:24.764265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.763969 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3f38af1e-cd92-4cd8-a643-151b10b8da12-konnectivity-ca\") pod \"konnectivity-agent-2nqhp\" (UID: \"3f38af1e-cd92-4cd8-a643-151b10b8da12\") " pod="kube-system/konnectivity-agent-2nqhp" Oct 30 12:52:24.764265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.764011 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f89a7014-2966-43c7-a901-709c690b0003-console-oauth-config\") pod \"console-85cbd9968c-p6js7\" (UID: \"f89a7014-2966-43c7-a901-709c690b0003\") " pod="openshift-console/console-85cbd9968c-p6js7" Oct 30 12:52:24.764265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.764070 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.764265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.764087 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8fd0fdb6-c254-446d-8832-5a2fb284f03d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:24.764265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.764091 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0992e81-44ba-41c3-8079-c801a2d40aa4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76d946b6b9-5lvf8\" (UID: \"b0992e81-44ba-41c3-8079-c801a2d40aa4\") " pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" Oct 30 12:52:24.764265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.764117 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/56928a4d-7923-46e6-97bb-6690514f89a4-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-xxb2q\" (UID: \"56928a4d-7923-46e6-97bb-6690514f89a4\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xxb2q" Oct 30 12:52:24.764265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.764125 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01873344-73c8-47b2-b6ba-8e338ba18315-console-config\") pod \"console-689bbd6cb4-lmgqn\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:24.764265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.764103 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.764265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.764095 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4dca982f-f509-454b-a057-d7fd9b3f34bd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cfgs4\" (UID: \"4dca982f-f509-454b-a057-d7fd9b3f34bd\") " pod="openshift-insights/insights-runtime-extractor-cfgs4" Oct 30 12:52:24.764265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.764166 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4dca982f-f509-454b-a057-d7fd9b3f34bd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cfgs4\" (UID: \"4dca982f-f509-454b-a057-d7fd9b3f34bd\") " pod="openshift-insights/insights-runtime-extractor-cfgs4" Oct 30 12:52:24.764265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.764183 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.764265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.764207 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-node-log\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.764265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.764232 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxwlx\" (UniqueName: \"kubernetes.io/projected/1fcccc7b-a83c-4c52-a648-cb549e70b54d-kube-api-access-xxwlx\") pod \"network-metrics-daemon-tpz8b\" (UID: \"1fcccc7b-a83c-4c52-a648-cb549e70b54d\") " pod="openshift-multus/network-metrics-daemon-tpz8b" Oct 30 12:52:24.764265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.764258 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-etc-sysconfig\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:52:24.764265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.764267 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f89a7014-2966-43c7-a901-709c690b0003-console-oauth-config\") pod \"console-85cbd9968c-p6js7\" (UID: \"f89a7014-2966-43c7-a901-709c690b0003\") " pod="openshift-console/console-85cbd9968c-p6js7" Oct 30 12:52:24.765265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.764279 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0992e81-44ba-41c3-8079-c801a2d40aa4-metrics-client-ca\") pod \"telemeter-client-76d946b6b9-5lvf8\" (UID: \"b0992e81-44ba-41c3-8079-c801a2d40aa4\") " pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" Oct 30 12:52:24.765265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.764409 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4dca982f-f509-454b-a057-d7fd9b3f34bd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cfgs4\" (UID: \"4dca982f-f509-454b-a057-d7fd9b3f34bd\") " pod="openshift-insights/insights-runtime-extractor-cfgs4" Oct 30 12:52:24.765265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.764339 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4dca982f-f509-454b-a057-d7fd9b3f34bd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cfgs4\" (UID: \"4dca982f-f509-454b-a057-d7fd9b3f34bd\") " pod="openshift-insights/insights-runtime-extractor-cfgs4" Oct 30 12:52:24.765265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.764372 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.765265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.764395 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd3d929e-3197-4794-9b31-794427a55693-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-4tq96\" (UID: \"dd3d929e-3197-4794-9b31-794427a55693\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4tq96" Oct 30 12:52:24.765265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.764329 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b0992e81-44ba-41c3-8079-c801a2d40aa4-secret-telemeter-client\") pod \"telemeter-client-76d946b6b9-5lvf8\" (UID: \"b0992e81-44ba-41c3-8079-c801a2d40aa4\") " pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" Oct 30 12:52:24.765265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.764518 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b0992e81-44ba-41c3-8079-c801a2d40aa4-telemeter-client-tls\") pod \"telemeter-client-76d946b6b9-5lvf8\" (UID: \"b0992e81-44ba-41c3-8079-c801a2d40aa4\") " pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" Oct 30 12:52:24.765265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.764540 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8fhd\" (UniqueName: \"kubernetes.io/projected/bea99b12-e97c-4c0e-9360-07e542a4294d-kube-api-access-m8fhd\") pod \"aws-ebs-csi-driver-node-dsh66\" (UID: \"bea99b12-e97c-4c0e-9360-07e542a4294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" Oct 30 12:52:24.765265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.764545 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.765265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.764550 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b0992e81-44ba-41c3-8079-c801a2d40aa4-secret-telemeter-client\") pod \"telemeter-client-76d946b6b9-5lvf8\" (UID: \"b0992e81-44ba-41c3-8079-c801a2d40aa4\") " pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" Oct 30 12:52:24.765265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.764557 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-etc-openvswitch\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.765265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.764573 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbr4h\" (UniqueName: \"kubernetes.io/projected/ab451161-af25-47e2-860e-f7d5186d4491-kube-api-access-gbr4h\") pod \"ingress-canary-854v4\" (UID: \"ab451161-af25-47e2-860e-f7d5186d4491\") " pod="openshift-ingress-canary/ingress-canary-854v4" Oct 30 12:52:24.765265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.764715 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01873344-73c8-47b2-b6ba-8e338ba18315-trusted-ca-bundle\") pod \"console-689bbd6cb4-lmgqn\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:24.765265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.764774 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-multus-cni-dir\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.765265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.764773 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b0992e81-44ba-41c3-8079-c801a2d40aa4-telemeter-client-tls\") pod \"telemeter-client-76d946b6b9-5lvf8\" (UID: \"b0992e81-44ba-41c3-8079-c801a2d40aa4\") " pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" Oct 30 12:52:24.765265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.764812 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f89a7014-2966-43c7-a901-709c690b0003-tmp\") pod \"console-85cbd9968c-p6js7\" (UID: \"f89a7014-2966-43c7-a901-709c690b0003\") " pod="openshift-console/console-85cbd9968c-p6js7" Oct 30 12:52:24.765265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.764861 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.765265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.764890 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dd3d929e-3197-4794-9b31-794427a55693-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-4tq96\" (UID: \"dd3d929e-3197-4794-9b31-794427a55693\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4tq96" Oct 30 12:52:24.765265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.764907 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fcccc7b-a83c-4c52-a648-cb549e70b54d-metrics-certs\") pod \"network-metrics-daemon-tpz8b\" (UID: \"1fcccc7b-a83c-4c52-a648-cb549e70b54d\") " pod="openshift-multus/network-metrics-daemon-tpz8b" Oct 30 12:52:24.765265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.764927 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ee23e57d-7b77-4dba-95a5-84ad59133981-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-75bqd\" (UID: \"ee23e57d-7b77-4dba-95a5-84ad59133981\") " pod="openshift-multus/multus-additional-cni-plugins-75bqd" Oct 30 12:52:24.765265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.764866 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f89a7014-2966-43c7-a901-709c690b0003-tmp\") pod \"console-85cbd9968c-p6js7\" (UID: \"f89a7014-2966-43c7-a901-709c690b0003\") " pod="openshift-console/console-85cbd9968c-p6js7" Oct 30 12:52:24.765265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.764950 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01873344-73c8-47b2-b6ba-8e338ba18315-service-ca\") pod \"console-689bbd6cb4-lmgqn\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:24.765265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.764976 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-multus-socket-dir-parent\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.765265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.764979 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.765265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.765004 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01873344-73c8-47b2-b6ba-8e338ba18315-trusted-ca-bundle\") pod \"console-689bbd6cb4-lmgqn\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:24.765265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.765019 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/51a542f5-796e-46b1-95bd-31751d50b52b-metrics-tls\") pod \"dns-default-v7skv\" (UID: \"51a542f5-796e-46b1-95bd-31751d50b52b\") " pod="openshift-dns/dns-default-v7skv" Oct 30 12:52:24.765265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.765048 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.765265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.765078 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v6nm\" (UniqueName: \"kubernetes.io/projected/56928a4d-7923-46e6-97bb-6690514f89a4-kube-api-access-9v6nm\") pod \"prometheus-operator-9cd89b74c-xxb2q\" (UID: \"56928a4d-7923-46e6-97bb-6690514f89a4\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xxb2q" Oct 30 12:52:24.765265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.765109 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvdzg\" (UniqueName: \"kubernetes.io/projected/14c0845c-1947-44b0-ada5-4b68ff9974f2-kube-api-access-wvdzg\") pod \"monitoring-plugin-77f4f76594-spsvf\" (UID: \"14c0845c-1947-44b0-ada5-4b68ff9974f2\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-spsvf" Oct 30 12:52:24.765265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.765124 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dd3d929e-3197-4794-9b31-794427a55693-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-4tq96\" (UID: \"dd3d929e-3197-4794-9b31-794427a55693\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4tq96" Oct 30 12:52:24.765265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.765133 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-cni-binary-copy\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.765265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.765142 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fcccc7b-a83c-4c52-a648-cb549e70b54d-metrics-certs\") pod \"network-metrics-daemon-tpz8b\" (UID: \"1fcccc7b-a83c-4c52-a648-cb549e70b54d\") " pod="openshift-multus/network-metrics-daemon-tpz8b" Oct 30 12:52:24.765265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.765150 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01873344-73c8-47b2-b6ba-8e338ba18315-service-ca\") pod \"console-689bbd6cb4-lmgqn\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:24.765265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.765132 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ee23e57d-7b77-4dba-95a5-84ad59133981-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-75bqd\" (UID: \"ee23e57d-7b77-4dba-95a5-84ad59133981\") " pod="openshift-multus/multus-additional-cni-plugins-75bqd" Oct 30 12:52:24.765265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.765160 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9x52\" (UniqueName: \"kubernetes.io/projected/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-kube-api-access-x9x52\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.765265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.765188 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/51a542f5-796e-46b1-95bd-31751d50b52b-metrics-tls\") pod \"dns-default-v7skv\" (UID: \"51a542f5-796e-46b1-95bd-31751d50b52b\") " pod="openshift-dns/dns-default-v7skv" Oct 30 12:52:24.765265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.765188 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.765265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.765235 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkldf\" (UniqueName: \"kubernetes.io/projected/11a0b508-2f9c-4be6-b5c7-b3d74766e801-kube-api-access-tkldf\") pod \"iptables-alerter-zvw7s\" (UID: \"11a0b508-2f9c-4be6-b5c7-b3d74766e801\") " pod="openshift-network-operator/iptables-alerter-zvw7s" Oct 30 12:52:24.765265 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.765265 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-host-run-ovn-kubernetes\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.766637 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.765295 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c44f520d-f602-40e0-94d4-23277e56d67e-metrics-server-audit-profiles\") pod \"metrics-server-6d7f54bf5f-jkjtm\" (UID: \"c44f520d-f602-40e0-94d4-23277e56d67e\") " pod="openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm" Oct 30 12:52:24.766637 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.765331 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-cni-binary-copy\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.766637 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.765338 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.766637 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.765329 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3f38af1e-cd92-4cd8-a643-151b10b8da12-agent-certs\") pod \"konnectivity-agent-2nqhp\" (UID: \"3f38af1e-cd92-4cd8-a643-151b10b8da12\") " pod="kube-system/konnectivity-agent-2nqhp" Oct 30 12:52:24.766637 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.765433 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.766637 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.765514 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3f38af1e-cd92-4cd8-a643-151b10b8da12-agent-certs\") pod \"konnectivity-agent-2nqhp\" (UID: \"3f38af1e-cd92-4cd8-a643-151b10b8da12\") " pod="kube-system/konnectivity-agent-2nqhp" Oct 30 12:52:24.766637 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.765606 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-sys\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:52:24.766637 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.765665 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11a0b508-2f9c-4be6-b5c7-b3d74766e801-host-slash\") pod \"iptables-alerter-zvw7s\" (UID: \"11a0b508-2f9c-4be6-b5c7-b3d74766e801\") " pod="openshift-network-operator/iptables-alerter-zvw7s" Oct 30 12:52:24.766637 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.765685 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-config-volume\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.766637 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.765688 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c44f520d-f602-40e0-94d4-23277e56d67e-metrics-server-audit-profiles\") pod \"metrics-server-6d7f54bf5f-jkjtm\" (UID: \"c44f520d-f602-40e0-94d4-23277e56d67e\") " pod="openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm" Oct 30 12:52:24.766637 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.765704 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82x4j\" (UniqueName: \"kubernetes.io/projected/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-kube-api-access-82x4j\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.766637 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.765789 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-var-lib-openvswitch\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.766637 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.765813 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-host-run-netns\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.766637 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.765838 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-multus-conf-dir\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.766637 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.765868 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvf5p\" (UniqueName: \"kubernetes.io/projected/51a542f5-796e-46b1-95bd-31751d50b52b-kube-api-access-tvf5p\") pod \"dns-default-v7skv\" (UID: \"51a542f5-796e-46b1-95bd-31751d50b52b\") " pod="openshift-dns/dns-default-v7skv" Oct 30 12:52:24.766637 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.765910 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50bec85b-5592-437d-aa6f-4899aeb494a5-original-pull-secret\") pod \"global-pull-secret-syncer-s7fsp\" (UID: \"50bec85b-5592-437d-aa6f-4899aeb494a5\") " pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:52:24.766637 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.765946 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-config-volume\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.766637 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.765978 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bea99b12-e97c-4c0e-9360-07e542a4294d-registration-dir\") pod \"aws-ebs-csi-driver-node-dsh66\" (UID: \"bea99b12-e97c-4c0e-9360-07e542a4294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" Oct 30 12:52:24.766637 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.766007 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-etc-sysctl-d\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:52:24.766637 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.766043 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8fd0fdb6-c254-446d-8832-5a2fb284f03d-sys\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:24.766637 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.766068 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee23e57d-7b77-4dba-95a5-84ad59133981-system-cni-dir\") pod \"multus-additional-cni-plugins-75bqd\" (UID: \"ee23e57d-7b77-4dba-95a5-84ad59133981\") " pod="openshift-multus/multus-additional-cni-plugins-75bqd" Oct 30 12:52:24.766637 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.766084 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-host-run-k8s-cni-cncf-io\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.766637 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.766099 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-host-var-lib-kubelet\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.766637 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.766127 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-etc-sysctl-conf\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:52:24.766637 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.766154 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-lib-modules\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:52:24.766637 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.766171 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-var-lib-kubelet\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:52:24.766637 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.766198 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d876w\" (UniqueName: \"kubernetes.io/projected/ecd3d4d5-a59a-4b0e-a981-956b10d08531-kube-api-access-d876w\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.766637 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.766231 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-log-socket\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.766637 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.766258 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50bec85b-5592-437d-aa6f-4899aeb494a5-original-pull-secret\") pod \"global-pull-secret-syncer-s7fsp\" (UID: \"50bec85b-5592-437d-aa6f-4899aeb494a5\") " pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:52:24.766637 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.766321 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-etc-kubernetes\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:52:24.766637 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.766356 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-run\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:52:24.766637 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.766415 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-config-out\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.766637 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.766468 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.766637 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.766514 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-config-out\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.766637 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.766544 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/dd3d929e-3197-4794-9b31-794427a55693-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-4tq96\" (UID: \"dd3d929e-3197-4794-9b31-794427a55693\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4tq96" Oct 30 12:52:24.766637 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.766589 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-web-config\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.766637 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.766633 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4dca982f-f509-454b-a057-d7fd9b3f34bd-data-volume\") pod \"insights-runtime-extractor-cfgs4\" (UID: \"4dca982f-f509-454b-a057-d7fd9b3f34bd\") " pod="openshift-insights/insights-runtime-extractor-cfgs4" Oct 30 12:52:24.766637 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.766658 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c44f520d-f602-40e0-94d4-23277e56d67e-client-ca-bundle\") pod \"metrics-server-6d7f54bf5f-jkjtm\" (UID: \"c44f520d-f602-40e0-94d4-23277e56d67e\") " pod="openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.766723 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.766748 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4dca982f-f509-454b-a057-d7fd9b3f34bd-data-volume\") pod \"insights-runtime-extractor-cfgs4\" (UID: \"4dca982f-f509-454b-a057-d7fd9b3f34bd\") " pod="openshift-insights/insights-runtime-extractor-cfgs4" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.766773 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee23e57d-7b77-4dba-95a5-84ad59133981-tuning-conf-dir\") pod \"multus-additional-cni-plugins-75bqd\" (UID: \"ee23e57d-7b77-4dba-95a5-84ad59133981\") " pod="openshift-multus/multus-additional-cni-plugins-75bqd" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.766799 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-config\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.766832 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.766862 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c44f520d-f602-40e0-94d4-23277e56d67e-client-ca-bundle\") pod \"metrics-server-6d7f54bf5f-jkjtm\" (UID: \"c44f520d-f602-40e0-94d4-23277e56d67e\") " pod="openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.766865 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.766815 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/dd3d929e-3197-4794-9b31-794427a55693-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-4tq96\" (UID: \"dd3d929e-3197-4794-9b31-794427a55693\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4tq96" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.766901 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-web-config\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.766904 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/dd3d929e-3197-4794-9b31-794427a55693-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-4tq96\" (UID: \"dd3d929e-3197-4794-9b31-794427a55693\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4tq96" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.766932 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee23e57d-7b77-4dba-95a5-84ad59133981-cnibin\") pod \"multus-additional-cni-plugins-75bqd\" (UID: \"ee23e57d-7b77-4dba-95a5-84ad59133981\") " pod="openshift-multus/multus-additional-cni-plugins-75bqd" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.766954 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4dca982f-f509-454b-a057-d7fd9b3f34bd-crio-socket\") pod \"insights-runtime-extractor-cfgs4\" (UID: \"4dca982f-f509-454b-a057-d7fd9b3f34bd\") " pod="openshift-insights/insights-runtime-extractor-cfgs4" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.766980 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767009 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767012 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/dd3d929e-3197-4794-9b31-794427a55693-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-4tq96\" (UID: \"dd3d929e-3197-4794-9b31-794427a55693\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4tq96" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767014 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-config\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767039 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767043 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bea99b12-e97c-4c0e-9360-07e542a4294d-etc-selinux\") pod \"aws-ebs-csi-driver-node-dsh66\" (UID: \"bea99b12-e97c-4c0e-9360-07e542a4294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767059 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767079 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f89a7014-2966-43c7-a901-709c690b0003-console-serving-cert\") pod \"console-85cbd9968c-p6js7\" (UID: \"f89a7014-2966-43c7-a901-709c690b0003\") " pod="openshift-console/console-85cbd9968c-p6js7" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767105 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f89a7014-2966-43c7-a901-709c690b0003-oauth-serving-cert\") pod \"console-85cbd9968c-p6js7\" (UID: \"f89a7014-2966-43c7-a901-709c690b0003\") " pod="openshift-console/console-85cbd9968c-p6js7" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767130 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b0992e81-44ba-41c3-8079-c801a2d40aa4-federate-client-tls\") pod \"telemeter-client-76d946b6b9-5lvf8\" (UID: \"b0992e81-44ba-41c3-8079-c801a2d40aa4\") " pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767160 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767182 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767186 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d9b03988-7080-4352-ba1e-08ee68ba3df8-ovn-node-metrics-cert\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767213 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-host-run-multus-certs\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767240 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767266 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767292 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767305 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f89a7014-2966-43c7-a901-709c690b0003-oauth-serving-cert\") pod \"console-85cbd9968c-p6js7\" (UID: \"f89a7014-2966-43c7-a901-709c690b0003\") " pod="openshift-console/console-85cbd9968c-p6js7" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767319 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bea99b12-e97c-4c0e-9360-07e542a4294d-socket-dir\") pod \"aws-ebs-csi-driver-node-dsh66\" (UID: \"bea99b12-e97c-4c0e-9360-07e542a4294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767243 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f89a7014-2966-43c7-a901-709c690b0003-console-serving-cert\") pod \"console-85cbd9968c-p6js7\" (UID: \"f89a7014-2966-43c7-a901-709c690b0003\") " pod="openshift-console/console-85cbd9968c-p6js7" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767376 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b0992e81-44ba-41c3-8079-c801a2d40aa4-federate-client-tls\") pod \"telemeter-client-76d946b6b9-5lvf8\" (UID: \"b0992e81-44ba-41c3-8079-c801a2d40aa4\") " pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767383 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d9b03988-7080-4352-ba1e-08ee68ba3df8-ovn-node-metrics-cert\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767391 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767404 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767444 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-host-cni-netd\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767461 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767472 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d9b03988-7080-4352-ba1e-08ee68ba3df8-env-overrides\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767500 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c44f520d-f602-40e0-94d4-23277e56d67e-secret-metrics-server-tls\") pod \"metrics-server-6d7f54bf5f-jkjtm\" (UID: \"c44f520d-f602-40e0-94d4-23277e56d67e\") " pod="openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767525 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/66172ef8-aaad-4875-b3b3-bd81b9e9ba59-host\") pod \"node-ca-bcbg8\" (UID: \"66172ef8-aaad-4875-b3b3-bd81b9e9ba59\") " pod="openshift-image-registry/node-ca-bcbg8" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767535 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767527 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767555 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-etc-systemd\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767583 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/accde534-15e4-42cf-94df-c0aa085f0603-tmp\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767599 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d9b03988-7080-4352-ba1e-08ee68ba3df8-env-overrides\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767622 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wksmn\" (UniqueName: \"kubernetes.io/projected/accde534-15e4-42cf-94df-c0aa085f0603-kube-api-access-wksmn\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767644 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767660 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/11a0b508-2f9c-4be6-b5c7-b3d74766e801-iptables-alerter-script\") pod \"iptables-alerter-zvw7s\" (UID: \"11a0b508-2f9c-4be6-b5c7-b3d74766e801\") " pod="openshift-network-operator/iptables-alerter-zvw7s" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767675 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpgrm\" (UniqueName: \"kubernetes.io/projected/fe19a15d-5112-4b76-a226-4b8462e2fb33-kube-api-access-qpgrm\") pod \"node-resolver-9szkr\" (UID: \"fe19a15d-5112-4b76-a226-4b8462e2fb33\") " pod="openshift-dns/node-resolver-9szkr" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767695 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bea99b12-e97c-4c0e-9360-07e542a4294d-device-dir\") pod \"aws-ebs-csi-driver-node-dsh66\" (UID: \"bea99b12-e97c-4c0e-9360-07e542a4294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767672 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/accde534-15e4-42cf-94df-c0aa085f0603-tmp\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767721 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-host-cni-bin\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767729 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c44f520d-f602-40e0-94d4-23277e56d67e-secret-metrics-server-tls\") pod \"metrics-server-6d7f54bf5f-jkjtm\" (UID: \"c44f520d-f602-40e0-94d4-23277e56d67e\") " pod="openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767745 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ee23e57d-7b77-4dba-95a5-84ad59133981-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-75bqd\" (UID: \"ee23e57d-7b77-4dba-95a5-84ad59133981\") " pod="openshift-multus/multus-additional-cni-plugins-75bqd" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767762 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01873344-73c8-47b2-b6ba-8e338ba18315-console-serving-cert\") pod \"console-689bbd6cb4-lmgqn\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767791 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01873344-73c8-47b2-b6ba-8e338ba18315-tmp\") pod \"console-689bbd6cb4-lmgqn\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767820 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-hostroot\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767848 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767860 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/11a0b508-2f9c-4be6-b5c7-b3d74766e801-iptables-alerter-script\") pod \"iptables-alerter-zvw7s\" (UID: \"11a0b508-2f9c-4be6-b5c7-b3d74766e801\") " pod="openshift-network-operator/iptables-alerter-zvw7s" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767877 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8fd0fdb6-c254-446d-8832-5a2fb284f03d-node-exporter-accelerators-collector-config\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767895 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01873344-73c8-47b2-b6ba-8e338ba18315-tmp\") pod \"console-689bbd6cb4-lmgqn\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767943 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767946 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ee23e57d-7b77-4dba-95a5-84ad59133981-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-75bqd\" (UID: \"ee23e57d-7b77-4dba-95a5-84ad59133981\") " pod="openshift-multus/multus-additional-cni-plugins-75bqd" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767976 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27dm6\" (UniqueName: \"kubernetes.io/projected/8fd0fdb6-c254-446d-8832-5a2fb284f03d-kube-api-access-27dm6\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.767997 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-run-ovn\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.768015 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c44f520d-f602-40e0-94d4-23277e56d67e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6d7f54bf5f-jkjtm\" (UID: \"c44f520d-f602-40e0-94d4-23277e56d67e\") " pod="openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.768032 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-host-var-lib-cni-multus\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.768056 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01873344-73c8-47b2-b6ba-8e338ba18315-console-serving-cert\") pod \"console-689bbd6cb4-lmgqn\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.768064 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ecd3d4d5-a59a-4b0e-a981-956b10d08531-config-out\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.768082 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.768117 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8fd0fdb6-c254-446d-8832-5a2fb284f03d-node-exporter-accelerators-collector-config\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.768126 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-host-run-netns\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.768157 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.768161 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ecd3d4d5-a59a-4b0e-a981-956b10d08531-config-out\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.768191 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c44f520d-f602-40e0-94d4-23277e56d67e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6d7f54bf5f-jkjtm\" (UID: \"c44f520d-f602-40e0-94d4-23277e56d67e\") " pod="openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.768205 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/51a542f5-796e-46b1-95bd-31751d50b52b-config-volume\") pod \"dns-default-v7skv\" (UID: \"51a542f5-796e-46b1-95bd-31751d50b52b\") " pod="openshift-dns/dns-default-v7skv" Oct 30 12:52:24.768216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.768237 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.770569 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.768277 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.770569 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.768285 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.770569 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.768351 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/51a542f5-796e-46b1-95bd-31751d50b52b-config-volume\") pod \"dns-default-v7skv\" (UID: \"51a542f5-796e-46b1-95bd-31751d50b52b\") " pod="openshift-dns/dns-default-v7skv" Oct 30 12:52:24.770569 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.768374 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/50bec85b-5592-437d-aa6f-4899aeb494a5-kubelet-config\") pod \"global-pull-secret-syncer-s7fsp\" (UID: \"50bec85b-5592-437d-aa6f-4899aeb494a5\") " pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:52:24.770569 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.768418 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/50bec85b-5592-437d-aa6f-4899aeb494a5-dbus\") pod \"global-pull-secret-syncer-s7fsp\" (UID: \"50bec85b-5592-437d-aa6f-4899aeb494a5\") " pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:52:24.770569 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.768449 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvxmx\" (UniqueName: \"kubernetes.io/projected/66172ef8-aaad-4875-b3b3-bd81b9e9ba59-kube-api-access-wvxmx\") pod \"node-ca-bcbg8\" (UID: \"66172ef8-aaad-4875-b3b3-bd81b9e9ba59\") " pod="openshift-image-registry/node-ca-bcbg8" Oct 30 12:52:24.770569 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.768474 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f89a7014-2966-43c7-a901-709c690b0003-console-config\") pod \"console-85cbd9968c-p6js7\" (UID: \"f89a7014-2966-43c7-a901-709c690b0003\") " pod="openshift-console/console-85cbd9968c-p6js7" Oct 30 12:52:24.770569 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.768497 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.770569 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.768500 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-host-var-lib-cni-bin\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.770569 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.768538 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-systemd-units\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.770569 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.768565 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d9b03988-7080-4352-ba1e-08ee68ba3df8-ovnkube-script-lib\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.770569 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.768539 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.770569 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.768583 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/56928a4d-7923-46e6-97bb-6690514f89a4-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-xxb2q\" (UID: \"56928a4d-7923-46e6-97bb-6690514f89a4\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xxb2q" Oct 30 12:52:24.770569 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.768595 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f89a7014-2966-43c7-a901-709c690b0003-console-config\") pod \"console-85cbd9968c-p6js7\" (UID: \"f89a7014-2966-43c7-a901-709c690b0003\") " pod="openshift-console/console-85cbd9968c-p6js7" Oct 30 12:52:24.770569 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.768651 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c44f520d-f602-40e0-94d4-23277e56d67e-secret-metrics-server-client-certs\") pod \"metrics-server-6d7f54bf5f-jkjtm\" (UID: \"c44f520d-f602-40e0-94d4-23277e56d67e\") " pod="openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm" Oct 30 12:52:24.770569 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.768671 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/66172ef8-aaad-4875-b3b3-bd81b9e9ba59-serviceca\") pod \"node-ca-bcbg8\" (UID: \"66172ef8-aaad-4875-b3b3-bd81b9e9ba59\") " pod="openshift-image-registry/node-ca-bcbg8" Oct 30 12:52:24.770569 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.768690 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-cnibin\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.770569 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.768710 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-host\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:52:24.770569 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.768736 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-run-systemd\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.770569 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.768782 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/56928a4d-7923-46e6-97bb-6690514f89a4-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-xxb2q\" (UID: \"56928a4d-7923-46e6-97bb-6690514f89a4\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xxb2q" Oct 30 12:52:24.770569 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.768792 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d9b03988-7080-4352-ba1e-08ee68ba3df8-ovnkube-script-lib\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.770569 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.768860 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c44f520d-f602-40e0-94d4-23277e56d67e-secret-metrics-server-client-certs\") pod \"metrics-server-6d7f54bf5f-jkjtm\" (UID: \"c44f520d-f602-40e0-94d4-23277e56d67e\") " pod="openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm" Oct 30 12:52:24.770569 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.768916 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/66172ef8-aaad-4875-b3b3-bd81b9e9ba59-serviceca\") pod \"node-ca-bcbg8\" (UID: \"66172ef8-aaad-4875-b3b3-bd81b9e9ba59\") " pod="openshift-image-registry/node-ca-bcbg8" Oct 30 12:52:24.793468 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.793414 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhkpc\" (UniqueName: \"kubernetes.io/projected/c44f520d-f602-40e0-94d4-23277e56d67e-kube-api-access-jhkpc\") pod \"metrics-server-6d7f54bf5f-jkjtm\" (UID: \"c44f520d-f602-40e0-94d4-23277e56d67e\") " pod="openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm" Oct 30 12:52:24.793468 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.793449 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssl6b\" (UniqueName: \"kubernetes.io/projected/f89a7014-2966-43c7-a901-709c690b0003-kube-api-access-ssl6b\") pod \"console-85cbd9968c-p6js7\" (UID: \"f89a7014-2966-43c7-a901-709c690b0003\") " pod="openshift-console/console-85cbd9968c-p6js7" Oct 30 12:52:24.793468 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.793455 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m944\" (UniqueName: \"kubernetes.io/projected/4dca982f-f509-454b-a057-d7fd9b3f34bd-kube-api-access-5m944\") pod \"insights-runtime-extractor-cfgs4\" (UID: \"4dca982f-f509-454b-a057-d7fd9b3f34bd\") " pod="openshift-insights/insights-runtime-extractor-cfgs4" Oct 30 12:52:24.793707 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.793501 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqvdp\" (UniqueName: \"kubernetes.io/projected/1bb48bfa-827b-4179-b9fa-418fc14f0f44-kube-api-access-dqvdp\") pod \"network-check-target-95n9l\" (UID: \"1bb48bfa-827b-4179-b9fa-418fc14f0f44\") " pod="openshift-network-diagnostics/network-check-target-95n9l" Oct 30 12:52:24.793707 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.793588 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w592\" (UniqueName: \"kubernetes.io/projected/01873344-73c8-47b2-b6ba-8e338ba18315-kube-api-access-2w592\") pod \"console-689bbd6cb4-lmgqn\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:24.793707 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.793639 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8xd7\" (UniqueName: \"kubernetes.io/projected/dd3d929e-3197-4794-9b31-794427a55693-kube-api-access-m8xd7\") pod \"kube-state-metrics-5dc8d7d89b-4tq96\" (UID: \"dd3d929e-3197-4794-9b31-794427a55693\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4tq96" Oct 30 12:52:24.793707 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.793654 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lltjs\" (UniqueName: \"kubernetes.io/projected/b0992e81-44ba-41c3-8079-c801a2d40aa4-kube-api-access-lltjs\") pod \"telemeter-client-76d946b6b9-5lvf8\" (UID: \"b0992e81-44ba-41c3-8079-c801a2d40aa4\") " pod="openshift-monitoring/telemeter-client-76d946b6b9-5lvf8" Oct 30 12:52:24.794125 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.794107 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwddx\" (UniqueName: \"kubernetes.io/projected/ee23e57d-7b77-4dba-95a5-84ad59133981-kube-api-access-hwddx\") pod \"multus-additional-cni-plugins-75bqd\" (UID: \"ee23e57d-7b77-4dba-95a5-84ad59133981\") " pod="openshift-multus/multus-additional-cni-plugins-75bqd" Oct 30 12:52:24.795893 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.795875 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkldf\" (UniqueName: \"kubernetes.io/projected/11a0b508-2f9c-4be6-b5c7-b3d74766e801-kube-api-access-tkldf\") pod \"iptables-alerter-zvw7s\" (UID: \"11a0b508-2f9c-4be6-b5c7-b3d74766e801\") " pod="openshift-network-operator/iptables-alerter-zvw7s" Oct 30 12:52:24.797865 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.797846 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvf5p\" (UniqueName: \"kubernetes.io/projected/51a542f5-796e-46b1-95bd-31751d50b52b-kube-api-access-tvf5p\") pod \"dns-default-v7skv\" (UID: \"51a542f5-796e-46b1-95bd-31751d50b52b\") " pod="openshift-dns/dns-default-v7skv" Oct 30 12:52:24.798110 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.798091 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v6nm\" (UniqueName: \"kubernetes.io/projected/56928a4d-7923-46e6-97bb-6690514f89a4-kube-api-access-9v6nm\") pod \"prometheus-operator-9cd89b74c-xxb2q\" (UID: \"56928a4d-7923-46e6-97bb-6690514f89a4\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xxb2q" Oct 30 12:52:24.798110 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.798102 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82x4j\" (UniqueName: \"kubernetes.io/projected/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-kube-api-access-82x4j\") pod \"alertmanager-main-0\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.798182 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.798142 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxwlx\" (UniqueName: \"kubernetes.io/projected/1fcccc7b-a83c-4c52-a648-cb549e70b54d-kube-api-access-xxwlx\") pod \"network-metrics-daemon-tpz8b\" (UID: \"1fcccc7b-a83c-4c52-a648-cb549e70b54d\") " pod="openshift-multus/network-metrics-daemon-tpz8b" Oct 30 12:52:24.798393 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.798372 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvdzg\" (UniqueName: \"kubernetes.io/projected/14c0845c-1947-44b0-ada5-4b68ff9974f2-kube-api-access-wvdzg\") pod \"monitoring-plugin-77f4f76594-spsvf\" (UID: \"14c0845c-1947-44b0-ada5-4b68ff9974f2\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-spsvf" Oct 30 12:52:24.798451 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.798438 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbr4h\" (UniqueName: \"kubernetes.io/projected/ab451161-af25-47e2-860e-f7d5186d4491-kube-api-access-gbr4h\") pod \"ingress-canary-854v4\" (UID: \"ab451161-af25-47e2-860e-f7d5186d4491\") " pod="openshift-ingress-canary/ingress-canary-854v4" Oct 30 12:52:24.799079 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.799062 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9x52\" (UniqueName: \"kubernetes.io/projected/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-kube-api-access-x9x52\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.800213 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.800196 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8fhd\" (UniqueName: \"kubernetes.io/projected/bea99b12-e97c-4c0e-9360-07e542a4294d-kube-api-access-m8fhd\") pod \"aws-ebs-csi-driver-node-dsh66\" (UID: \"bea99b12-e97c-4c0e-9360-07e542a4294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" Oct 30 12:52:24.801080 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.801062 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvxmx\" (UniqueName: \"kubernetes.io/projected/66172ef8-aaad-4875-b3b3-bd81b9e9ba59-kube-api-access-wvxmx\") pod \"node-ca-bcbg8\" (UID: \"66172ef8-aaad-4875-b3b3-bd81b9e9ba59\") " pod="openshift-image-registry/node-ca-bcbg8" Oct 30 12:52:24.801183 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.801166 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp4zz\" (UniqueName: \"kubernetes.io/projected/d9b03988-7080-4352-ba1e-08ee68ba3df8-kube-api-access-qp4zz\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.801641 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.801602 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wksmn\" (UniqueName: \"kubernetes.io/projected/accde534-15e4-42cf-94df-c0aa085f0603-kube-api-access-wksmn\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:52:24.802820 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.802804 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27dm6\" (UniqueName: \"kubernetes.io/projected/8fd0fdb6-c254-446d-8832-5a2fb284f03d-kube-api-access-27dm6\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:24.803457 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.803442 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d876w\" (UniqueName: \"kubernetes.io/projected/ecd3d4d5-a59a-4b0e-a981-956b10d08531-kube-api-access-d876w\") pod \"prometheus-k8s-0\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.803780 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.803764 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpgrm\" (UniqueName: \"kubernetes.io/projected/fe19a15d-5112-4b76-a226-4b8462e2fb33-kube-api-access-qpgrm\") pod \"node-resolver-9szkr\" (UID: \"fe19a15d-5112-4b76-a226-4b8462e2fb33\") " pod="openshift-dns/node-resolver-9szkr" Oct 30 12:52:24.869310 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869262 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bea99b12-e97c-4c0e-9360-07e542a4294d-etc-selinux\") pod \"aws-ebs-csi-driver-node-dsh66\" (UID: \"bea99b12-e97c-4c0e-9360-07e542a4294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" Oct 30 12:52:24.869310 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869287 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-host-run-multus-certs\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.869310 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869302 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bea99b12-e97c-4c0e-9360-07e542a4294d-socket-dir\") pod \"aws-ebs-csi-driver-node-dsh66\" (UID: \"bea99b12-e97c-4c0e-9360-07e542a4294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" Oct 30 12:52:24.869454 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869317 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-host-cni-netd\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.869454 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869333 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/66172ef8-aaad-4875-b3b3-bd81b9e9ba59-host\") pod \"node-ca-bcbg8\" (UID: \"66172ef8-aaad-4875-b3b3-bd81b9e9ba59\") " pod="openshift-image-registry/node-ca-bcbg8" Oct 30 12:52:24.869454 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869365 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/66172ef8-aaad-4875-b3b3-bd81b9e9ba59-host\") pod \"node-ca-bcbg8\" (UID: \"66172ef8-aaad-4875-b3b3-bd81b9e9ba59\") " pod="openshift-image-registry/node-ca-bcbg8" Oct 30 12:52:24.869454 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869364 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-host-run-multus-certs\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.869454 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869393 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-host-cni-netd\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.869454 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869402 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bea99b12-e97c-4c0e-9360-07e542a4294d-etc-selinux\") pod \"aws-ebs-csi-driver-node-dsh66\" (UID: \"bea99b12-e97c-4c0e-9360-07e542a4294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" Oct 30 12:52:24.869454 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869402 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bea99b12-e97c-4c0e-9360-07e542a4294d-socket-dir\") pod \"aws-ebs-csi-driver-node-dsh66\" (UID: \"bea99b12-e97c-4c0e-9360-07e542a4294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" Oct 30 12:52:24.869454 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869423 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-etc-systemd\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:52:24.869711 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869461 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-etc-systemd\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:52:24.869711 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869466 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bea99b12-e97c-4c0e-9360-07e542a4294d-device-dir\") pod \"aws-ebs-csi-driver-node-dsh66\" (UID: \"bea99b12-e97c-4c0e-9360-07e542a4294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" Oct 30 12:52:24.869711 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869490 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-host-cni-bin\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.869711 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869507 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-hostroot\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.869711 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869524 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-run-ovn\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.869711 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869540 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-host-cni-bin\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.869711 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869546 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-host-var-lib-cni-multus\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.869711 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869566 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-hostroot\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.869711 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869572 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-host-run-netns\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.869711 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869577 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-run-ovn\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.869711 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869586 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bea99b12-e97c-4c0e-9360-07e542a4294d-device-dir\") pod \"aws-ebs-csi-driver-node-dsh66\" (UID: \"bea99b12-e97c-4c0e-9360-07e542a4294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" Oct 30 12:52:24.869711 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869588 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-host-var-lib-cni-multus\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.869711 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869598 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/50bec85b-5592-437d-aa6f-4899aeb494a5-kubelet-config\") pod \"global-pull-secret-syncer-s7fsp\" (UID: \"50bec85b-5592-437d-aa6f-4899aeb494a5\") " pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:52:24.869711 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869633 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/50bec85b-5592-437d-aa6f-4899aeb494a5-dbus\") pod \"global-pull-secret-syncer-s7fsp\" (UID: \"50bec85b-5592-437d-aa6f-4899aeb494a5\") " pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:52:24.869711 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869636 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-host-run-netns\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.869711 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869652 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-host-var-lib-cni-bin\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.869711 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869676 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-systemd-units\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.869711 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869693 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/50bec85b-5592-437d-aa6f-4899aeb494a5-kubelet-config\") pod \"global-pull-secret-syncer-s7fsp\" (UID: \"50bec85b-5592-437d-aa6f-4899aeb494a5\") " pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:52:24.869711 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869702 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-cnibin\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869725 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/50bec85b-5592-437d-aa6f-4899aeb494a5-dbus\") pod \"global-pull-secret-syncer-s7fsp\" (UID: \"50bec85b-5592-437d-aa6f-4899aeb494a5\") " pod="kube-system/global-pull-secret-syncer-s7fsp" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869733 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-systemd-units\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869726 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-host-var-lib-cni-bin\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869731 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-host\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869769 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-run-systemd\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869790 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-etc-kubernetes\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869771 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-host\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869808 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-run-openvswitch\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869834 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-system-cni-dir\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869843 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-run-openvswitch\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869835 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-etc-kubernetes\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869857 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8fd0fdb6-c254-446d-8832-5a2fb284f03d-node-exporter-wtmp\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869746 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-cnibin\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869812 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-run-systemd\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869879 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bea99b12-e97c-4c0e-9360-07e542a4294d-sys-fs\") pod \"aws-ebs-csi-driver-node-dsh66\" (UID: \"bea99b12-e97c-4c0e-9360-07e542a4294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869899 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-etc-modprobe-d\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869915 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-system-cni-dir\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869922 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8fd0fdb6-c254-446d-8832-5a2fb284f03d-node-exporter-wtmp\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869933 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bea99b12-e97c-4c0e-9360-07e542a4294d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dsh66\" (UID: \"bea99b12-e97c-4c0e-9360-07e542a4294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869945 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bea99b12-e97c-4c0e-9360-07e542a4294d-sys-fs\") pod \"aws-ebs-csi-driver-node-dsh66\" (UID: \"bea99b12-e97c-4c0e-9360-07e542a4294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869954 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-host-kubelet\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869971 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bea99b12-e97c-4c0e-9360-07e542a4294d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dsh66\" (UID: \"bea99b12-e97c-4c0e-9360-07e542a4294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869972 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-host-slash\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.869998 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870001 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-host-slash\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870011 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-host-kubelet\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870004 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-etc-modprobe-d\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870031 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870041 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8fd0fdb6-c254-446d-8832-5a2fb284f03d-root\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870024 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8fd0fdb6-c254-446d-8832-5a2fb284f03d-root\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870064 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fe19a15d-5112-4b76-a226-4b8462e2fb33-hosts-file\") pod \"node-resolver-9szkr\" (UID: \"fe19a15d-5112-4b76-a226-4b8462e2fb33\") " pod="openshift-dns/node-resolver-9szkr" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870082 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-node-log\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870100 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fe19a15d-5112-4b76-a226-4b8462e2fb33-hosts-file\") pod \"node-resolver-9szkr\" (UID: \"fe19a15d-5112-4b76-a226-4b8462e2fb33\") " pod="openshift-dns/node-resolver-9szkr" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870106 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-etc-sysconfig\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870133 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-node-log\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870142 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-etc-sysconfig\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870134 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-etc-openvswitch\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870162 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-etc-openvswitch\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870169 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-multus-cni-dir\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870193 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-multus-socket-dir-parent\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870211 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-host-run-ovn-kubernetes\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870221 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-multus-cni-dir\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870225 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-sys\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870244 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11a0b508-2f9c-4be6-b5c7-b3d74766e801-host-slash\") pod \"iptables-alerter-zvw7s\" (UID: \"11a0b508-2f9c-4be6-b5c7-b3d74766e801\") " pod="openshift-network-operator/iptables-alerter-zvw7s" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870260 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-sys\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870263 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-var-lib-openvswitch\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870270 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-host-run-ovn-kubernetes\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870280 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-multus-socket-dir-parent\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.870269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870289 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11a0b508-2f9c-4be6-b5c7-b3d74766e801-host-slash\") pod \"iptables-alerter-zvw7s\" (UID: \"11a0b508-2f9c-4be6-b5c7-b3d74766e801\") " pod="openshift-network-operator/iptables-alerter-zvw7s" Oct 30 12:52:24.871623 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870290 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-host-run-netns\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.871623 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870315 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-multus-conf-dir\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.871623 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870332 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bea99b12-e97c-4c0e-9360-07e542a4294d-registration-dir\") pod \"aws-ebs-csi-driver-node-dsh66\" (UID: \"bea99b12-e97c-4c0e-9360-07e542a4294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" Oct 30 12:52:24.871623 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870313 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-host-run-netns\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.871623 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870363 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-multus-conf-dir\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.871623 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870371 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-var-lib-openvswitch\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.871623 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870376 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bea99b12-e97c-4c0e-9360-07e542a4294d-registration-dir\") pod \"aws-ebs-csi-driver-node-dsh66\" (UID: \"bea99b12-e97c-4c0e-9360-07e542a4294d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsh66" Oct 30 12:52:24.871623 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870390 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-etc-sysctl-d\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:52:24.871623 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870407 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8fd0fdb6-c254-446d-8832-5a2fb284f03d-sys\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:24.871623 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870433 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8fd0fdb6-c254-446d-8832-5a2fb284f03d-sys\") pod \"node-exporter-c4jgj\" (UID: \"8fd0fdb6-c254-446d-8832-5a2fb284f03d\") " pod="openshift-monitoring/node-exporter-c4jgj" Oct 30 12:52:24.871623 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870449 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee23e57d-7b77-4dba-95a5-84ad59133981-system-cni-dir\") pod \"multus-additional-cni-plugins-75bqd\" (UID: \"ee23e57d-7b77-4dba-95a5-84ad59133981\") " pod="openshift-multus/multus-additional-cni-plugins-75bqd" Oct 30 12:52:24.871623 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870464 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-host-run-k8s-cni-cncf-io\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.871623 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870478 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-host-var-lib-kubelet\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.871623 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870491 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-etc-sysctl-conf\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:52:24.871623 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870493 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-etc-sysctl-d\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:52:24.871623 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870505 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-lib-modules\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:52:24.871623 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870529 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-host-run-k8s-cni-cncf-io\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.871623 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870554 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dad46b5c-e3e3-4efa-b8f1-b13de224ef8c-host-var-lib-kubelet\") pod \"multus-sw8rj\" (UID: \"dad46b5c-e3e3-4efa-b8f1-b13de224ef8c\") " pod="openshift-multus/multus-sw8rj" Oct 30 12:52:24.871623 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870564 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee23e57d-7b77-4dba-95a5-84ad59133981-system-cni-dir\") pod \"multus-additional-cni-plugins-75bqd\" (UID: \"ee23e57d-7b77-4dba-95a5-84ad59133981\") " pod="openshift-multus/multus-additional-cni-plugins-75bqd" Oct 30 12:52:24.871623 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870522 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-var-lib-kubelet\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:52:24.871623 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870589 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-log-socket\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.871623 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870599 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-var-lib-kubelet\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:52:24.871623 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870604 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-etc-kubernetes\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:52:24.871623 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870630 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-lib-modules\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:52:24.871623 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870644 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-etc-sysctl-conf\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:52:24.871623 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870654 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-etc-kubernetes\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:52:24.871623 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870645 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-run\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:52:24.871623 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870673 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/accde534-15e4-42cf-94df-c0aa085f0603-run\") pod \"tuned-k76bj\" (UID: \"accde534-15e4-42cf-94df-c0aa085f0603\") " pod="openshift-cluster-node-tuning-operator/tuned-k76bj" Oct 30 12:52:24.871623 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870687 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee23e57d-7b77-4dba-95a5-84ad59133981-tuning-conf-dir\") pod \"multus-additional-cni-plugins-75bqd\" (UID: \"ee23e57d-7b77-4dba-95a5-84ad59133981\") " pod="openshift-multus/multus-additional-cni-plugins-75bqd" Oct 30 12:52:24.871623 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870657 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d9b03988-7080-4352-ba1e-08ee68ba3df8-log-socket\") pod \"ovnkube-node-nwrx6\" (UID: \"d9b03988-7080-4352-ba1e-08ee68ba3df8\") " pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:24.871623 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870704 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee23e57d-7b77-4dba-95a5-84ad59133981-cnibin\") pod \"multus-additional-cni-plugins-75bqd\" (UID: \"ee23e57d-7b77-4dba-95a5-84ad59133981\") " pod="openshift-multus/multus-additional-cni-plugins-75bqd" Oct 30 12:52:24.871623 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870720 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4dca982f-f509-454b-a057-d7fd9b3f34bd-crio-socket\") pod \"insights-runtime-extractor-cfgs4\" (UID: \"4dca982f-f509-454b-a057-d7fd9b3f34bd\") " pod="openshift-insights/insights-runtime-extractor-cfgs4" Oct 30 12:52:24.871623 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870737 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee23e57d-7b77-4dba-95a5-84ad59133981-tuning-conf-dir\") pod \"multus-additional-cni-plugins-75bqd\" (UID: \"ee23e57d-7b77-4dba-95a5-84ad59133981\") " pod="openshift-multus/multus-additional-cni-plugins-75bqd" Oct 30 12:52:24.871623 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870757 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee23e57d-7b77-4dba-95a5-84ad59133981-cnibin\") pod \"multus-additional-cni-plugins-75bqd\" (UID: \"ee23e57d-7b77-4dba-95a5-84ad59133981\") " pod="openshift-multus/multus-additional-cni-plugins-75bqd" Oct 30 12:52:24.871623 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:24.870785 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4dca982f-f509-454b-a057-d7fd9b3f34bd-crio-socket\") pod \"insights-runtime-extractor-cfgs4\" (UID: \"4dca982f-f509-454b-a057-d7fd9b3f34bd\") " pod="openshift-insights/insights-runtime-extractor-cfgs4" Oct 30 12:52:25.297628 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:25.297586 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85cbd9968c-p6js7"] Oct 30 12:52:25.668370 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:25.668301 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-v7skv" Oct 30 12:52:25.784039 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:25.784005 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:25.980675 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:25.980653 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:25.999186 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:25.999151 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:26.362680 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:26.362580 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:26.502306 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:26.502270 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:27.467075 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:27.467047 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:27.998520 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:27.998496 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-95n9l" Oct 30 12:52:29.307156 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:29.307123 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-spsvf" Oct 30 12:52:29.763267 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:29.763240 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:30.745859 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:30.745823 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:30.903117 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:30.903086 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:30.917567 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:30.917544 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-85cbd9968c-p6js7" Oct 30 12:52:31.040681 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:31.040622 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:31.040820 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:31.040808 5922 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 12:52:32.213463 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:32.213425 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:32.349388 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:32.349358 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:34.969638 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:34.969545 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:35.084884 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:35.084848 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:35.198621 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:35.198596 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:35.789462 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:35.789434 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:35.793058 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:35.793036 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:52:37.043227 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:37.043197 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:37.132721 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:37.132698 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6d7f54bf5f-jkjtm" Oct 30 12:52:37.160135 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:37.160073 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:37.278971 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:37.278939 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:37.552944 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:37.552130 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:37.668070 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:37.668026 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:40.394278 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:40.394247 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-74d99cdf4-q476v"] Oct 30 12:52:40.397523 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:40.397506 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74d99cdf4-q476v" Oct 30 12:52:40.435967 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:40.435941 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74d99cdf4-q476v"] Oct 30 12:52:40.485457 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:40.485431 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f7e5b498-e68d-4c37-afae-5ee9e28d0fca-console-oauth-config\") pod \"console-74d99cdf4-q476v\" (UID: \"f7e5b498-e68d-4c37-afae-5ee9e28d0fca\") " pod="openshift-console/console-74d99cdf4-q476v" Oct 30 12:52:40.485571 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:40.485468 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f7e5b498-e68d-4c37-afae-5ee9e28d0fca-console-serving-cert\") pod \"console-74d99cdf4-q476v\" (UID: \"f7e5b498-e68d-4c37-afae-5ee9e28d0fca\") " pod="openshift-console/console-74d99cdf4-q476v" Oct 30 12:52:40.485571 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:40.485487 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f7e5b498-e68d-4c37-afae-5ee9e28d0fca-console-config\") pod \"console-74d99cdf4-q476v\" (UID: \"f7e5b498-e68d-4c37-afae-5ee9e28d0fca\") " pod="openshift-console/console-74d99cdf4-q476v" Oct 30 12:52:40.485571 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:40.485512 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f7e5b498-e68d-4c37-afae-5ee9e28d0fca-oauth-serving-cert\") pod \"console-74d99cdf4-q476v\" (UID: \"f7e5b498-e68d-4c37-afae-5ee9e28d0fca\") " pod="openshift-console/console-74d99cdf4-q476v" Oct 30 12:52:40.485719 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:40.485654 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krn2z\" (UniqueName: \"kubernetes.io/projected/f7e5b498-e68d-4c37-afae-5ee9e28d0fca-kube-api-access-krn2z\") pod \"console-74d99cdf4-q476v\" (UID: \"f7e5b498-e68d-4c37-afae-5ee9e28d0fca\") " pod="openshift-console/console-74d99cdf4-q476v" Oct 30 12:52:40.485719 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:40.485709 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7e5b498-e68d-4c37-afae-5ee9e28d0fca-tmp\") pod \"console-74d99cdf4-q476v\" (UID: \"f7e5b498-e68d-4c37-afae-5ee9e28d0fca\") " pod="openshift-console/console-74d99cdf4-q476v" Oct 30 12:52:40.485819 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:40.485804 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f7e5b498-e68d-4c37-afae-5ee9e28d0fca-service-ca\") pod \"console-74d99cdf4-q476v\" (UID: \"f7e5b498-e68d-4c37-afae-5ee9e28d0fca\") " pod="openshift-console/console-74d99cdf4-q476v" Oct 30 12:52:40.485874 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:40.485844 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7e5b498-e68d-4c37-afae-5ee9e28d0fca-trusted-ca-bundle\") pod \"console-74d99cdf4-q476v\" (UID: \"f7e5b498-e68d-4c37-afae-5ee9e28d0fca\") " pod="openshift-console/console-74d99cdf4-q476v" Oct 30 12:52:40.586199 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:40.586173 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f7e5b498-e68d-4c37-afae-5ee9e28d0fca-service-ca\") pod \"console-74d99cdf4-q476v\" (UID: \"f7e5b498-e68d-4c37-afae-5ee9e28d0fca\") " pod="openshift-console/console-74d99cdf4-q476v" Oct 30 12:52:40.586296 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:40.586202 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7e5b498-e68d-4c37-afae-5ee9e28d0fca-trusted-ca-bundle\") pod \"console-74d99cdf4-q476v\" (UID: \"f7e5b498-e68d-4c37-afae-5ee9e28d0fca\") " pod="openshift-console/console-74d99cdf4-q476v" Oct 30 12:52:40.586296 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:40.586226 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f7e5b498-e68d-4c37-afae-5ee9e28d0fca-console-oauth-config\") pod \"console-74d99cdf4-q476v\" (UID: \"f7e5b498-e68d-4c37-afae-5ee9e28d0fca\") " pod="openshift-console/console-74d99cdf4-q476v" Oct 30 12:52:40.586296 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:40.586265 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f7e5b498-e68d-4c37-afae-5ee9e28d0fca-console-serving-cert\") pod \"console-74d99cdf4-q476v\" (UID: \"f7e5b498-e68d-4c37-afae-5ee9e28d0fca\") " pod="openshift-console/console-74d99cdf4-q476v" Oct 30 12:52:40.586296 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:40.586286 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f7e5b498-e68d-4c37-afae-5ee9e28d0fca-console-config\") pod \"console-74d99cdf4-q476v\" (UID: \"f7e5b498-e68d-4c37-afae-5ee9e28d0fca\") " pod="openshift-console/console-74d99cdf4-q476v" Oct 30 12:52:40.586443 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:40.586311 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f7e5b498-e68d-4c37-afae-5ee9e28d0fca-oauth-serving-cert\") pod \"console-74d99cdf4-q476v\" (UID: \"f7e5b498-e68d-4c37-afae-5ee9e28d0fca\") " pod="openshift-console/console-74d99cdf4-q476v" Oct 30 12:52:40.586443 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:40.586342 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krn2z\" (UniqueName: \"kubernetes.io/projected/f7e5b498-e68d-4c37-afae-5ee9e28d0fca-kube-api-access-krn2z\") pod \"console-74d99cdf4-q476v\" (UID: \"f7e5b498-e68d-4c37-afae-5ee9e28d0fca\") " pod="openshift-console/console-74d99cdf4-q476v" Oct 30 12:52:40.586443 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:40.586403 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7e5b498-e68d-4c37-afae-5ee9e28d0fca-tmp\") pod \"console-74d99cdf4-q476v\" (UID: \"f7e5b498-e68d-4c37-afae-5ee9e28d0fca\") " pod="openshift-console/console-74d99cdf4-q476v" Oct 30 12:52:40.586733 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:40.586711 5922 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 12:52:40.587048 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:40.587027 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f7e5b498-e68d-4c37-afae-5ee9e28d0fca-service-ca\") pod \"console-74d99cdf4-q476v\" (UID: \"f7e5b498-e68d-4c37-afae-5ee9e28d0fca\") " pod="openshift-console/console-74d99cdf4-q476v" Oct 30 12:52:40.587123 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:40.587078 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f7e5b498-e68d-4c37-afae-5ee9e28d0fca-console-config\") pod \"console-74d99cdf4-q476v\" (UID: \"f7e5b498-e68d-4c37-afae-5ee9e28d0fca\") " pod="openshift-console/console-74d99cdf4-q476v" Oct 30 12:52:40.587182 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:40.587114 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f7e5b498-e68d-4c37-afae-5ee9e28d0fca-oauth-serving-cert\") pod \"console-74d99cdf4-q476v\" (UID: \"f7e5b498-e68d-4c37-afae-5ee9e28d0fca\") " pod="openshift-console/console-74d99cdf4-q476v" Oct 30 12:52:40.587182 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:40.587119 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7e5b498-e68d-4c37-afae-5ee9e28d0fca-trusted-ca-bundle\") pod \"console-74d99cdf4-q476v\" (UID: \"f7e5b498-e68d-4c37-afae-5ee9e28d0fca\") " pod="openshift-console/console-74d99cdf4-q476v" Oct 30 12:52:40.587303 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:40.587286 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7e5b498-e68d-4c37-afae-5ee9e28d0fca-tmp\") pod \"console-74d99cdf4-q476v\" (UID: \"f7e5b498-e68d-4c37-afae-5ee9e28d0fca\") " pod="openshift-console/console-74d99cdf4-q476v" Oct 30 12:52:40.588819 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:40.588799 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f7e5b498-e68d-4c37-afae-5ee9e28d0fca-console-serving-cert\") pod \"console-74d99cdf4-q476v\" (UID: \"f7e5b498-e68d-4c37-afae-5ee9e28d0fca\") " pod="openshift-console/console-74d99cdf4-q476v" Oct 30 12:52:40.588973 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:40.588954 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f7e5b498-e68d-4c37-afae-5ee9e28d0fca-console-oauth-config\") pod \"console-74d99cdf4-q476v\" (UID: \"f7e5b498-e68d-4c37-afae-5ee9e28d0fca\") " pod="openshift-console/console-74d99cdf4-q476v" Oct 30 12:52:40.615402 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:40.615384 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krn2z\" (UniqueName: \"kubernetes.io/projected/f7e5b498-e68d-4c37-afae-5ee9e28d0fca-kube-api-access-krn2z\") pod \"console-74d99cdf4-q476v\" (UID: \"f7e5b498-e68d-4c37-afae-5ee9e28d0fca\") " pod="openshift-console/console-74d99cdf4-q476v" Oct 30 12:52:40.669005 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:40.668951 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:40.707821 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:40.707797 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74d99cdf4-q476v" Oct 30 12:52:40.808418 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:40.808384 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:41.015219 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:41.015195 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74d99cdf4-q476v"] Oct 30 12:52:41.017832 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:41.017797 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7e5b498_e68d_4c37_afae_5ee9e28d0fca.slice/crio-8a0ae64c552bc67eb16334bb0934bfcf15b80ae2684d6efde2708e2f1ef9752a WatchSource:0}: Error finding container 8a0ae64c552bc67eb16334bb0934bfcf15b80ae2684d6efde2708e2f1ef9752a: Status 404 returned error can't find the container with id 8a0ae64c552bc67eb16334bb0934bfcf15b80ae2684d6efde2708e2f1ef9752a Oct 30 12:52:41.778705 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:41.778673 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74d99cdf4-q476v" event={"ID":"f7e5b498-e68d-4c37-afae-5ee9e28d0fca","Type":"ContainerStarted","Data":"0a05c4ec501a94d33e4ab1882abe718dcbb4f3a87fb40a876f00a3299b1666e9"} Oct 30 12:52:41.779108 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:41.778730 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74d99cdf4-q476v" event={"ID":"f7e5b498-e68d-4c37-afae-5ee9e28d0fca","Type":"ContainerStarted","Data":"8a0ae64c552bc67eb16334bb0934bfcf15b80ae2684d6efde2708e2f1ef9752a"} Oct 30 12:52:41.822919 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:41.822876 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-74d99cdf4-q476v" podStartSLOduration=1.822863425 podStartE2EDuration="1.822863425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:41.822032426 +0000 UTC m=+18.291448171" watchObservedRunningTime="2026-10-30 12:52:41.822863425 +0000 UTC m=+18.292279243" Oct 30 12:52:42.449027 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:42.448991 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:42.474543 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:42.474513 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 12:52:42.474844 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:42.474821 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" containerName="alertmanager" containerID="cri-o://c9a7dacaac898cf7e5afc6344acdf7f321d449ebd7f94c4fa42c6ac24c83535c" gracePeriod=120 Oct 30 12:52:42.474903 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:42.474875 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" containerName="kube-rbac-proxy-metric" containerID="cri-o://f9faebd70ebc3a8c927575ff0f95fccf09a0cfe531b51c4f1e15b0dcc1b893a0" gracePeriod=120 Oct 30 12:52:42.474957 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:42.474942 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" containerName="config-reloader" containerID="cri-o://152c54719bb4d0ba719cbc354f48222394e0c7a65551449a7f4aa9b18422c840" gracePeriod=120 Oct 30 12:52:42.475019 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:42.474927 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" containerName="prom-label-proxy" containerID="cri-o://459c34ebd1810e07df534942315e093a3b0368420e828a7f3f8b76b44110cfee" gracePeriod=120 Oct 30 12:52:42.475019 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:42.474954 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" containerName="kube-rbac-proxy-web" containerID="cri-o://f286e0644ea0a9ec6d4f6a8fb58ccc6203530437c28865238d9646445df9d24b" gracePeriod=120 Oct 30 12:52:42.475138 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:42.474954 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" containerName="kube-rbac-proxy" containerID="cri-o://0f2ffec3b4f4126d8652136503fd01a5b2b2777198c21196ae9fb1417ffbc09d" gracePeriod=120 Oct 30 12:52:42.786973 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:42.786906 5922 generic.go:356] "Generic (PLEG): container finished" podID="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" containerID="459c34ebd1810e07df534942315e093a3b0368420e828a7f3f8b76b44110cfee" exitCode=0 Oct 30 12:52:42.786973 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:42.786930 5922 generic.go:356] "Generic (PLEG): container finished" podID="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" containerID="0f2ffec3b4f4126d8652136503fd01a5b2b2777198c21196ae9fb1417ffbc09d" exitCode=0 Oct 30 12:52:42.786973 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:42.786936 5922 generic.go:356] "Generic (PLEG): container finished" podID="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" containerID="152c54719bb4d0ba719cbc354f48222394e0c7a65551449a7f4aa9b18422c840" exitCode=0 Oct 30 12:52:42.786973 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:42.786944 5922 generic.go:356] "Generic (PLEG): container finished" podID="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" containerID="c9a7dacaac898cf7e5afc6344acdf7f321d449ebd7f94c4fa42c6ac24c83535c" exitCode=0 Oct 30 12:52:42.787376 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:42.786973 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6","Type":"ContainerDied","Data":"459c34ebd1810e07df534942315e093a3b0368420e828a7f3f8b76b44110cfee"} Oct 30 12:52:42.787376 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:42.787042 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6","Type":"ContainerDied","Data":"0f2ffec3b4f4126d8652136503fd01a5b2b2777198c21196ae9fb1417ffbc09d"} Oct 30 12:52:42.787376 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:42.787071 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6","Type":"ContainerDied","Data":"152c54719bb4d0ba719cbc354f48222394e0c7a65551449a7f4aa9b18422c840"} Oct 30 12:52:42.787376 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:42.787100 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6","Type":"ContainerDied","Data":"c9a7dacaac898cf7e5afc6344acdf7f321d449ebd7f94c4fa42c6ac24c83535c"} Oct 30 12:52:43.700501 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.700468 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:43.742659 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.742640 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:43.792577 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.792549 5922 generic.go:356] "Generic (PLEG): container finished" podID="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" containerID="f9faebd70ebc3a8c927575ff0f95fccf09a0cfe531b51c4f1e15b0dcc1b893a0" exitCode=0 Oct 30 12:52:43.792577 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.792576 5922 generic.go:356] "Generic (PLEG): container finished" podID="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" containerID="f286e0644ea0a9ec6d4f6a8fb58ccc6203530437c28865238d9646445df9d24b" exitCode=0 Oct 30 12:52:43.793002 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.792626 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6","Type":"ContainerDied","Data":"f9faebd70ebc3a8c927575ff0f95fccf09a0cfe531b51c4f1e15b0dcc1b893a0"} Oct 30 12:52:43.793002 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.792641 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:43.793002 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.792682 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6","Type":"ContainerDied","Data":"f286e0644ea0a9ec6d4f6a8fb58ccc6203530437c28865238d9646445df9d24b"} Oct 30 12:52:43.793002 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.792710 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6","Type":"ContainerDied","Data":"58287815249aca0628fe9cfd989113fcd3966286481e51121bf107cd4599d14f"} Oct 30 12:52:43.793002 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.792747 5922 scope.go:122] "RemoveContainer" containerID="459c34ebd1810e07df534942315e093a3b0368420e828a7f3f8b76b44110cfee" Oct 30 12:52:43.800151 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.800133 5922 scope.go:122] "RemoveContainer" containerID="f9faebd70ebc3a8c927575ff0f95fccf09a0cfe531b51c4f1e15b0dcc1b893a0" Oct 30 12:52:43.808334 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.808320 5922 scope.go:122] "RemoveContainer" containerID="0f2ffec3b4f4126d8652136503fd01a5b2b2777198c21196ae9fb1417ffbc09d" Oct 30 12:52:43.814626 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.814599 5922 scope.go:122] "RemoveContainer" containerID="f286e0644ea0a9ec6d4f6a8fb58ccc6203530437c28865238d9646445df9d24b" Oct 30 12:52:43.825248 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.825228 5922 scope.go:122] "RemoveContainer" containerID="152c54719bb4d0ba719cbc354f48222394e0c7a65551449a7f4aa9b18422c840" Oct 30 12:52:43.831910 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.831870 5922 scope.go:122] "RemoveContainer" containerID="c9a7dacaac898cf7e5afc6344acdf7f321d449ebd7f94c4fa42c6ac24c83535c" Oct 30 12:52:43.842095 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.842068 5922 scope.go:122] "RemoveContainer" containerID="6665af12009c0962bf45c992c8dbd0324d4b4e5094dd12d19baa5bd145b23ecc" Oct 30 12:52:43.850126 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.850106 5922 scope.go:122] "RemoveContainer" containerID="459c34ebd1810e07df534942315e093a3b0368420e828a7f3f8b76b44110cfee" Oct 30 12:52:43.850507 ip-10-0-137-21 kubenswrapper[5922]: E1030 12:52:43.850457 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"459c34ebd1810e07df534942315e093a3b0368420e828a7f3f8b76b44110cfee\": container with ID starting with 459c34ebd1810e07df534942315e093a3b0368420e828a7f3f8b76b44110cfee not found: ID does not exist" containerID="459c34ebd1810e07df534942315e093a3b0368420e828a7f3f8b76b44110cfee" Oct 30 12:52:43.850590 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.850519 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"459c34ebd1810e07df534942315e093a3b0368420e828a7f3f8b76b44110cfee"} err="failed to get container status \"459c34ebd1810e07df534942315e093a3b0368420e828a7f3f8b76b44110cfee\": rpc error: code = NotFound desc = could not find container \"459c34ebd1810e07df534942315e093a3b0368420e828a7f3f8b76b44110cfee\": container with ID starting with 459c34ebd1810e07df534942315e093a3b0368420e828a7f3f8b76b44110cfee not found: ID does not exist" Oct 30 12:52:43.850590 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.850570 5922 scope.go:122] "RemoveContainer" containerID="f9faebd70ebc3a8c927575ff0f95fccf09a0cfe531b51c4f1e15b0dcc1b893a0" Oct 30 12:52:43.850931 ip-10-0-137-21 kubenswrapper[5922]: E1030 12:52:43.850908 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9faebd70ebc3a8c927575ff0f95fccf09a0cfe531b51c4f1e15b0dcc1b893a0\": container with ID starting with f9faebd70ebc3a8c927575ff0f95fccf09a0cfe531b51c4f1e15b0dcc1b893a0 not found: ID does not exist" containerID="f9faebd70ebc3a8c927575ff0f95fccf09a0cfe531b51c4f1e15b0dcc1b893a0" Oct 30 12:52:43.850984 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.850938 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9faebd70ebc3a8c927575ff0f95fccf09a0cfe531b51c4f1e15b0dcc1b893a0"} err="failed to get container status \"f9faebd70ebc3a8c927575ff0f95fccf09a0cfe531b51c4f1e15b0dcc1b893a0\": rpc error: code = NotFound desc = could not find container \"f9faebd70ebc3a8c927575ff0f95fccf09a0cfe531b51c4f1e15b0dcc1b893a0\": container with ID starting with f9faebd70ebc3a8c927575ff0f95fccf09a0cfe531b51c4f1e15b0dcc1b893a0 not found: ID does not exist" Oct 30 12:52:43.850984 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.850958 5922 scope.go:122] "RemoveContainer" containerID="0f2ffec3b4f4126d8652136503fd01a5b2b2777198c21196ae9fb1417ffbc09d" Oct 30 12:52:43.851240 ip-10-0-137-21 kubenswrapper[5922]: E1030 12:52:43.851218 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f2ffec3b4f4126d8652136503fd01a5b2b2777198c21196ae9fb1417ffbc09d\": container with ID starting with 0f2ffec3b4f4126d8652136503fd01a5b2b2777198c21196ae9fb1417ffbc09d not found: ID does not exist" containerID="0f2ffec3b4f4126d8652136503fd01a5b2b2777198c21196ae9fb1417ffbc09d" Oct 30 12:52:43.851306 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.851258 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f2ffec3b4f4126d8652136503fd01a5b2b2777198c21196ae9fb1417ffbc09d"} err="failed to get container status \"0f2ffec3b4f4126d8652136503fd01a5b2b2777198c21196ae9fb1417ffbc09d\": rpc error: code = NotFound desc = could not find container \"0f2ffec3b4f4126d8652136503fd01a5b2b2777198c21196ae9fb1417ffbc09d\": container with ID starting with 0f2ffec3b4f4126d8652136503fd01a5b2b2777198c21196ae9fb1417ffbc09d not found: ID does not exist" Oct 30 12:52:43.851306 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.851281 5922 scope.go:122] "RemoveContainer" containerID="f286e0644ea0a9ec6d4f6a8fb58ccc6203530437c28865238d9646445df9d24b" Oct 30 12:52:43.851587 ip-10-0-137-21 kubenswrapper[5922]: E1030 12:52:43.851564 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f286e0644ea0a9ec6d4f6a8fb58ccc6203530437c28865238d9646445df9d24b\": container with ID starting with f286e0644ea0a9ec6d4f6a8fb58ccc6203530437c28865238d9646445df9d24b not found: ID does not exist" containerID="f286e0644ea0a9ec6d4f6a8fb58ccc6203530437c28865238d9646445df9d24b" Oct 30 12:52:43.851684 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.851596 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f286e0644ea0a9ec6d4f6a8fb58ccc6203530437c28865238d9646445df9d24b"} err="failed to get container status \"f286e0644ea0a9ec6d4f6a8fb58ccc6203530437c28865238d9646445df9d24b\": rpc error: code = NotFound desc = could not find container \"f286e0644ea0a9ec6d4f6a8fb58ccc6203530437c28865238d9646445df9d24b\": container with ID starting with f286e0644ea0a9ec6d4f6a8fb58ccc6203530437c28865238d9646445df9d24b not found: ID does not exist" Oct 30 12:52:43.851684 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.851631 5922 scope.go:122] "RemoveContainer" containerID="152c54719bb4d0ba719cbc354f48222394e0c7a65551449a7f4aa9b18422c840" Oct 30 12:52:43.851923 ip-10-0-137-21 kubenswrapper[5922]: E1030 12:52:43.851907 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"152c54719bb4d0ba719cbc354f48222394e0c7a65551449a7f4aa9b18422c840\": container with ID starting with 152c54719bb4d0ba719cbc354f48222394e0c7a65551449a7f4aa9b18422c840 not found: ID does not exist" containerID="152c54719bb4d0ba719cbc354f48222394e0c7a65551449a7f4aa9b18422c840" Oct 30 12:52:43.851983 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.851928 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"152c54719bb4d0ba719cbc354f48222394e0c7a65551449a7f4aa9b18422c840"} err="failed to get container status \"152c54719bb4d0ba719cbc354f48222394e0c7a65551449a7f4aa9b18422c840\": rpc error: code = NotFound desc = could not find container \"152c54719bb4d0ba719cbc354f48222394e0c7a65551449a7f4aa9b18422c840\": container with ID starting with 152c54719bb4d0ba719cbc354f48222394e0c7a65551449a7f4aa9b18422c840 not found: ID does not exist" Oct 30 12:52:43.851983 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.851941 5922 scope.go:122] "RemoveContainer" containerID="c9a7dacaac898cf7e5afc6344acdf7f321d449ebd7f94c4fa42c6ac24c83535c" Oct 30 12:52:43.852227 ip-10-0-137-21 kubenswrapper[5922]: E1030 12:52:43.852192 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9a7dacaac898cf7e5afc6344acdf7f321d449ebd7f94c4fa42c6ac24c83535c\": container with ID starting with c9a7dacaac898cf7e5afc6344acdf7f321d449ebd7f94c4fa42c6ac24c83535c not found: ID does not exist" containerID="c9a7dacaac898cf7e5afc6344acdf7f321d449ebd7f94c4fa42c6ac24c83535c" Oct 30 12:52:43.852298 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.852234 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9a7dacaac898cf7e5afc6344acdf7f321d449ebd7f94c4fa42c6ac24c83535c"} err="failed to get container status \"c9a7dacaac898cf7e5afc6344acdf7f321d449ebd7f94c4fa42c6ac24c83535c\": rpc error: code = NotFound desc = could not find container \"c9a7dacaac898cf7e5afc6344acdf7f321d449ebd7f94c4fa42c6ac24c83535c\": container with ID starting with c9a7dacaac898cf7e5afc6344acdf7f321d449ebd7f94c4fa42c6ac24c83535c not found: ID does not exist" Oct 30 12:52:43.852298 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.852253 5922 scope.go:122] "RemoveContainer" containerID="6665af12009c0962bf45c992c8dbd0324d4b4e5094dd12d19baa5bd145b23ecc" Oct 30 12:52:43.852495 ip-10-0-137-21 kubenswrapper[5922]: E1030 12:52:43.852479 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6665af12009c0962bf45c992c8dbd0324d4b4e5094dd12d19baa5bd145b23ecc\": container with ID starting with 6665af12009c0962bf45c992c8dbd0324d4b4e5094dd12d19baa5bd145b23ecc not found: ID does not exist" containerID="6665af12009c0962bf45c992c8dbd0324d4b4e5094dd12d19baa5bd145b23ecc" Oct 30 12:52:43.852541 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.852500 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6665af12009c0962bf45c992c8dbd0324d4b4e5094dd12d19baa5bd145b23ecc"} err="failed to get container status \"6665af12009c0962bf45c992c8dbd0324d4b4e5094dd12d19baa5bd145b23ecc\": rpc error: code = NotFound desc = could not find container \"6665af12009c0962bf45c992c8dbd0324d4b4e5094dd12d19baa5bd145b23ecc\": container with ID starting with 6665af12009c0962bf45c992c8dbd0324d4b4e5094dd12d19baa5bd145b23ecc not found: ID does not exist" Oct 30 12:52:43.852541 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.852513 5922 scope.go:122] "RemoveContainer" containerID="459c34ebd1810e07df534942315e093a3b0368420e828a7f3f8b76b44110cfee" Oct 30 12:52:43.853080 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.853050 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"459c34ebd1810e07df534942315e093a3b0368420e828a7f3f8b76b44110cfee"} err="failed to get container status \"459c34ebd1810e07df534942315e093a3b0368420e828a7f3f8b76b44110cfee\": rpc error: code = NotFound desc = could not find container \"459c34ebd1810e07df534942315e093a3b0368420e828a7f3f8b76b44110cfee\": container with ID starting with 459c34ebd1810e07df534942315e093a3b0368420e828a7f3f8b76b44110cfee not found: ID does not exist" Oct 30 12:52:43.853150 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.853081 5922 scope.go:122] "RemoveContainer" containerID="f9faebd70ebc3a8c927575ff0f95fccf09a0cfe531b51c4f1e15b0dcc1b893a0" Oct 30 12:52:43.853424 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.853386 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9faebd70ebc3a8c927575ff0f95fccf09a0cfe531b51c4f1e15b0dcc1b893a0"} err="failed to get container status \"f9faebd70ebc3a8c927575ff0f95fccf09a0cfe531b51c4f1e15b0dcc1b893a0\": rpc error: code = NotFound desc = could not find container \"f9faebd70ebc3a8c927575ff0f95fccf09a0cfe531b51c4f1e15b0dcc1b893a0\": container with ID starting with f9faebd70ebc3a8c927575ff0f95fccf09a0cfe531b51c4f1e15b0dcc1b893a0 not found: ID does not exist" Oct 30 12:52:43.853424 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.853415 5922 scope.go:122] "RemoveContainer" containerID="0f2ffec3b4f4126d8652136503fd01a5b2b2777198c21196ae9fb1417ffbc09d" Oct 30 12:52:43.853783 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.853757 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f2ffec3b4f4126d8652136503fd01a5b2b2777198c21196ae9fb1417ffbc09d"} err="failed to get container status \"0f2ffec3b4f4126d8652136503fd01a5b2b2777198c21196ae9fb1417ffbc09d\": rpc error: code = NotFound desc = could not find container \"0f2ffec3b4f4126d8652136503fd01a5b2b2777198c21196ae9fb1417ffbc09d\": container with ID starting with 0f2ffec3b4f4126d8652136503fd01a5b2b2777198c21196ae9fb1417ffbc09d not found: ID does not exist" Oct 30 12:52:43.853783 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.853782 5922 scope.go:122] "RemoveContainer" containerID="f286e0644ea0a9ec6d4f6a8fb58ccc6203530437c28865238d9646445df9d24b" Oct 30 12:52:43.854060 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.854043 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f286e0644ea0a9ec6d4f6a8fb58ccc6203530437c28865238d9646445df9d24b"} err="failed to get container status \"f286e0644ea0a9ec6d4f6a8fb58ccc6203530437c28865238d9646445df9d24b\": rpc error: code = NotFound desc = could not find container \"f286e0644ea0a9ec6d4f6a8fb58ccc6203530437c28865238d9646445df9d24b\": container with ID starting with f286e0644ea0a9ec6d4f6a8fb58ccc6203530437c28865238d9646445df9d24b not found: ID does not exist" Oct 30 12:52:43.854113 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.854059 5922 scope.go:122] "RemoveContainer" containerID="152c54719bb4d0ba719cbc354f48222394e0c7a65551449a7f4aa9b18422c840" Oct 30 12:52:43.854304 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.854282 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"152c54719bb4d0ba719cbc354f48222394e0c7a65551449a7f4aa9b18422c840"} err="failed to get container status \"152c54719bb4d0ba719cbc354f48222394e0c7a65551449a7f4aa9b18422c840\": rpc error: code = NotFound desc = could not find container \"152c54719bb4d0ba719cbc354f48222394e0c7a65551449a7f4aa9b18422c840\": container with ID starting with 152c54719bb4d0ba719cbc354f48222394e0c7a65551449a7f4aa9b18422c840 not found: ID does not exist" Oct 30 12:52:43.854379 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.854306 5922 scope.go:122] "RemoveContainer" containerID="c9a7dacaac898cf7e5afc6344acdf7f321d449ebd7f94c4fa42c6ac24c83535c" Oct 30 12:52:43.854572 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.854553 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9a7dacaac898cf7e5afc6344acdf7f321d449ebd7f94c4fa42c6ac24c83535c"} err="failed to get container status \"c9a7dacaac898cf7e5afc6344acdf7f321d449ebd7f94c4fa42c6ac24c83535c\": rpc error: code = NotFound desc = could not find container \"c9a7dacaac898cf7e5afc6344acdf7f321d449ebd7f94c4fa42c6ac24c83535c\": container with ID starting with c9a7dacaac898cf7e5afc6344acdf7f321d449ebd7f94c4fa42c6ac24c83535c not found: ID does not exist" Oct 30 12:52:43.854679 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.854573 5922 scope.go:122] "RemoveContainer" containerID="6665af12009c0962bf45c992c8dbd0324d4b4e5094dd12d19baa5bd145b23ecc" Oct 30 12:52:43.854880 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.854863 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6665af12009c0962bf45c992c8dbd0324d4b4e5094dd12d19baa5bd145b23ecc"} err="failed to get container status \"6665af12009c0962bf45c992c8dbd0324d4b4e5094dd12d19baa5bd145b23ecc\": rpc error: code = NotFound desc = could not find container \"6665af12009c0962bf45c992c8dbd0324d4b4e5094dd12d19baa5bd145b23ecc\": container with ID starting with 6665af12009c0962bf45c992c8dbd0324d4b4e5094dd12d19baa5bd145b23ecc not found: ID does not exist" Oct 30 12:52:43.920211 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.920181 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-config-out\" (UniqueName: \"kubernetes.io/empty-dir/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-config-out\") pod \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " Oct 30 12:52:43.920323 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.920227 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-kube-api-access-82x4j\" (UniqueName: \"kubernetes.io/projected/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-kube-api-access-82x4j\") pod \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " Oct 30 12:52:43.920323 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.920254 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-web-config\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-web-config\") pod \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " Oct 30 12:52:43.920323 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.920273 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-alertmanager-trusted-ca-bundle\") pod \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " Oct 30 12:52:43.920323 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.920315 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-secret-alertmanager-kube-rbac-proxy\") pod \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " Oct 30 12:52:43.920585 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.920341 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-secret-alertmanager-kube-rbac-proxy-web\") pod \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " Oct 30 12:52:43.920585 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.920384 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " Oct 30 12:52:43.920585 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.920412 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-alertmanager-main-db\") pod \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " Oct 30 12:52:43.920585 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.920441 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-cluster-tls-config\") pod \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " Oct 30 12:52:43.920585 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.920494 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-config-volume\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-config-volume\") pod \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " Oct 30 12:52:43.920585 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.920532 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-metrics-client-ca\") pod \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " Oct 30 12:52:43.920585 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.920568 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-tls-assets\" (UniqueName: \"kubernetes.io/projected/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-tls-assets\") pod \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " Oct 30 12:52:43.920966 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.920605 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-secret-alertmanager-main-tls\") pod \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\" (UID: \"c18cb3fe-0392-4dca-b636-4e3ab99eb0f6\") " Oct 30 12:52:43.920966 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.920718 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-alertmanager-trusted-ca-bundle" pod "c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" (UID: "c18cb3fe-0392-4dca-b636-4e3ab99eb0f6"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:52:43.921075 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.920963 5922 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:52:43.921129 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.921096 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-alertmanager-main-db" pod "c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" (UID: "c18cb3fe-0392-4dca-b636-4e3ab99eb0f6"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:52:43.921460 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.921433 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-metrics-client-ca" pod "c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" (UID: "c18cb3fe-0392-4dca-b636-4e3ab99eb0f6"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:52:43.923121 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.923094 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-secret-alertmanager-kube-rbac-proxy-web" pod "c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" (UID: "c18cb3fe-0392-4dca-b636-4e3ab99eb0f6"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:52:43.924057 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.924014 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-secret-alertmanager-kube-rbac-proxy" pod "c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" (UID: "c18cb3fe-0392-4dca-b636-4e3ab99eb0f6"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:52:43.924232 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.924199 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-secret-alertmanager-main-tls" pod "c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" (UID: "c18cb3fe-0392-4dca-b636-4e3ab99eb0f6"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:52:43.924364 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.924254 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-config-out" pod "c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" (UID: "c18cb3fe-0392-4dca-b636-4e3ab99eb0f6"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:52:43.924364 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.924318 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-kube-api-access-82x4j" pod "c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" (UID: "c18cb3fe-0392-4dca-b636-4e3ab99eb0f6"). InnerVolumeSpecName "kube-api-access-82x4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:52:43.924364 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.924342 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-config-volume" pod "c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" (UID: "c18cb3fe-0392-4dca-b636-4e3ab99eb0f6"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:52:43.924758 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.924727 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-secret-alertmanager-kube-rbac-proxy-metric" pod "c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" (UID: "c18cb3fe-0392-4dca-b636-4e3ab99eb0f6"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:52:43.925281 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.925260 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-tls-assets" pod "c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" (UID: "c18cb3fe-0392-4dca-b636-4e3ab99eb0f6"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:52:43.928335 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.928317 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-cluster-tls-config" pod "c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" (UID: "c18cb3fe-0392-4dca-b636-4e3ab99eb0f6"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:52:43.934730 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:43.934710 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-web-config" pod "c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" (UID: "c18cb3fe-0392-4dca-b636-4e3ab99eb0f6"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:52:44.022051 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:44.022027 5922 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:52:44.022051 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:44.022050 5922 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:52:44.022164 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:44.022060 5922 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:52:44.022164 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:44.022070 5922 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-alertmanager-main-db\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:52:44.022164 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:44.022080 5922 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-cluster-tls-config\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:52:44.022164 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:44.022088 5922 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-config-volume\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:52:44.022164 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:44.022097 5922 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-metrics-client-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:52:44.022164 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:44.022104 5922 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-tls-assets\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:52:44.022164 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:44.022113 5922 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-secret-alertmanager-main-tls\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:52:44.022164 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:44.022121 5922 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-config-out\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:52:44.022164 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:44.022129 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82x4j\" (UniqueName: \"kubernetes.io/projected/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-kube-api-access-82x4j\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:52:44.022164 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:44.022137 5922 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6-web-config\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:52:44.139302 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:44.139280 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 12:52:44.150975 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:44.150952 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 12:52:44.721401 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:44.721361 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:44.849456 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:44.849390 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:45.642371 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:45.642339 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" path="/var/lib/kubelet/pods/c18cb3fe-0392-4dca-b636-4e3ab99eb0f6/volumes" Oct 30 12:52:47.057800 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:47.057772 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2nqhp" Oct 30 12:52:47.306243 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:47.306208 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 12:52:47.306561 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:47.306536 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ecd3d4d5-a59a-4b0e-a981-956b10d08531" containerName="prometheus" containerID="cri-o://f758417ec1675d0ce680c01e967da15988b5cc0396604b5b43b45c7a87d11498" gracePeriod=600 Oct 30 12:52:47.306703 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:47.306570 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ecd3d4d5-a59a-4b0e-a981-956b10d08531" containerName="kube-rbac-proxy" containerID="cri-o://3d067f86de45fb7da4d703f9d8ddfdf02e51e9db07b94ac0e02685e0a8f0e5f7" gracePeriod=600 Oct 30 12:52:47.306703 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:47.306589 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ecd3d4d5-a59a-4b0e-a981-956b10d08531" containerName="kube-rbac-proxy-thanos" containerID="cri-o://733c9a46d88bf34e394c3f801c2a74a66a16cf1b882c2f5efbc9ca100a4714e6" gracePeriod=600 Oct 30 12:52:47.306703 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:47.306600 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ecd3d4d5-a59a-4b0e-a981-956b10d08531" containerName="thanos-sidecar" containerID="cri-o://966d7771b9f840d69179f59a1557d5269698ef1b542ddcf4e8791e225816c6d4" gracePeriod=600 Oct 30 12:52:47.306843 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:47.306724 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ecd3d4d5-a59a-4b0e-a981-956b10d08531" containerName="kube-rbac-proxy-web" containerID="cri-o://c626537d69a0eccf8091bdf1e44c648d000558fd2f939e9f7c4124f238c3219d" gracePeriod=600 Oct 30 12:52:47.306843 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:47.306725 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ecd3d4d5-a59a-4b0e-a981-956b10d08531" containerName="config-reloader" containerID="cri-o://389ef54daca67f771705e3e74fe397815377121bb962ae7740b822516e1ce5a3" gracePeriod=600 Oct 30 12:52:47.812838 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:47.812802 5922 generic.go:356] "Generic (PLEG): container finished" podID="ecd3d4d5-a59a-4b0e-a981-956b10d08531" containerID="c626537d69a0eccf8091bdf1e44c648d000558fd2f939e9f7c4124f238c3219d" exitCode=0 Oct 30 12:52:47.812838 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:47.812834 5922 generic.go:356] "Generic (PLEG): container finished" podID="ecd3d4d5-a59a-4b0e-a981-956b10d08531" containerID="966d7771b9f840d69179f59a1557d5269698ef1b542ddcf4e8791e225816c6d4" exitCode=0 Oct 30 12:52:47.812838 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:47.812845 5922 generic.go:356] "Generic (PLEG): container finished" podID="ecd3d4d5-a59a-4b0e-a981-956b10d08531" containerID="389ef54daca67f771705e3e74fe397815377121bb962ae7740b822516e1ce5a3" exitCode=0 Oct 30 12:52:47.813082 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:47.812871 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ecd3d4d5-a59a-4b0e-a981-956b10d08531","Type":"ContainerDied","Data":"c626537d69a0eccf8091bdf1e44c648d000558fd2f939e9f7c4124f238c3219d"} Oct 30 12:52:47.813082 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:47.812956 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ecd3d4d5-a59a-4b0e-a981-956b10d08531","Type":"ContainerDied","Data":"966d7771b9f840d69179f59a1557d5269698ef1b542ddcf4e8791e225816c6d4"} Oct 30 12:52:47.813082 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:47.812990 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ecd3d4d5-a59a-4b0e-a981-956b10d08531","Type":"ContainerDied","Data":"389ef54daca67f771705e3e74fe397815377121bb962ae7740b822516e1ce5a3"} Oct 30 12:52:48.573917 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.573892 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:48.659090 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.659035 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-configmap-metrics-client-ca\") pod \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " Oct 30 12:52:48.659090 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.659073 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-k8s-rulefiles-0\") pod \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " Oct 30 12:52:48.659227 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.659100 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-config\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-config\") pod \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " Oct 30 12:52:48.659227 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.659133 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-grpc-tls\") pod \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " Oct 30 12:52:48.659227 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.659171 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-metrics-client-certs\") pod \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " Oct 30 12:52:48.659227 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.659198 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-configmap-serving-certs-ca-bundle\") pod \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " Oct 30 12:52:48.659423 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.659239 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " Oct 30 12:52:48.659423 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.659271 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-k8s-rulefiles-1\") pod \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " Oct 30 12:52:48.659423 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.659300 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecd3d4d5-a59a-4b0e-a981-956b10d08531-kube-api-access-d876w\" (UniqueName: \"kubernetes.io/projected/ecd3d4d5-a59a-4b0e-a981-956b10d08531-kube-api-access-d876w\") pod \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " Oct 30 12:52:48.659423 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.659333 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " Oct 30 12:52:48.659423 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.659375 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-k8s-db\") pod \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " Oct 30 12:52:48.659423 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.659408 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-kube-rbac-proxy\") pod \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " Oct 30 12:52:48.659743 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.659451 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecd3d4d5-a59a-4b0e-a981-956b10d08531-config-out\" (UniqueName: \"kubernetes.io/empty-dir/ecd3d4d5-a59a-4b0e-a981-956b10d08531-config-out\") pod \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " Oct 30 12:52:48.659743 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.659451 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-configmap-metrics-client-ca" pod "ecd3d4d5-a59a-4b0e-a981-956b10d08531" (UID: "ecd3d4d5-a59a-4b0e-a981-956b10d08531"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:52:48.659743 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.659480 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-prometheus-k8s-tls\") pod \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " Oct 30 12:52:48.659743 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.659651 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecd3d4d5-a59a-4b0e-a981-956b10d08531-tls-assets\" (UniqueName: \"kubernetes.io/projected/ecd3d4d5-a59a-4b0e-a981-956b10d08531-tls-assets\") pod \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " Oct 30 12:52:48.659743 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.659692 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-web-config\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-web-config\") pod \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " Oct 30 12:52:48.659743 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.659698 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-configmap-serving-certs-ca-bundle" pod "ecd3d4d5-a59a-4b0e-a981-956b10d08531" (UID: "ecd3d4d5-a59a-4b0e-a981-956b10d08531"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:52:48.659743 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.659730 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-thanos-prometheus-http-client-file\") pod \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " Oct 30 12:52:48.660116 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.659772 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-k8s-rulefiles-2\") pod \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " Oct 30 12:52:48.660116 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.659804 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-trusted-ca-bundle\") pod \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " Oct 30 12:52:48.660116 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.659836 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-configmap-kubelet-serving-ca-bundle\") pod \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\" (UID: \"ecd3d4d5-a59a-4b0e-a981-956b10d08531\") " Oct 30 12:52:48.660116 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.660109 5922 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-configmap-metrics-client-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:52:48.660305 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.660133 5922 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:52:48.660363 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.660336 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-k8s-rulefiles-1" pod "ecd3d4d5-a59a-4b0e-a981-956b10d08531" (UID: "ecd3d4d5-a59a-4b0e-a981-956b10d08531"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:52:48.660682 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.660472 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-configmap-kubelet-serving-ca-bundle" pod "ecd3d4d5-a59a-4b0e-a981-956b10d08531" (UID: "ecd3d4d5-a59a-4b0e-a981-956b10d08531"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:52:48.660682 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.660670 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-k8s-rulefiles-2" pod "ecd3d4d5-a59a-4b0e-a981-956b10d08531" (UID: "ecd3d4d5-a59a-4b0e-a981-956b10d08531"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:52:48.660935 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.660899 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-trusted-ca-bundle" pod "ecd3d4d5-a59a-4b0e-a981-956b10d08531" (UID: "ecd3d4d5-a59a-4b0e-a981-956b10d08531"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:52:48.661015 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.660997 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-k8s-rulefiles-0" pod "ecd3d4d5-a59a-4b0e-a981-956b10d08531" (UID: "ecd3d4d5-a59a-4b0e-a981-956b10d08531"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:52:48.662030 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.661992 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-prometheus-k8s-tls" pod "ecd3d4d5-a59a-4b0e-a981-956b10d08531" (UID: "ecd3d4d5-a59a-4b0e-a981-956b10d08531"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:52:48.662330 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.662233 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-grpc-tls" pod "ecd3d4d5-a59a-4b0e-a981-956b10d08531" (UID: "ecd3d4d5-a59a-4b0e-a981-956b10d08531"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:52:48.662330 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.662249 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-k8s-db" pod "ecd3d4d5-a59a-4b0e-a981-956b10d08531" (UID: "ecd3d4d5-a59a-4b0e-a981-956b10d08531"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:52:48.662646 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.662586 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-prometheus-k8s-kube-rbac-proxy-web" pod "ecd3d4d5-a59a-4b0e-a981-956b10d08531" (UID: "ecd3d4d5-a59a-4b0e-a981-956b10d08531"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:52:48.663971 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.663944 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-metrics-client-certs" pod "ecd3d4d5-a59a-4b0e-a981-956b10d08531" (UID: "ecd3d4d5-a59a-4b0e-a981-956b10d08531"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:52:48.664141 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.664099 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-config" pod "ecd3d4d5-a59a-4b0e-a981-956b10d08531" (UID: "ecd3d4d5-a59a-4b0e-a981-956b10d08531"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:52:48.664399 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.664348 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-prometheus-k8s-thanos-sidecar-tls" pod "ecd3d4d5-a59a-4b0e-a981-956b10d08531" (UID: "ecd3d4d5-a59a-4b0e-a981-956b10d08531"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:52:48.664485 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.664422 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-kube-rbac-proxy" pod "ecd3d4d5-a59a-4b0e-a981-956b10d08531" (UID: "ecd3d4d5-a59a-4b0e-a981-956b10d08531"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:52:48.664573 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.664546 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecd3d4d5-a59a-4b0e-a981-956b10d08531-kube-api-access-d876w" pod "ecd3d4d5-a59a-4b0e-a981-956b10d08531" (UID: "ecd3d4d5-a59a-4b0e-a981-956b10d08531"). InnerVolumeSpecName "kube-api-access-d876w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:52:48.664724 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.664700 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecd3d4d5-a59a-4b0e-a981-956b10d08531-config-out" pod "ecd3d4d5-a59a-4b0e-a981-956b10d08531" (UID: "ecd3d4d5-a59a-4b0e-a981-956b10d08531"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:52:48.665030 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.665002 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecd3d4d5-a59a-4b0e-a981-956b10d08531-tls-assets" pod "ecd3d4d5-a59a-4b0e-a981-956b10d08531" (UID: "ecd3d4d5-a59a-4b0e-a981-956b10d08531"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:52:48.665197 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.665175 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-thanos-prometheus-http-client-file" pod "ecd3d4d5-a59a-4b0e-a981-956b10d08531" (UID: "ecd3d4d5-a59a-4b0e-a981-956b10d08531"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:52:48.676397 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.676372 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-web-config" pod "ecd3d4d5-a59a-4b0e-a981-956b10d08531" (UID: "ecd3d4d5-a59a-4b0e-a981-956b10d08531"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:52:48.760856 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.760816 5922 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-kube-rbac-proxy\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:52:48.760856 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.760846 5922 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ecd3d4d5-a59a-4b0e-a981-956b10d08531-config-out\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:52:48.760856 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.760857 5922 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:52:48.761055 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.760866 5922 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ecd3d4d5-a59a-4b0e-a981-956b10d08531-tls-assets\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:52:48.761055 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.760875 5922 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-web-config\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:52:48.761055 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.760884 5922 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:52:48.761055 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.760893 5922 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:52:48.761055 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.760902 5922 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:52:48.761055 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.760921 5922 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:52:48.761055 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.760929 5922 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:52:48.761055 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.760937 5922 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-config\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:52:48.761055 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.760945 5922 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-grpc-tls\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:52:48.761055 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.760954 5922 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-metrics-client-certs\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:52:48.761055 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.760961 5922 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:52:48.761055 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.760970 5922 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:52:48.761055 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.760978 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d876w\" (UniqueName: \"kubernetes.io/projected/ecd3d4d5-a59a-4b0e-a981-956b10d08531-kube-api-access-d876w\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:52:48.761055 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.760986 5922 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ecd3d4d5-a59a-4b0e-a981-956b10d08531-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:52:48.761055 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.760995 5922 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ecd3d4d5-a59a-4b0e-a981-956b10d08531-prometheus-k8s-db\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:52:48.820047 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.820014 5922 generic.go:356] "Generic (PLEG): container finished" podID="ecd3d4d5-a59a-4b0e-a981-956b10d08531" containerID="733c9a46d88bf34e394c3f801c2a74a66a16cf1b882c2f5efbc9ca100a4714e6" exitCode=0 Oct 30 12:52:48.820047 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.820045 5922 generic.go:356] "Generic (PLEG): container finished" podID="ecd3d4d5-a59a-4b0e-a981-956b10d08531" containerID="3d067f86de45fb7da4d703f9d8ddfdf02e51e9db07b94ac0e02685e0a8f0e5f7" exitCode=0 Oct 30 12:52:48.820205 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.820056 5922 generic.go:356] "Generic (PLEG): container finished" podID="ecd3d4d5-a59a-4b0e-a981-956b10d08531" containerID="f758417ec1675d0ce680c01e967da15988b5cc0396604b5b43b45c7a87d11498" exitCode=0 Oct 30 12:52:48.820205 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.820109 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:48.820205 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.820101 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ecd3d4d5-a59a-4b0e-a981-956b10d08531","Type":"ContainerDied","Data":"733c9a46d88bf34e394c3f801c2a74a66a16cf1b882c2f5efbc9ca100a4714e6"} Oct 30 12:52:48.820295 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.820251 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ecd3d4d5-a59a-4b0e-a981-956b10d08531","Type":"ContainerDied","Data":"3d067f86de45fb7da4d703f9d8ddfdf02e51e9db07b94ac0e02685e0a8f0e5f7"} Oct 30 12:52:48.820295 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.820287 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ecd3d4d5-a59a-4b0e-a981-956b10d08531","Type":"ContainerDied","Data":"f758417ec1675d0ce680c01e967da15988b5cc0396604b5b43b45c7a87d11498"} Oct 30 12:52:48.820355 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.820320 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ecd3d4d5-a59a-4b0e-a981-956b10d08531","Type":"ContainerDied","Data":"3410ea027ea71f3e80e819cc87d3a28cf98bce131de13e4a81f8b081e6458cae"} Oct 30 12:52:48.820355 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.820326 5922 scope.go:122] "RemoveContainer" containerID="733c9a46d88bf34e394c3f801c2a74a66a16cf1b882c2f5efbc9ca100a4714e6" Oct 30 12:52:48.827846 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.827749 5922 scope.go:122] "RemoveContainer" containerID="3d067f86de45fb7da4d703f9d8ddfdf02e51e9db07b94ac0e02685e0a8f0e5f7" Oct 30 12:52:48.834443 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.834422 5922 scope.go:122] "RemoveContainer" containerID="c626537d69a0eccf8091bdf1e44c648d000558fd2f939e9f7c4124f238c3219d" Oct 30 12:52:48.840607 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.840592 5922 scope.go:122] "RemoveContainer" containerID="966d7771b9f840d69179f59a1557d5269698ef1b542ddcf4e8791e225816c6d4" Oct 30 12:52:48.847120 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.847102 5922 scope.go:122] "RemoveContainer" containerID="389ef54daca67f771705e3e74fe397815377121bb962ae7740b822516e1ce5a3" Oct 30 12:52:48.853283 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.853269 5922 scope.go:122] "RemoveContainer" containerID="f758417ec1675d0ce680c01e967da15988b5cc0396604b5b43b45c7a87d11498" Oct 30 12:52:48.859846 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.859831 5922 scope.go:122] "RemoveContainer" containerID="2bdb163e8f97e8e5afd18e9046bd2c976fa179cc65c409ed26d34fa58f041ab4" Oct 30 12:52:48.866094 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.866078 5922 scope.go:122] "RemoveContainer" containerID="733c9a46d88bf34e394c3f801c2a74a66a16cf1b882c2f5efbc9ca100a4714e6" Oct 30 12:52:48.866345 ip-10-0-137-21 kubenswrapper[5922]: E1030 12:52:48.866326 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"733c9a46d88bf34e394c3f801c2a74a66a16cf1b882c2f5efbc9ca100a4714e6\": container with ID starting with 733c9a46d88bf34e394c3f801c2a74a66a16cf1b882c2f5efbc9ca100a4714e6 not found: ID does not exist" containerID="733c9a46d88bf34e394c3f801c2a74a66a16cf1b882c2f5efbc9ca100a4714e6" Oct 30 12:52:48.866394 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.866356 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"733c9a46d88bf34e394c3f801c2a74a66a16cf1b882c2f5efbc9ca100a4714e6"} err="failed to get container status \"733c9a46d88bf34e394c3f801c2a74a66a16cf1b882c2f5efbc9ca100a4714e6\": rpc error: code = NotFound desc = could not find container \"733c9a46d88bf34e394c3f801c2a74a66a16cf1b882c2f5efbc9ca100a4714e6\": container with ID starting with 733c9a46d88bf34e394c3f801c2a74a66a16cf1b882c2f5efbc9ca100a4714e6 not found: ID does not exist" Oct 30 12:52:48.866394 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.866377 5922 scope.go:122] "RemoveContainer" containerID="3d067f86de45fb7da4d703f9d8ddfdf02e51e9db07b94ac0e02685e0a8f0e5f7" Oct 30 12:52:48.866597 ip-10-0-137-21 kubenswrapper[5922]: E1030 12:52:48.866582 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d067f86de45fb7da4d703f9d8ddfdf02e51e9db07b94ac0e02685e0a8f0e5f7\": container with ID starting with 3d067f86de45fb7da4d703f9d8ddfdf02e51e9db07b94ac0e02685e0a8f0e5f7 not found: ID does not exist" containerID="3d067f86de45fb7da4d703f9d8ddfdf02e51e9db07b94ac0e02685e0a8f0e5f7" Oct 30 12:52:48.866652 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.866603 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d067f86de45fb7da4d703f9d8ddfdf02e51e9db07b94ac0e02685e0a8f0e5f7"} err="failed to get container status \"3d067f86de45fb7da4d703f9d8ddfdf02e51e9db07b94ac0e02685e0a8f0e5f7\": rpc error: code = NotFound desc = could not find container \"3d067f86de45fb7da4d703f9d8ddfdf02e51e9db07b94ac0e02685e0a8f0e5f7\": container with ID starting with 3d067f86de45fb7da4d703f9d8ddfdf02e51e9db07b94ac0e02685e0a8f0e5f7 not found: ID does not exist" Oct 30 12:52:48.866652 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.866629 5922 scope.go:122] "RemoveContainer" containerID="c626537d69a0eccf8091bdf1e44c648d000558fd2f939e9f7c4124f238c3219d" Oct 30 12:52:48.866851 ip-10-0-137-21 kubenswrapper[5922]: E1030 12:52:48.866836 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c626537d69a0eccf8091bdf1e44c648d000558fd2f939e9f7c4124f238c3219d\": container with ID starting with c626537d69a0eccf8091bdf1e44c648d000558fd2f939e9f7c4124f238c3219d not found: ID does not exist" containerID="c626537d69a0eccf8091bdf1e44c648d000558fd2f939e9f7c4124f238c3219d" Oct 30 12:52:48.866894 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.866855 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c626537d69a0eccf8091bdf1e44c648d000558fd2f939e9f7c4124f238c3219d"} err="failed to get container status \"c626537d69a0eccf8091bdf1e44c648d000558fd2f939e9f7c4124f238c3219d\": rpc error: code = NotFound desc = could not find container \"c626537d69a0eccf8091bdf1e44c648d000558fd2f939e9f7c4124f238c3219d\": container with ID starting with c626537d69a0eccf8091bdf1e44c648d000558fd2f939e9f7c4124f238c3219d not found: ID does not exist" Oct 30 12:52:48.866894 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.866868 5922 scope.go:122] "RemoveContainer" containerID="966d7771b9f840d69179f59a1557d5269698ef1b542ddcf4e8791e225816c6d4" Oct 30 12:52:48.867079 ip-10-0-137-21 kubenswrapper[5922]: E1030 12:52:48.867064 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"966d7771b9f840d69179f59a1557d5269698ef1b542ddcf4e8791e225816c6d4\": container with ID starting with 966d7771b9f840d69179f59a1557d5269698ef1b542ddcf4e8791e225816c6d4 not found: ID does not exist" containerID="966d7771b9f840d69179f59a1557d5269698ef1b542ddcf4e8791e225816c6d4" Oct 30 12:52:48.867118 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.867082 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"966d7771b9f840d69179f59a1557d5269698ef1b542ddcf4e8791e225816c6d4"} err="failed to get container status \"966d7771b9f840d69179f59a1557d5269698ef1b542ddcf4e8791e225816c6d4\": rpc error: code = NotFound desc = could not find container \"966d7771b9f840d69179f59a1557d5269698ef1b542ddcf4e8791e225816c6d4\": container with ID starting with 966d7771b9f840d69179f59a1557d5269698ef1b542ddcf4e8791e225816c6d4 not found: ID does not exist" Oct 30 12:52:48.867118 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.867094 5922 scope.go:122] "RemoveContainer" containerID="389ef54daca67f771705e3e74fe397815377121bb962ae7740b822516e1ce5a3" Oct 30 12:52:48.867289 ip-10-0-137-21 kubenswrapper[5922]: E1030 12:52:48.867276 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"389ef54daca67f771705e3e74fe397815377121bb962ae7740b822516e1ce5a3\": container with ID starting with 389ef54daca67f771705e3e74fe397815377121bb962ae7740b822516e1ce5a3 not found: ID does not exist" containerID="389ef54daca67f771705e3e74fe397815377121bb962ae7740b822516e1ce5a3" Oct 30 12:52:48.867325 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.867294 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"389ef54daca67f771705e3e74fe397815377121bb962ae7740b822516e1ce5a3"} err="failed to get container status \"389ef54daca67f771705e3e74fe397815377121bb962ae7740b822516e1ce5a3\": rpc error: code = NotFound desc = could not find container \"389ef54daca67f771705e3e74fe397815377121bb962ae7740b822516e1ce5a3\": container with ID starting with 389ef54daca67f771705e3e74fe397815377121bb962ae7740b822516e1ce5a3 not found: ID does not exist" Oct 30 12:52:48.867325 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.867305 5922 scope.go:122] "RemoveContainer" containerID="f758417ec1675d0ce680c01e967da15988b5cc0396604b5b43b45c7a87d11498" Oct 30 12:52:48.867497 ip-10-0-137-21 kubenswrapper[5922]: E1030 12:52:48.867478 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f758417ec1675d0ce680c01e967da15988b5cc0396604b5b43b45c7a87d11498\": container with ID starting with f758417ec1675d0ce680c01e967da15988b5cc0396604b5b43b45c7a87d11498 not found: ID does not exist" containerID="f758417ec1675d0ce680c01e967da15988b5cc0396604b5b43b45c7a87d11498" Oct 30 12:52:48.867545 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.867501 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f758417ec1675d0ce680c01e967da15988b5cc0396604b5b43b45c7a87d11498"} err="failed to get container status \"f758417ec1675d0ce680c01e967da15988b5cc0396604b5b43b45c7a87d11498\": rpc error: code = NotFound desc = could not find container \"f758417ec1675d0ce680c01e967da15988b5cc0396604b5b43b45c7a87d11498\": container with ID starting with f758417ec1675d0ce680c01e967da15988b5cc0396604b5b43b45c7a87d11498 not found: ID does not exist" Oct 30 12:52:48.867545 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.867513 5922 scope.go:122] "RemoveContainer" containerID="2bdb163e8f97e8e5afd18e9046bd2c976fa179cc65c409ed26d34fa58f041ab4" Oct 30 12:52:48.867734 ip-10-0-137-21 kubenswrapper[5922]: E1030 12:52:48.867719 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bdb163e8f97e8e5afd18e9046bd2c976fa179cc65c409ed26d34fa58f041ab4\": container with ID starting with 2bdb163e8f97e8e5afd18e9046bd2c976fa179cc65c409ed26d34fa58f041ab4 not found: ID does not exist" containerID="2bdb163e8f97e8e5afd18e9046bd2c976fa179cc65c409ed26d34fa58f041ab4" Oct 30 12:52:48.867786 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.867737 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bdb163e8f97e8e5afd18e9046bd2c976fa179cc65c409ed26d34fa58f041ab4"} err="failed to get container status \"2bdb163e8f97e8e5afd18e9046bd2c976fa179cc65c409ed26d34fa58f041ab4\": rpc error: code = NotFound desc = could not find container \"2bdb163e8f97e8e5afd18e9046bd2c976fa179cc65c409ed26d34fa58f041ab4\": container with ID starting with 2bdb163e8f97e8e5afd18e9046bd2c976fa179cc65c409ed26d34fa58f041ab4 not found: ID does not exist" Oct 30 12:52:48.867786 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.867749 5922 scope.go:122] "RemoveContainer" containerID="733c9a46d88bf34e394c3f801c2a74a66a16cf1b882c2f5efbc9ca100a4714e6" Oct 30 12:52:48.868030 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.868006 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"733c9a46d88bf34e394c3f801c2a74a66a16cf1b882c2f5efbc9ca100a4714e6"} err="failed to get container status \"733c9a46d88bf34e394c3f801c2a74a66a16cf1b882c2f5efbc9ca100a4714e6\": rpc error: code = NotFound desc = could not find container \"733c9a46d88bf34e394c3f801c2a74a66a16cf1b882c2f5efbc9ca100a4714e6\": container with ID starting with 733c9a46d88bf34e394c3f801c2a74a66a16cf1b882c2f5efbc9ca100a4714e6 not found: ID does not exist" Oct 30 12:52:48.868030 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.868029 5922 scope.go:122] "RemoveContainer" containerID="3d067f86de45fb7da4d703f9d8ddfdf02e51e9db07b94ac0e02685e0a8f0e5f7" Oct 30 12:52:48.868235 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.868216 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d067f86de45fb7da4d703f9d8ddfdf02e51e9db07b94ac0e02685e0a8f0e5f7"} err="failed to get container status \"3d067f86de45fb7da4d703f9d8ddfdf02e51e9db07b94ac0e02685e0a8f0e5f7\": rpc error: code = NotFound desc = could not find container \"3d067f86de45fb7da4d703f9d8ddfdf02e51e9db07b94ac0e02685e0a8f0e5f7\": container with ID starting with 3d067f86de45fb7da4d703f9d8ddfdf02e51e9db07b94ac0e02685e0a8f0e5f7 not found: ID does not exist" Oct 30 12:52:48.868235 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.868235 5922 scope.go:122] "RemoveContainer" containerID="c626537d69a0eccf8091bdf1e44c648d000558fd2f939e9f7c4124f238c3219d" Oct 30 12:52:48.868448 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.868432 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c626537d69a0eccf8091bdf1e44c648d000558fd2f939e9f7c4124f238c3219d"} err="failed to get container status \"c626537d69a0eccf8091bdf1e44c648d000558fd2f939e9f7c4124f238c3219d\": rpc error: code = NotFound desc = could not find container \"c626537d69a0eccf8091bdf1e44c648d000558fd2f939e9f7c4124f238c3219d\": container with ID starting with c626537d69a0eccf8091bdf1e44c648d000558fd2f939e9f7c4124f238c3219d not found: ID does not exist" Oct 30 12:52:48.868494 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.868448 5922 scope.go:122] "RemoveContainer" containerID="966d7771b9f840d69179f59a1557d5269698ef1b542ddcf4e8791e225816c6d4" Oct 30 12:52:48.868670 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.868654 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"966d7771b9f840d69179f59a1557d5269698ef1b542ddcf4e8791e225816c6d4"} err="failed to get container status \"966d7771b9f840d69179f59a1557d5269698ef1b542ddcf4e8791e225816c6d4\": rpc error: code = NotFound desc = could not find container \"966d7771b9f840d69179f59a1557d5269698ef1b542ddcf4e8791e225816c6d4\": container with ID starting with 966d7771b9f840d69179f59a1557d5269698ef1b542ddcf4e8791e225816c6d4 not found: ID does not exist" Oct 30 12:52:48.868670 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.868669 5922 scope.go:122] "RemoveContainer" containerID="389ef54daca67f771705e3e74fe397815377121bb962ae7740b822516e1ce5a3" Oct 30 12:52:48.868845 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.868831 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"389ef54daca67f771705e3e74fe397815377121bb962ae7740b822516e1ce5a3"} err="failed to get container status \"389ef54daca67f771705e3e74fe397815377121bb962ae7740b822516e1ce5a3\": rpc error: code = NotFound desc = could not find container \"389ef54daca67f771705e3e74fe397815377121bb962ae7740b822516e1ce5a3\": container with ID starting with 389ef54daca67f771705e3e74fe397815377121bb962ae7740b822516e1ce5a3 not found: ID does not exist" Oct 30 12:52:48.868891 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.868845 5922 scope.go:122] "RemoveContainer" containerID="f758417ec1675d0ce680c01e967da15988b5cc0396604b5b43b45c7a87d11498" Oct 30 12:52:48.869051 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.869034 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f758417ec1675d0ce680c01e967da15988b5cc0396604b5b43b45c7a87d11498"} err="failed to get container status \"f758417ec1675d0ce680c01e967da15988b5cc0396604b5b43b45c7a87d11498\": rpc error: code = NotFound desc = could not find container \"f758417ec1675d0ce680c01e967da15988b5cc0396604b5b43b45c7a87d11498\": container with ID starting with f758417ec1675d0ce680c01e967da15988b5cc0396604b5b43b45c7a87d11498 not found: ID does not exist" Oct 30 12:52:48.869093 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.869051 5922 scope.go:122] "RemoveContainer" containerID="2bdb163e8f97e8e5afd18e9046bd2c976fa179cc65c409ed26d34fa58f041ab4" Oct 30 12:52:48.869225 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.869209 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bdb163e8f97e8e5afd18e9046bd2c976fa179cc65c409ed26d34fa58f041ab4"} err="failed to get container status \"2bdb163e8f97e8e5afd18e9046bd2c976fa179cc65c409ed26d34fa58f041ab4\": rpc error: code = NotFound desc = could not find container \"2bdb163e8f97e8e5afd18e9046bd2c976fa179cc65c409ed26d34fa58f041ab4\": container with ID starting with 2bdb163e8f97e8e5afd18e9046bd2c976fa179cc65c409ed26d34fa58f041ab4 not found: ID does not exist" Oct 30 12:52:48.869270 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.869225 5922 scope.go:122] "RemoveContainer" containerID="733c9a46d88bf34e394c3f801c2a74a66a16cf1b882c2f5efbc9ca100a4714e6" Oct 30 12:52:48.869404 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.869387 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"733c9a46d88bf34e394c3f801c2a74a66a16cf1b882c2f5efbc9ca100a4714e6"} err="failed to get container status \"733c9a46d88bf34e394c3f801c2a74a66a16cf1b882c2f5efbc9ca100a4714e6\": rpc error: code = NotFound desc = could not find container \"733c9a46d88bf34e394c3f801c2a74a66a16cf1b882c2f5efbc9ca100a4714e6\": container with ID starting with 733c9a46d88bf34e394c3f801c2a74a66a16cf1b882c2f5efbc9ca100a4714e6 not found: ID does not exist" Oct 30 12:52:48.869448 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.869404 5922 scope.go:122] "RemoveContainer" containerID="3d067f86de45fb7da4d703f9d8ddfdf02e51e9db07b94ac0e02685e0a8f0e5f7" Oct 30 12:52:48.869595 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.869581 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d067f86de45fb7da4d703f9d8ddfdf02e51e9db07b94ac0e02685e0a8f0e5f7"} err="failed to get container status \"3d067f86de45fb7da4d703f9d8ddfdf02e51e9db07b94ac0e02685e0a8f0e5f7\": rpc error: code = NotFound desc = could not find container \"3d067f86de45fb7da4d703f9d8ddfdf02e51e9db07b94ac0e02685e0a8f0e5f7\": container with ID starting with 3d067f86de45fb7da4d703f9d8ddfdf02e51e9db07b94ac0e02685e0a8f0e5f7 not found: ID does not exist" Oct 30 12:52:48.869661 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.869595 5922 scope.go:122] "RemoveContainer" containerID="c626537d69a0eccf8091bdf1e44c648d000558fd2f939e9f7c4124f238c3219d" Oct 30 12:52:48.869797 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.869782 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c626537d69a0eccf8091bdf1e44c648d000558fd2f939e9f7c4124f238c3219d"} err="failed to get container status \"c626537d69a0eccf8091bdf1e44c648d000558fd2f939e9f7c4124f238c3219d\": rpc error: code = NotFound desc = could not find container \"c626537d69a0eccf8091bdf1e44c648d000558fd2f939e9f7c4124f238c3219d\": container with ID starting with c626537d69a0eccf8091bdf1e44c648d000558fd2f939e9f7c4124f238c3219d not found: ID does not exist" Oct 30 12:52:48.869832 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.869797 5922 scope.go:122] "RemoveContainer" containerID="966d7771b9f840d69179f59a1557d5269698ef1b542ddcf4e8791e225816c6d4" Oct 30 12:52:48.869976 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.869961 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"966d7771b9f840d69179f59a1557d5269698ef1b542ddcf4e8791e225816c6d4"} err="failed to get container status \"966d7771b9f840d69179f59a1557d5269698ef1b542ddcf4e8791e225816c6d4\": rpc error: code = NotFound desc = could not find container \"966d7771b9f840d69179f59a1557d5269698ef1b542ddcf4e8791e225816c6d4\": container with ID starting with 966d7771b9f840d69179f59a1557d5269698ef1b542ddcf4e8791e225816c6d4 not found: ID does not exist" Oct 30 12:52:48.870021 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.869976 5922 scope.go:122] "RemoveContainer" containerID="389ef54daca67f771705e3e74fe397815377121bb962ae7740b822516e1ce5a3" Oct 30 12:52:48.870167 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.870151 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"389ef54daca67f771705e3e74fe397815377121bb962ae7740b822516e1ce5a3"} err="failed to get container status \"389ef54daca67f771705e3e74fe397815377121bb962ae7740b822516e1ce5a3\": rpc error: code = NotFound desc = could not find container \"389ef54daca67f771705e3e74fe397815377121bb962ae7740b822516e1ce5a3\": container with ID starting with 389ef54daca67f771705e3e74fe397815377121bb962ae7740b822516e1ce5a3 not found: ID does not exist" Oct 30 12:52:48.870206 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.870166 5922 scope.go:122] "RemoveContainer" containerID="f758417ec1675d0ce680c01e967da15988b5cc0396604b5b43b45c7a87d11498" Oct 30 12:52:48.870346 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.870331 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f758417ec1675d0ce680c01e967da15988b5cc0396604b5b43b45c7a87d11498"} err="failed to get container status \"f758417ec1675d0ce680c01e967da15988b5cc0396604b5b43b45c7a87d11498\": rpc error: code = NotFound desc = could not find container \"f758417ec1675d0ce680c01e967da15988b5cc0396604b5b43b45c7a87d11498\": container with ID starting with f758417ec1675d0ce680c01e967da15988b5cc0396604b5b43b45c7a87d11498 not found: ID does not exist" Oct 30 12:52:48.870385 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.870346 5922 scope.go:122] "RemoveContainer" containerID="2bdb163e8f97e8e5afd18e9046bd2c976fa179cc65c409ed26d34fa58f041ab4" Oct 30 12:52:48.870522 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.870506 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bdb163e8f97e8e5afd18e9046bd2c976fa179cc65c409ed26d34fa58f041ab4"} err="failed to get container status \"2bdb163e8f97e8e5afd18e9046bd2c976fa179cc65c409ed26d34fa58f041ab4\": rpc error: code = NotFound desc = could not find container \"2bdb163e8f97e8e5afd18e9046bd2c976fa179cc65c409ed26d34fa58f041ab4\": container with ID starting with 2bdb163e8f97e8e5afd18e9046bd2c976fa179cc65c409ed26d34fa58f041ab4 not found: ID does not exist" Oct 30 12:52:48.901094 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.901074 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 12:52:48.907919 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.907899 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 12:52:48.959548 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.959521 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 12:52:48.959832 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.959819 5922 cpu_manager.go:414] "containerMap: removing container" podUID="ecd3d4d5-a59a-4b0e-a981-956b10d08531" containerName="thanos-sidecar" Oct 30 12:52:48.959894 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.959835 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecd3d4d5-a59a-4b0e-a981-956b10d08531" containerName="thanos-sidecar" Oct 30 12:52:48.959894 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.959845 5922 cpu_manager.go:414] "containerMap: removing container" podUID="ecd3d4d5-a59a-4b0e-a981-956b10d08531" containerName="config-reloader" Oct 30 12:52:48.959894 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.959850 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecd3d4d5-a59a-4b0e-a981-956b10d08531" containerName="config-reloader" Oct 30 12:52:48.959894 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.959859 5922 cpu_manager.go:414] "containerMap: removing container" podUID="ecd3d4d5-a59a-4b0e-a981-956b10d08531" containerName="prometheus" Oct 30 12:52:48.959894 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.959864 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecd3d4d5-a59a-4b0e-a981-956b10d08531" containerName="prometheus" Oct 30 12:52:48.959894 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.959871 5922 cpu_manager.go:414] "containerMap: removing container" podUID="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" containerName="kube-rbac-proxy-metric" Oct 30 12:52:48.959894 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.959875 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" containerName="kube-rbac-proxy-metric" Oct 30 12:52:48.959894 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.959883 5922 cpu_manager.go:414] "containerMap: removing container" podUID="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" containerName="config-reloader" Oct 30 12:52:48.959894 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.959887 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" containerName="config-reloader" Oct 30 12:52:48.959894 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.959893 5922 cpu_manager.go:414] "containerMap: removing container" podUID="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" containerName="alertmanager" Oct 30 12:52:48.959894 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.959898 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" containerName="alertmanager" Oct 30 12:52:48.960210 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.959904 5922 cpu_manager.go:414] "containerMap: removing container" podUID="ecd3d4d5-a59a-4b0e-a981-956b10d08531" containerName="kube-rbac-proxy" Oct 30 12:52:48.960210 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.959909 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecd3d4d5-a59a-4b0e-a981-956b10d08531" containerName="kube-rbac-proxy" Oct 30 12:52:48.960210 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.959917 5922 cpu_manager.go:414] "containerMap: removing container" podUID="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" containerName="kube-rbac-proxy-web" Oct 30 12:52:48.960210 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.959921 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" containerName="kube-rbac-proxy-web" Oct 30 12:52:48.960210 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.959926 5922 cpu_manager.go:414] "containerMap: removing container" podUID="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" containerName="init-config-reloader" Oct 30 12:52:48.960210 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.959931 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" containerName="init-config-reloader" Oct 30 12:52:48.960210 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.959940 5922 cpu_manager.go:414] "containerMap: removing container" podUID="ecd3d4d5-a59a-4b0e-a981-956b10d08531" containerName="kube-rbac-proxy-thanos" Oct 30 12:52:48.960210 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.959945 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecd3d4d5-a59a-4b0e-a981-956b10d08531" containerName="kube-rbac-proxy-thanos" Oct 30 12:52:48.960210 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.959951 5922 cpu_manager.go:414] "containerMap: removing container" podUID="ecd3d4d5-a59a-4b0e-a981-956b10d08531" containerName="kube-rbac-proxy-web" Oct 30 12:52:48.960210 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.959955 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecd3d4d5-a59a-4b0e-a981-956b10d08531" containerName="kube-rbac-proxy-web" Oct 30 12:52:48.960210 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.959963 5922 cpu_manager.go:414] "containerMap: removing container" podUID="ecd3d4d5-a59a-4b0e-a981-956b10d08531" containerName="init-config-reloader" Oct 30 12:52:48.960210 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.959968 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecd3d4d5-a59a-4b0e-a981-956b10d08531" containerName="init-config-reloader" Oct 30 12:52:48.960210 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.959977 5922 cpu_manager.go:414] "containerMap: removing container" podUID="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" containerName="kube-rbac-proxy" Oct 30 12:52:48.960210 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.959981 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" containerName="kube-rbac-proxy" Oct 30 12:52:48.960210 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.959989 5922 cpu_manager.go:414] "containerMap: removing container" podUID="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" containerName="prom-label-proxy" Oct 30 12:52:48.960210 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.959993 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" containerName="prom-label-proxy" Oct 30 12:52:48.960210 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.960028 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecd3d4d5-a59a-4b0e-a981-956b10d08531" containerName="kube-rbac-proxy-thanos" Oct 30 12:52:48.960210 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.960034 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecd3d4d5-a59a-4b0e-a981-956b10d08531" containerName="kube-rbac-proxy-web" Oct 30 12:52:48.960210 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.960040 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecd3d4d5-a59a-4b0e-a981-956b10d08531" containerName="init-config-reloader" Oct 30 12:52:48.960210 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.960046 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" containerName="kube-rbac-proxy" Oct 30 12:52:48.960210 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.960053 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" containerName="prom-label-proxy" Oct 30 12:52:48.960210 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.960059 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecd3d4d5-a59a-4b0e-a981-956b10d08531" containerName="thanos-sidecar" Oct 30 12:52:48.960210 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.960064 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecd3d4d5-a59a-4b0e-a981-956b10d08531" containerName="config-reloader" Oct 30 12:52:48.960210 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.960069 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecd3d4d5-a59a-4b0e-a981-956b10d08531" containerName="prometheus" Oct 30 12:52:48.960210 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.960074 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" containerName="kube-rbac-proxy-metric" Oct 30 12:52:48.960210 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.960079 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" containerName="config-reloader" Oct 30 12:52:48.960210 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.960083 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" containerName="alertmanager" Oct 30 12:52:48.960210 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.960087 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecd3d4d5-a59a-4b0e-a981-956b10d08531" containerName="kube-rbac-proxy" Oct 30 12:52:48.960210 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.960092 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" containerName="kube-rbac-proxy-web" Oct 30 12:52:48.960210 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.960097 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="c18cb3fe-0392-4dca-b636-4e3ab99eb0f6" containerName="init-config-reloader" Oct 30 12:52:48.963854 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.963825 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:48.985277 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.985255 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 12:52:48.985277 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.985278 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 12:52:48.985428 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.985311 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 12:52:48.985428 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.985319 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 12:52:48.985428 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.985392 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5rv5r3ic93mpd\"" Oct 30 12:52:48.985551 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.985449 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5s6kd\"" Oct 30 12:52:48.995674 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.995652 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 12:52:48.996113 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.996073 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 12:52:48.996274 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.996258 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 12:52:48.996375 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:48.996273 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 12:52:49.008643 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.008597 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 12:52:49.008800 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.008785 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 12:52:49.063933 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.063915 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/83be9513-9c26-43a6-b2bc-485b5f0e386e-config\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.064018 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.063939 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/83be9513-9c26-43a6-b2bc-485b5f0e386e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.064018 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.063957 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/83be9513-9c26-43a6-b2bc-485b5f0e386e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.064018 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.063977 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/83be9513-9c26-43a6-b2bc-485b5f0e386e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.064130 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.064029 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/83be9513-9c26-43a6-b2bc-485b5f0e386e-web-config\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.064130 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.064066 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/83be9513-9c26-43a6-b2bc-485b5f0e386e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.064130 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.064098 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/83be9513-9c26-43a6-b2bc-485b5f0e386e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.064221 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.064142 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/83be9513-9c26-43a6-b2bc-485b5f0e386e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.064259 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.064226 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/83be9513-9c26-43a6-b2bc-485b5f0e386e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.064259 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.064253 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83be9513-9c26-43a6-b2bc-485b5f0e386e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.064317 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.064269 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83be9513-9c26-43a6-b2bc-485b5f0e386e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.064317 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.064285 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/83be9513-9c26-43a6-b2bc-485b5f0e386e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.064399 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.064383 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/83be9513-9c26-43a6-b2bc-485b5f0e386e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.064434 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.064426 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5qpc\" (UniqueName: \"kubernetes.io/projected/83be9513-9c26-43a6-b2bc-485b5f0e386e-kube-api-access-g5qpc\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.064468 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.064446 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83be9513-9c26-43a6-b2bc-485b5f0e386e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.064468 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.064462 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/83be9513-9c26-43a6-b2bc-485b5f0e386e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.064548 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.064482 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83be9513-9c26-43a6-b2bc-485b5f0e386e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.064605 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.064541 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/83be9513-9c26-43a6-b2bc-485b5f0e386e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.064605 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.064586 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/83be9513-9c26-43a6-b2bc-485b5f0e386e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.064700 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.064642 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/83be9513-9c26-43a6-b2bc-485b5f0e386e-config-out\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.080835 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.080812 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 12:52:49.080835 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.080833 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 12:52:49.080975 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.080851 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 12:52:49.080975 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.080891 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 12:52:49.081042 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.080979 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 12:52:49.165376 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.165357 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/83be9513-9c26-43a6-b2bc-485b5f0e386e-config\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.165457 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.165378 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/83be9513-9c26-43a6-b2bc-485b5f0e386e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.165457 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.165395 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/83be9513-9c26-43a6-b2bc-485b5f0e386e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.165457 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.165412 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/83be9513-9c26-43a6-b2bc-485b5f0e386e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.165457 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.165427 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/83be9513-9c26-43a6-b2bc-485b5f0e386e-web-config\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.165457 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.165445 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/83be9513-9c26-43a6-b2bc-485b5f0e386e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.165725 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.165654 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/83be9513-9c26-43a6-b2bc-485b5f0e386e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.165725 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.165690 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/83be9513-9c26-43a6-b2bc-485b5f0e386e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.165964 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.165730 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/83be9513-9c26-43a6-b2bc-485b5f0e386e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.165964 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.165775 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83be9513-9c26-43a6-b2bc-485b5f0e386e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.165964 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.165791 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/83be9513-9c26-43a6-b2bc-485b5f0e386e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.165964 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.165801 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83be9513-9c26-43a6-b2bc-485b5f0e386e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.166175 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.165966 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/83be9513-9c26-43a6-b2bc-485b5f0e386e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.166175 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.166029 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/83be9513-9c26-43a6-b2bc-485b5f0e386e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.166175 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.166054 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5qpc\" (UniqueName: \"kubernetes.io/projected/83be9513-9c26-43a6-b2bc-485b5f0e386e-kube-api-access-g5qpc\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.166175 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.166083 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83be9513-9c26-43a6-b2bc-485b5f0e386e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.166175 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.166112 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/83be9513-9c26-43a6-b2bc-485b5f0e386e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.166175 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.166138 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83be9513-9c26-43a6-b2bc-485b5f0e386e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.166175 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.166164 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/83be9513-9c26-43a6-b2bc-485b5f0e386e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.166523 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.166193 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/83be9513-9c26-43a6-b2bc-485b5f0e386e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.166523 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.166228 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/83be9513-9c26-43a6-b2bc-485b5f0e386e-config-out\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.166523 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.166372 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/83be9513-9c26-43a6-b2bc-485b5f0e386e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.169353 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.167339 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83be9513-9c26-43a6-b2bc-485b5f0e386e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.169353 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.167737 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83be9513-9c26-43a6-b2bc-485b5f0e386e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.169353 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.168638 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/83be9513-9c26-43a6-b2bc-485b5f0e386e-config-out\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.169353 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.168650 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83be9513-9c26-43a6-b2bc-485b5f0e386e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.169353 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.168687 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/83be9513-9c26-43a6-b2bc-485b5f0e386e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.169353 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.168732 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/83be9513-9c26-43a6-b2bc-485b5f0e386e-config\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.169353 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.168986 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/83be9513-9c26-43a6-b2bc-485b5f0e386e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.169353 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.169076 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/83be9513-9c26-43a6-b2bc-485b5f0e386e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.169731 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.169532 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/83be9513-9c26-43a6-b2bc-485b5f0e386e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.169731 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.169549 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/83be9513-9c26-43a6-b2bc-485b5f0e386e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.169731 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.169653 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83be9513-9c26-43a6-b2bc-485b5f0e386e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.169840 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.169822 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/83be9513-9c26-43a6-b2bc-485b5f0e386e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.170190 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.170157 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/83be9513-9c26-43a6-b2bc-485b5f0e386e-web-config\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.170304 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.170286 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/83be9513-9c26-43a6-b2bc-485b5f0e386e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.171159 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.171140 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/83be9513-9c26-43a6-b2bc-485b5f0e386e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.171645 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.171628 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/83be9513-9c26-43a6-b2bc-485b5f0e386e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.172108 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.172091 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/83be9513-9c26-43a6-b2bc-485b5f0e386e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.192138 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.192113 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5qpc\" (UniqueName: \"kubernetes.io/projected/83be9513-9c26-43a6-b2bc-485b5f0e386e-kube-api-access-g5qpc\") pod \"prometheus-k8s-0\" (UID: \"83be9513-9c26-43a6-b2bc-485b5f0e386e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.280270 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.280221 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:49.610781 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.610720 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 12:52:49.614815 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:52:49.614789 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83be9513_9c26_43a6_b2bc_485b5f0e386e.slice/crio-b7622abf63882624e696d2776e62ed4438c0d9830f6afb3850593a844a02724e WatchSource:0}: Error finding container b7622abf63882624e696d2776e62ed4438c0d9830f6afb3850593a844a02724e: Status 404 returned error can't find the container with id b7622abf63882624e696d2776e62ed4438c0d9830f6afb3850593a844a02724e Oct 30 12:52:49.641424 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.641402 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecd3d4d5-a59a-4b0e-a981-956b10d08531" path="/var/lib/kubelet/pods/ecd3d4d5-a59a-4b0e-a981-956b10d08531/volumes" Oct 30 12:52:49.825933 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.825905 5922 generic.go:356] "Generic (PLEG): container finished" podID="83be9513-9c26-43a6-b2bc-485b5f0e386e" containerID="54d6ea052f94ba6f0e4718ffb02a936388b79302a9dc82f04cb49c59a8a0642c" exitCode=0 Oct 30 12:52:49.826049 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.825939 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83be9513-9c26-43a6-b2bc-485b5f0e386e","Type":"ContainerDied","Data":"54d6ea052f94ba6f0e4718ffb02a936388b79302a9dc82f04cb49c59a8a0642c"} Oct 30 12:52:49.826049 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:49.826009 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83be9513-9c26-43a6-b2bc-485b5f0e386e","Type":"ContainerStarted","Data":"b7622abf63882624e696d2776e62ed4438c0d9830f6afb3850593a844a02724e"} Oct 30 12:52:50.708471 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:50.708446 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74d99cdf4-q476v" Oct 30 12:52:50.708927 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:50.708499 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74d99cdf4-q476v" Oct 30 12:52:50.712809 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:50.712791 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74d99cdf4-q476v" Oct 30 12:52:50.731483 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:50.731454 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-85cbd9968c-p6js7" podUID="f89a7014-2966-43c7-a901-709c690b0003" containerName="console" containerID="cri-o://4062791b86ea9ea99e6ffa00a7b797c6dfc7855c0fbcf5ab34e6ca6dec169367" gracePeriod=15 Oct 30 12:52:50.832392 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:50.832367 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83be9513-9c26-43a6-b2bc-485b5f0e386e","Type":"ContainerStarted","Data":"74c1625e04b96462ff25c7cd07d7807f46c223f164ed781e7df713a69fe30e4d"} Oct 30 12:52:50.832518 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:50.832449 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83be9513-9c26-43a6-b2bc-485b5f0e386e","Type":"ContainerStarted","Data":"567fd53b5ad7fb5d487995aa481ebbd2e3abaa14ba03b9f22753fb067cd13d55"} Oct 30 12:52:50.832518 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:50.832497 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83be9513-9c26-43a6-b2bc-485b5f0e386e","Type":"ContainerStarted","Data":"b648a9b7285e0d44247ca5c2fbdfaa5bcefab664ec3dcd7343f2570a0ad741e5"} Oct 30 12:52:50.832645 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:50.832535 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83be9513-9c26-43a6-b2bc-485b5f0e386e","Type":"ContainerStarted","Data":"990af9e45308938c83e7d81b53b614b81a89902f5638157c122fd9f856cbf3fb"} Oct 30 12:52:50.832645 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:50.832575 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83be9513-9c26-43a6-b2bc-485b5f0e386e","Type":"ContainerStarted","Data":"216a4c10cfd6f7140d98227dcdce262adbebf155763d38ed3e48d25e27932731"} Oct 30 12:52:50.832645 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:50.832627 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83be9513-9c26-43a6-b2bc-485b5f0e386e","Type":"ContainerStarted","Data":"0c996b8daa2977d4b992e9d9f6b5f0c127263fee9f81e8351349b8ecf5320c5d"} Oct 30 12:52:50.836143 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:50.836125 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74d99cdf4-q476v" Oct 30 12:52:50.890701 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:50.890656 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.8762142539999997 podStartE2EDuration="2.89064172s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.427466ms" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:50.889758947 +0000 UTC m=+27.359174720" watchObservedRunningTime="2026-10-30 12:52:50.89064172 +0000 UTC m=+27.360057444" Oct 30 12:52:50.964337 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:50.964308 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-689bbd6cb4-lmgqn"] Oct 30 12:52:50.983178 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:50.983154 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85cbd9968c-p6js7" Oct 30 12:52:51.083688 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:51.083659 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f89a7014-2966-43c7-a901-709c690b0003-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f89a7014-2966-43c7-a901-709c690b0003-oauth-serving-cert\") pod \"f89a7014-2966-43c7-a901-709c690b0003\" (UID: \"f89a7014-2966-43c7-a901-709c690b0003\") " Oct 30 12:52:51.083827 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:51.083698 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f89a7014-2966-43c7-a901-709c690b0003-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f89a7014-2966-43c7-a901-709c690b0003-tmp\") pod \"f89a7014-2966-43c7-a901-709c690b0003\" (UID: \"f89a7014-2966-43c7-a901-709c690b0003\") " Oct 30 12:52:51.083827 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:51.083761 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f89a7014-2966-43c7-a901-709c690b0003-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f89a7014-2966-43c7-a901-709c690b0003-console-oauth-config\") pod \"f89a7014-2966-43c7-a901-709c690b0003\" (UID: \"f89a7014-2966-43c7-a901-709c690b0003\") " Oct 30 12:52:51.083827 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:51.083794 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f89a7014-2966-43c7-a901-709c690b0003-service-ca\" (UniqueName: \"kubernetes.io/configmap/f89a7014-2966-43c7-a901-709c690b0003-service-ca\") pod \"f89a7014-2966-43c7-a901-709c690b0003\" (UID: \"f89a7014-2966-43c7-a901-709c690b0003\") " Oct 30 12:52:51.083970 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:51.083858 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f89a7014-2966-43c7-a901-709c690b0003-console-config\" (UniqueName: \"kubernetes.io/configmap/f89a7014-2966-43c7-a901-709c690b0003-console-config\") pod \"f89a7014-2966-43c7-a901-709c690b0003\" (UID: \"f89a7014-2966-43c7-a901-709c690b0003\") " Oct 30 12:52:51.083970 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:51.083890 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f89a7014-2966-43c7-a901-709c690b0003-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f89a7014-2966-43c7-a901-709c690b0003-console-serving-cert\") pod \"f89a7014-2966-43c7-a901-709c690b0003\" (UID: \"f89a7014-2966-43c7-a901-709c690b0003\") " Oct 30 12:52:51.083970 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:51.083924 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f89a7014-2966-43c7-a901-709c690b0003-kube-api-access-ssl6b\" (UniqueName: \"kubernetes.io/projected/f89a7014-2966-43c7-a901-709c690b0003-kube-api-access-ssl6b\") pod \"f89a7014-2966-43c7-a901-709c690b0003\" (UID: \"f89a7014-2966-43c7-a901-709c690b0003\") " Oct 30 12:52:51.084115 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:51.084010 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f89a7014-2966-43c7-a901-709c690b0003-tmp" pod "f89a7014-2966-43c7-a901-709c690b0003" (UID: "f89a7014-2966-43c7-a901-709c690b0003"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:52:51.084179 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:51.084148 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f89a7014-2966-43c7-a901-709c690b0003-oauth-serving-cert" pod "f89a7014-2966-43c7-a901-709c690b0003" (UID: "f89a7014-2966-43c7-a901-709c690b0003"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:52:51.084230 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:51.084187 5922 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f89a7014-2966-43c7-a901-709c690b0003-tmp\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:52:51.084369 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:51.084344 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f89a7014-2966-43c7-a901-709c690b0003-console-config" pod "f89a7014-2966-43c7-a901-709c690b0003" (UID: "f89a7014-2966-43c7-a901-709c690b0003"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:52:51.084525 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:51.084497 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f89a7014-2966-43c7-a901-709c690b0003-service-ca" pod "f89a7014-2966-43c7-a901-709c690b0003" (UID: "f89a7014-2966-43c7-a901-709c690b0003"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:52:51.086128 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:51.086108 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f89a7014-2966-43c7-a901-709c690b0003-console-serving-cert" pod "f89a7014-2966-43c7-a901-709c690b0003" (UID: "f89a7014-2966-43c7-a901-709c690b0003"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:52:51.086334 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:51.086315 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f89a7014-2966-43c7-a901-709c690b0003-console-oauth-config" pod "f89a7014-2966-43c7-a901-709c690b0003" (UID: "f89a7014-2966-43c7-a901-709c690b0003"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:52:51.086438 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:51.086419 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f89a7014-2966-43c7-a901-709c690b0003-kube-api-access-ssl6b" pod "f89a7014-2966-43c7-a901-709c690b0003" (UID: "f89a7014-2966-43c7-a901-709c690b0003"). InnerVolumeSpecName "kube-api-access-ssl6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:52:51.184536 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:51.184513 5922 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f89a7014-2966-43c7-a901-709c690b0003-console-oauth-config\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:52:51.184536 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:51.184534 5922 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f89a7014-2966-43c7-a901-709c690b0003-service-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:52:51.184674 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:51.184545 5922 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f89a7014-2966-43c7-a901-709c690b0003-console-config\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:52:51.184674 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:51.184553 5922 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f89a7014-2966-43c7-a901-709c690b0003-console-serving-cert\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:52:51.184674 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:51.184561 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssl6b\" (UniqueName: \"kubernetes.io/projected/f89a7014-2966-43c7-a901-709c690b0003-kube-api-access-ssl6b\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:52:51.184674 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:51.184570 5922 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f89a7014-2966-43c7-a901-709c690b0003-oauth-serving-cert\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:52:51.836827 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:51.836797 5922 generic.go:356] "Generic (PLEG): container finished" podID="f89a7014-2966-43c7-a901-709c690b0003" containerID="4062791b86ea9ea99e6ffa00a7b797c6dfc7855c0fbcf5ab34e6ca6dec169367" exitCode=0 Oct 30 12:52:51.837201 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:51.836854 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85cbd9968c-p6js7" Oct 30 12:52:51.837201 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:51.836886 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85cbd9968c-p6js7" event={"ID":"f89a7014-2966-43c7-a901-709c690b0003","Type":"ContainerDied","Data":"4062791b86ea9ea99e6ffa00a7b797c6dfc7855c0fbcf5ab34e6ca6dec169367"} Oct 30 12:52:51.837201 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:51.836937 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85cbd9968c-p6js7" event={"ID":"f89a7014-2966-43c7-a901-709c690b0003","Type":"ContainerDied","Data":"815dcea6522fbada33cae727dbd21ebe8040c2dced78a4d7324e4652362ad187"} Oct 30 12:52:51.837201 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:51.836960 5922 scope.go:122] "RemoveContainer" containerID="4062791b86ea9ea99e6ffa00a7b797c6dfc7855c0fbcf5ab34e6ca6dec169367" Oct 30 12:52:51.845109 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:51.845088 5922 scope.go:122] "RemoveContainer" containerID="4062791b86ea9ea99e6ffa00a7b797c6dfc7855c0fbcf5ab34e6ca6dec169367" Oct 30 12:52:51.845372 ip-10-0-137-21 kubenswrapper[5922]: E1030 12:52:51.845354 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4062791b86ea9ea99e6ffa00a7b797c6dfc7855c0fbcf5ab34e6ca6dec169367\": container with ID starting with 4062791b86ea9ea99e6ffa00a7b797c6dfc7855c0fbcf5ab34e6ca6dec169367 not found: ID does not exist" containerID="4062791b86ea9ea99e6ffa00a7b797c6dfc7855c0fbcf5ab34e6ca6dec169367" Oct 30 12:52:51.845424 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:51.845380 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4062791b86ea9ea99e6ffa00a7b797c6dfc7855c0fbcf5ab34e6ca6dec169367"} err="failed to get container status \"4062791b86ea9ea99e6ffa00a7b797c6dfc7855c0fbcf5ab34e6ca6dec169367\": rpc error: code = NotFound desc = could not find container \"4062791b86ea9ea99e6ffa00a7b797c6dfc7855c0fbcf5ab34e6ca6dec169367\": container with ID starting with 4062791b86ea9ea99e6ffa00a7b797c6dfc7855c0fbcf5ab34e6ca6dec169367 not found: ID does not exist" Oct 30 12:52:51.888745 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:51.888722 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85cbd9968c-p6js7"] Oct 30 12:52:51.900545 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:51.900526 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-85cbd9968c-p6js7"] Oct 30 12:52:51.918649 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:51.918592 5922 patch_prober.go:29] interesting pod/console-85cbd9968c-p6js7 container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.134.0.12:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:52:51.918733 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:51.918648 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/console-85cbd9968c-p6js7" podUID="f89a7014-2966-43c7-a901-709c690b0003" containerName="console" probeResult="failure" output="Get \"https://10.134.0.12:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:52:52.546287 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:52.546223 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:52.661466 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:52.661439 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:53.550578 ip-10-0-137-21 kubenswrapper[5922]: E1030 12:52:53.550546 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 12:52:53.641505 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:53.641478 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f89a7014-2966-43c7-a901-709c690b0003" path="/var/lib/kubelet/pods/f89a7014-2966-43c7-a901-709c690b0003/volumes" Oct 30 12:52:54.280717 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:54.280691 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:54.905829 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:54.905801 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nwrx6" Oct 30 12:52:58.001191 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:52:58.001162 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-95n9l" Oct 30 12:53:11.869690 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:53:11.869656 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:15.986253 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:53:15.986196 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-689bbd6cb4-lmgqn" podUID="01873344-73c8-47b2-b6ba-8e338ba18315" containerName="console" containerID="cri-o://5a9387eb948f972dc6d4865e6a5a4f799b675aefee291ec250feccd574daea53" gracePeriod=15 Oct 30 12:53:16.232352 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:53:16.232329 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:53:16.273048 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:53:16.272994 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01873344-73c8-47b2-b6ba-8e338ba18315-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01873344-73c8-47b2-b6ba-8e338ba18315-oauth-serving-cert\") pod \"01873344-73c8-47b2-b6ba-8e338ba18315\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " Oct 30 12:53:16.273048 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:53:16.273028 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01873344-73c8-47b2-b6ba-8e338ba18315-service-ca\" (UniqueName: \"kubernetes.io/configmap/01873344-73c8-47b2-b6ba-8e338ba18315-service-ca\") pod \"01873344-73c8-47b2-b6ba-8e338ba18315\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " Oct 30 12:53:16.273219 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:53:16.273056 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01873344-73c8-47b2-b6ba-8e338ba18315-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01873344-73c8-47b2-b6ba-8e338ba18315-trusted-ca-bundle\") pod \"01873344-73c8-47b2-b6ba-8e338ba18315\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " Oct 30 12:53:16.273219 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:53:16.273077 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01873344-73c8-47b2-b6ba-8e338ba18315-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01873344-73c8-47b2-b6ba-8e338ba18315-console-serving-cert\") pod \"01873344-73c8-47b2-b6ba-8e338ba18315\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " Oct 30 12:53:16.273219 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:53:16.273145 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01873344-73c8-47b2-b6ba-8e338ba18315-kube-api-access-2w592\" (UniqueName: \"kubernetes.io/projected/01873344-73c8-47b2-b6ba-8e338ba18315-kube-api-access-2w592\") pod \"01873344-73c8-47b2-b6ba-8e338ba18315\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " Oct 30 12:53:16.273219 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:53:16.273183 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01873344-73c8-47b2-b6ba-8e338ba18315-tmp\" (UniqueName: \"kubernetes.io/empty-dir/01873344-73c8-47b2-b6ba-8e338ba18315-tmp\") pod \"01873344-73c8-47b2-b6ba-8e338ba18315\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " Oct 30 12:53:16.273417 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:53:16.273234 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01873344-73c8-47b2-b6ba-8e338ba18315-console-config\" (UniqueName: \"kubernetes.io/configmap/01873344-73c8-47b2-b6ba-8e338ba18315-console-config\") pod \"01873344-73c8-47b2-b6ba-8e338ba18315\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " Oct 30 12:53:16.273417 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:53:16.273281 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01873344-73c8-47b2-b6ba-8e338ba18315-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01873344-73c8-47b2-b6ba-8e338ba18315-console-oauth-config\") pod \"01873344-73c8-47b2-b6ba-8e338ba18315\" (UID: \"01873344-73c8-47b2-b6ba-8e338ba18315\") " Oct 30 12:53:16.273417 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:53:16.273382 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01873344-73c8-47b2-b6ba-8e338ba18315-service-ca" pod "01873344-73c8-47b2-b6ba-8e338ba18315" (UID: "01873344-73c8-47b2-b6ba-8e338ba18315"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:53:16.273566 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:53:16.273483 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01873344-73c8-47b2-b6ba-8e338ba18315-tmp" pod "01873344-73c8-47b2-b6ba-8e338ba18315" (UID: "01873344-73c8-47b2-b6ba-8e338ba18315"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:53:16.273566 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:53:16.273490 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01873344-73c8-47b2-b6ba-8e338ba18315-trusted-ca-bundle" pod "01873344-73c8-47b2-b6ba-8e338ba18315" (UID: "01873344-73c8-47b2-b6ba-8e338ba18315"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:53:16.273713 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:53:16.273586 5922 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01873344-73c8-47b2-b6ba-8e338ba18315-tmp\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:53:16.273713 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:53:16.273600 5922 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01873344-73c8-47b2-b6ba-8e338ba18315-service-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:53:16.273713 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:53:16.273649 5922 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01873344-73c8-47b2-b6ba-8e338ba18315-trusted-ca-bundle\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:53:16.273859 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:53:16.273811 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01873344-73c8-47b2-b6ba-8e338ba18315-oauth-serving-cert" pod "01873344-73c8-47b2-b6ba-8e338ba18315" (UID: "01873344-73c8-47b2-b6ba-8e338ba18315"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:53:16.273997 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:53:16.273972 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01873344-73c8-47b2-b6ba-8e338ba18315-console-config" pod "01873344-73c8-47b2-b6ba-8e338ba18315" (UID: "01873344-73c8-47b2-b6ba-8e338ba18315"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:53:16.275223 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:53:16.275200 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01873344-73c8-47b2-b6ba-8e338ba18315-console-serving-cert" pod "01873344-73c8-47b2-b6ba-8e338ba18315" (UID: "01873344-73c8-47b2-b6ba-8e338ba18315"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:53:16.275350 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:53:16.275330 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01873344-73c8-47b2-b6ba-8e338ba18315-kube-api-access-2w592" pod "01873344-73c8-47b2-b6ba-8e338ba18315" (UID: "01873344-73c8-47b2-b6ba-8e338ba18315"). InnerVolumeSpecName "kube-api-access-2w592". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:53:16.275390 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:53:16.275355 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01873344-73c8-47b2-b6ba-8e338ba18315-console-oauth-config" pod "01873344-73c8-47b2-b6ba-8e338ba18315" (UID: "01873344-73c8-47b2-b6ba-8e338ba18315"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:53:16.374235 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:53:16.374212 5922 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01873344-73c8-47b2-b6ba-8e338ba18315-console-config\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:53:16.374235 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:53:16.374235 5922 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01873344-73c8-47b2-b6ba-8e338ba18315-console-oauth-config\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:53:16.374392 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:53:16.374244 5922 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01873344-73c8-47b2-b6ba-8e338ba18315-oauth-serving-cert\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:53:16.374392 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:53:16.374253 5922 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01873344-73c8-47b2-b6ba-8e338ba18315-console-serving-cert\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:53:16.374392 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:53:16.374262 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w592\" (UniqueName: \"kubernetes.io/projected/01873344-73c8-47b2-b6ba-8e338ba18315-kube-api-access-2w592\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:53:16.929069 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:53:16.929034 5922 generic.go:356] "Generic (PLEG): container finished" podID="01873344-73c8-47b2-b6ba-8e338ba18315" containerID="5a9387eb948f972dc6d4865e6a5a4f799b675aefee291ec250feccd574daea53" exitCode=0 Oct 30 12:53:16.929225 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:53:16.929080 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-689bbd6cb4-lmgqn" Oct 30 12:53:16.929225 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:53:16.929125 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-689bbd6cb4-lmgqn" event={"ID":"01873344-73c8-47b2-b6ba-8e338ba18315","Type":"ContainerDied","Data":"5a9387eb948f972dc6d4865e6a5a4f799b675aefee291ec250feccd574daea53"} Oct 30 12:53:16.929225 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:53:16.929170 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-689bbd6cb4-lmgqn" event={"ID":"01873344-73c8-47b2-b6ba-8e338ba18315","Type":"ContainerDied","Data":"0647e9ff213d9818f0c5cc4f86bd064d5d41a6318831f00fb5d98ca6c711629e"} Oct 30 12:53:16.929225 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:53:16.929201 5922 scope.go:122] "RemoveContainer" containerID="5a9387eb948f972dc6d4865e6a5a4f799b675aefee291ec250feccd574daea53" Oct 30 12:53:16.937849 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:53:16.937834 5922 scope.go:122] "RemoveContainer" containerID="5a9387eb948f972dc6d4865e6a5a4f799b675aefee291ec250feccd574daea53" Oct 30 12:53:16.938104 ip-10-0-137-21 kubenswrapper[5922]: E1030 12:53:16.938086 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a9387eb948f972dc6d4865e6a5a4f799b675aefee291ec250feccd574daea53\": container with ID starting with 5a9387eb948f972dc6d4865e6a5a4f799b675aefee291ec250feccd574daea53 not found: ID does not exist" containerID="5a9387eb948f972dc6d4865e6a5a4f799b675aefee291ec250feccd574daea53" Oct 30 12:53:16.938170 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:53:16.938111 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a9387eb948f972dc6d4865e6a5a4f799b675aefee291ec250feccd574daea53"} err="failed to get container status \"5a9387eb948f972dc6d4865e6a5a4f799b675aefee291ec250feccd574daea53\": rpc error: code = NotFound desc = could not find container \"5a9387eb948f972dc6d4865e6a5a4f799b675aefee291ec250feccd574daea53\": container with ID starting with 5a9387eb948f972dc6d4865e6a5a4f799b675aefee291ec250feccd574daea53 not found: ID does not exist" Oct 30 12:53:16.980364 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:53:16.980334 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-689bbd6cb4-lmgqn"] Oct 30 12:53:16.992332 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:53:16.992312 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-689bbd6cb4-lmgqn"] Oct 30 12:53:17.643187 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:53:17.643147 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01873344-73c8-47b2-b6ba-8e338ba18315" path="/var/lib/kubelet/pods/01873344-73c8-47b2-b6ba-8e338ba18315/volumes" Oct 30 12:53:23.551262 ip-10-0-137-21 kubenswrapper[5922]: E1030 12:53:23.551230 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 12:53:24.188757 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:53:24.188719 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:24.306930 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:53:24.306902 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:30.858883 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:53:30.858845 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:36.683115 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:53:36.683084 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:53:36.698115 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:53:36.698094 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:53:37.009227 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:53:37.009204 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:53:53.551179 ip-10-0-137-21 kubenswrapper[5922]: E1030 12:53:53.551143 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 12:54:18.245900 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:54:18.245818 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:18.372961 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:54:18.372929 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:23.550993 ip-10-0-137-21 kubenswrapper[5922]: E1030 12:54:23.550955 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 12:54:24.624837 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:54:24.624803 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:30.978929 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:54:30.978894 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:42.221187 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:54:42.221149 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:42.340029 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:54:42.339999 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:51.271547 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:54:51.271509 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:53.551646 ip-10-0-137-21 kubenswrapper[5922]: E1030 12:54:53.551598 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 12:55:04.441330 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:04.441290 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:05.252760 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:05.252724 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:07.862090 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:07.862055 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:07.978359 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:07.978326 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:10.353009 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:10.352969 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:10.491096 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:10.491057 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:10.744335 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:10.744294 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:10.905855 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:10.905826 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:12.807938 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:12.807906 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:12.973009 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:12.972972 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:14.651247 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:14.651205 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:14.786119 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:14.786084 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:16.068272 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:16.068237 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:16.185382 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:16.185351 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:18.968023 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:18.967985 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:19.095335 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:19.095303 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:20.642373 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:20.642342 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:20.829042 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:20.829009 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-tt7l4"] Oct 30 12:55:20.829373 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:20.829355 5922 cpu_manager.go:414] "containerMap: removing container" podUID="f89a7014-2966-43c7-a901-709c690b0003" containerName="console" Oct 30 12:55:20.829451 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:20.829375 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f89a7014-2966-43c7-a901-709c690b0003" containerName="console" Oct 30 12:55:20.829451 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:20.829389 5922 cpu_manager.go:414] "containerMap: removing container" podUID="01873344-73c8-47b2-b6ba-8e338ba18315" containerName="console" Oct 30 12:55:20.829451 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:20.829396 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01873344-73c8-47b2-b6ba-8e338ba18315" containerName="console" Oct 30 12:55:20.829593 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:20.829489 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="f89a7014-2966-43c7-a901-709c690b0003" containerName="console" Oct 30 12:55:20.829593 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:20.829503 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="01873344-73c8-47b2-b6ba-8e338ba18315" containerName="console" Oct 30 12:55:20.832332 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:20.832306 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-tt7l4" Oct 30 12:55:20.859073 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:20.855208 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-tt7l4"] Oct 30 12:55:20.859554 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:20.859523 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 12:55:20.859694 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:20.859584 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 12:55:20.859694 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:20.859662 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-fcts5\"" Oct 30 12:55:20.964920 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:20.964897 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/92c3ace8-97df-424a-a247-c72bc7324994-bound-sa-token\") pod \"cert-manager-597b95cb84-tt7l4\" (UID: \"92c3ace8-97df-424a-a247-c72bc7324994\") " pod="cert-manager/cert-manager-597b95cb84-tt7l4" Oct 30 12:55:20.965032 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:20.964962 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkr5b\" (UniqueName: \"kubernetes.io/projected/92c3ace8-97df-424a-a247-c72bc7324994-kube-api-access-xkr5b\") pod \"cert-manager-597b95cb84-tt7l4\" (UID: \"92c3ace8-97df-424a-a247-c72bc7324994\") " pod="cert-manager/cert-manager-597b95cb84-tt7l4" Oct 30 12:55:21.066056 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:21.066032 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/92c3ace8-97df-424a-a247-c72bc7324994-bound-sa-token\") pod \"cert-manager-597b95cb84-tt7l4\" (UID: \"92c3ace8-97df-424a-a247-c72bc7324994\") " pod="cert-manager/cert-manager-597b95cb84-tt7l4" Oct 30 12:55:21.066188 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:21.066120 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkr5b\" (UniqueName: \"kubernetes.io/projected/92c3ace8-97df-424a-a247-c72bc7324994-kube-api-access-xkr5b\") pod \"cert-manager-597b95cb84-tt7l4\" (UID: \"92c3ace8-97df-424a-a247-c72bc7324994\") " pod="cert-manager/cert-manager-597b95cb84-tt7l4" Oct 30 12:55:21.090196 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:21.090177 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/92c3ace8-97df-424a-a247-c72bc7324994-bound-sa-token\") pod \"cert-manager-597b95cb84-tt7l4\" (UID: \"92c3ace8-97df-424a-a247-c72bc7324994\") " pod="cert-manager/cert-manager-597b95cb84-tt7l4" Oct 30 12:55:21.090371 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:21.090354 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkr5b\" (UniqueName: \"kubernetes.io/projected/92c3ace8-97df-424a-a247-c72bc7324994-kube-api-access-xkr5b\") pod \"cert-manager-597b95cb84-tt7l4\" (UID: \"92c3ace8-97df-424a-a247-c72bc7324994\") " pod="cert-manager/cert-manager-597b95cb84-tt7l4" Oct 30 12:55:21.158320 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:21.158299 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-tt7l4" Oct 30 12:55:21.449862 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:21.449829 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-tt7l4"] Oct 30 12:55:21.454197 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:55:21.454168 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92c3ace8_97df_424a_a247_c72bc7324994.slice/crio-4a08e49168bcabaa2bffe5ba57fb55d433412f89d32cb800a3d9005b1c138e9c WatchSource:0}: Error finding container 4a08e49168bcabaa2bffe5ba57fb55d433412f89d32cb800a3d9005b1c138e9c: Status 404 returned error can't find the container with id 4a08e49168bcabaa2bffe5ba57fb55d433412f89d32cb800a3d9005b1c138e9c Oct 30 12:55:21.456137 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:21.456120 5922 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 12:55:22.336874 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:22.336829 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-tt7l4" event={"ID":"92c3ace8-97df-424a-a247-c72bc7324994","Type":"ContainerStarted","Data":"4a08e49168bcabaa2bffe5ba57fb55d433412f89d32cb800a3d9005b1c138e9c"} Oct 30 12:55:22.803080 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:22.803044 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:22.915286 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:22.915213 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:23.554148 ip-10-0-137-21 kubenswrapper[5922]: E1030 12:55:23.554111 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 12:55:24.345822 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:24.345788 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-tt7l4" event={"ID":"92c3ace8-97df-424a-a247-c72bc7324994","Type":"ContainerStarted","Data":"62bbe8df45671354c01f23c246e934f82cb72f4d0d9420463497942a2e90592e"} Oct 30 12:55:24.386766 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:24.386715 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-tt7l4" podStartSLOduration=1.615239698 podStartE2EDuration="4.386698701s" totalImagesPullingTime="2.771459003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:55:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:24.386306087 +0000 UTC m=+180.855721832" watchObservedRunningTime="2026-10-30 12:55:24.386698701 +0000 UTC m=+180.856114447" Oct 30 12:55:26.277860 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:26.277819 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:26.398117 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:26.398087 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:31.101196 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:31.101165 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:32.419321 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:32.419287 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:32.535784 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:32.535755 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:34.263130 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:34.263084 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:44.398757 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:44.398657 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:44.516992 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:44.516961 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:45.065023 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:45.064988 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:45.329064 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:45.328974 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:45.450670 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:45.450637 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:45.696504 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:45.696467 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:45.853370 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:45.853338 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:45.939140 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:45.939107 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn"] Oct 30 12:55:45.943526 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:45.943505 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn" Oct 30 12:55:45.978920 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:45.978898 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn"] Oct 30 12:55:45.987772 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:45.987749 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 12:55:45.987889 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:45.987796 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 12:55:45.987889 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:45.987798 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 12:55:45.987889 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:45.987801 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 12:55:45.988043 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:45.988009 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-frp85\"" Oct 30 12:55:45.988084 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:45.988060 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\"" Oct 30 12:55:46.049374 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:46.049350 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/40764e2d-8e5e-427b-adb5-e3c0bf47e6c8-manifests\") pod \"opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn\" (UID: \"40764e2d-8e5e-427b-adb5-e3c0bf47e6c8\") " pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn" Oct 30 12:55:46.049465 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:46.049399 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwqjb\" (UniqueName: \"kubernetes.io/projected/40764e2d-8e5e-427b-adb5-e3c0bf47e6c8-kube-api-access-vwqjb\") pod \"opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn\" (UID: \"40764e2d-8e5e-427b-adb5-e3c0bf47e6c8\") " pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn" Oct 30 12:55:46.049508 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:46.049483 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40764e2d-8e5e-427b-adb5-e3c0bf47e6c8-tmp\") pod \"opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn\" (UID: \"40764e2d-8e5e-427b-adb5-e3c0bf47e6c8\") " pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn" Oct 30 12:55:46.049542 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:46.049511 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/40764e2d-8e5e-427b-adb5-e3c0bf47e6c8-metrics-cert\") pod \"opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn\" (UID: \"40764e2d-8e5e-427b-adb5-e3c0bf47e6c8\") " pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn" Oct 30 12:55:46.049605 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:46.049587 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40764e2d-8e5e-427b-adb5-e3c0bf47e6c8-webhook-cert\") pod \"opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn\" (UID: \"40764e2d-8e5e-427b-adb5-e3c0bf47e6c8\") " pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn" Oct 30 12:55:46.049675 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:46.049645 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/40764e2d-8e5e-427b-adb5-e3c0bf47e6c8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn\" (UID: \"40764e2d-8e5e-427b-adb5-e3c0bf47e6c8\") " pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn" Oct 30 12:55:46.083531 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:46.083509 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:46.150694 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:46.150668 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwqjb\" (UniqueName: \"kubernetes.io/projected/40764e2d-8e5e-427b-adb5-e3c0bf47e6c8-kube-api-access-vwqjb\") pod \"opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn\" (UID: \"40764e2d-8e5e-427b-adb5-e3c0bf47e6c8\") " pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn" Oct 30 12:55:46.150800 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:46.150711 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40764e2d-8e5e-427b-adb5-e3c0bf47e6c8-tmp\") pod \"opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn\" (UID: \"40764e2d-8e5e-427b-adb5-e3c0bf47e6c8\") " pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn" Oct 30 12:55:46.150800 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:46.150745 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/40764e2d-8e5e-427b-adb5-e3c0bf47e6c8-metrics-cert\") pod \"opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn\" (UID: \"40764e2d-8e5e-427b-adb5-e3c0bf47e6c8\") " pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn" Oct 30 12:55:46.150800 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:46.150775 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40764e2d-8e5e-427b-adb5-e3c0bf47e6c8-webhook-cert\") pod \"opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn\" (UID: \"40764e2d-8e5e-427b-adb5-e3c0bf47e6c8\") " pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn" Oct 30 12:55:46.150989 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:46.150812 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/40764e2d-8e5e-427b-adb5-e3c0bf47e6c8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn\" (UID: \"40764e2d-8e5e-427b-adb5-e3c0bf47e6c8\") " pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn" Oct 30 12:55:46.150989 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:46.150851 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/40764e2d-8e5e-427b-adb5-e3c0bf47e6c8-manifests\") pod \"opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn\" (UID: \"40764e2d-8e5e-427b-adb5-e3c0bf47e6c8\") " pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn" Oct 30 12:55:46.151216 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:46.151195 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/40764e2d-8e5e-427b-adb5-e3c0bf47e6c8-manifests\") pod \"opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn\" (UID: \"40764e2d-8e5e-427b-adb5-e3c0bf47e6c8\") " pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn" Oct 30 12:55:46.153201 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:46.153174 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40764e2d-8e5e-427b-adb5-e3c0bf47e6c8-tmp\") pod \"opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn\" (UID: \"40764e2d-8e5e-427b-adb5-e3c0bf47e6c8\") " pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn" Oct 30 12:55:46.153201 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:46.153189 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40764e2d-8e5e-427b-adb5-e3c0bf47e6c8-webhook-cert\") pod \"opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn\" (UID: \"40764e2d-8e5e-427b-adb5-e3c0bf47e6c8\") " pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn" Oct 30 12:55:46.153387 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:46.153367 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/40764e2d-8e5e-427b-adb5-e3c0bf47e6c8-metrics-cert\") pod \"opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn\" (UID: \"40764e2d-8e5e-427b-adb5-e3c0bf47e6c8\") " pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn" Oct 30 12:55:46.153424 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:46.153367 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/40764e2d-8e5e-427b-adb5-e3c0bf47e6c8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn\" (UID: \"40764e2d-8e5e-427b-adb5-e3c0bf47e6c8\") " pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn" Oct 30 12:55:46.175557 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:46.175528 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwqjb\" (UniqueName: \"kubernetes.io/projected/40764e2d-8e5e-427b-adb5-e3c0bf47e6c8-kube-api-access-vwqjb\") pod \"opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn\" (UID: \"40764e2d-8e5e-427b-adb5-e3c0bf47e6c8\") " pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn" Oct 30 12:55:46.206531 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:46.206476 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:46.259145 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:46.259125 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn" Oct 30 12:55:46.329654 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:46.329233 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:46.614145 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:46.614073 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn"] Oct 30 12:55:46.617742 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:55:46.617716 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40764e2d_8e5e_427b_adb5_e3c0bf47e6c8.slice/crio-0a1711c71e7547b4cb0e3e890d044765714013cb7370618bdc270f8ae1f9651d WatchSource:0}: Error finding container 0a1711c71e7547b4cb0e3e890d044765714013cb7370618bdc270f8ae1f9651d: Status 404 returned error can't find the container with id 0a1711c71e7547b4cb0e3e890d044765714013cb7370618bdc270f8ae1f9651d Oct 30 12:55:47.430648 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:47.425552 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn" event={"ID":"40764e2d-8e5e-427b-adb5-e3c0bf47e6c8","Type":"ContainerStarted","Data":"0a1711c71e7547b4cb0e3e890d044765714013cb7370618bdc270f8ae1f9651d"} Oct 30 12:55:49.434151 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:49.434120 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn" event={"ID":"40764e2d-8e5e-427b-adb5-e3c0bf47e6c8","Type":"ContainerStarted","Data":"4da1425a0ce69e6d24e8fb0fcd00243b28711d01bd23e1d3fee1ab638c7ea263"} Oct 30 12:55:50.438909 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:50.438880 5922 generic.go:356] "Generic (PLEG): container finished" podID="40764e2d-8e5e-427b-adb5-e3c0bf47e6c8" containerID="4da1425a0ce69e6d24e8fb0fcd00243b28711d01bd23e1d3fee1ab638c7ea263" exitCode=0 Oct 30 12:55:50.439289 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:50.438943 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn" event={"ID":"40764e2d-8e5e-427b-adb5-e3c0bf47e6c8","Type":"ContainerDied","Data":"4da1425a0ce69e6d24e8fb0fcd00243b28711d01bd23e1d3fee1ab638c7ea263"} Oct 30 12:55:51.443625 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:51.443589 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn" event={"ID":"40764e2d-8e5e-427b-adb5-e3c0bf47e6c8","Type":"ContainerStarted","Data":"dd55fe05c2c6da551e5817f6ff4d065c561e7443cdfa214886b8548a10c48869"} Oct 30 12:55:51.444045 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:51.443699 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn" Oct 30 12:55:51.534918 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:51.534874 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn" podStartSLOduration=3.020570805 podStartE2EDuration="6.534861467s" totalImagesPullingTime="3.077438641s" totalInitContainerRuntime="436.852021ms" isStatefulPod=false podCreationTimestamp="2026-10-30 12:55:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:51.522382005 +0000 UTC m=+207.991797749" watchObservedRunningTime="2026-10-30 12:55:51.534861467 +0000 UTC m=+208.004277211" Oct 30 12:55:53.551196 ip-10-0-137-21 kubenswrapper[5922]: E1030 12:55:53.551163 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 12:55:56.844964 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:56.844918 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:56.970638 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:55:56.970570 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:02.449160 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:02.449127 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn" Oct 30 12:56:04.857029 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:04.856986 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:05.920267 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:05.920230 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:07.989084 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:07.989050 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-57fb7b744b-n88dm"] Oct 30 12:56:07.992319 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:07.992302 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-57fb7b744b-n88dm" Oct 30 12:56:08.010693 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:08.010672 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Oct 30 12:56:08.022671 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:08.022639 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-bdndj\"" Oct 30 12:56:08.022825 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:08.022746 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Oct 30 12:56:08.024043 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:08.024024 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:56:08.024137 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:08.024053 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Oct 30 12:56:08.024137 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:08.024053 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Oct 30 12:56:08.024747 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:08.024732 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-57fb7b744b-n88dm"] Oct 30 12:56:08.099240 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:08.099217 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a708ca03-fd35-4e4e-b2f2-ca29e42b5926-cert\") pod \"lws-controller-manager-57fb7b744b-n88dm\" (UID: \"a708ca03-fd35-4e4e-b2f2-ca29e42b5926\") " pod="openshift-lws-operator/lws-controller-manager-57fb7b744b-n88dm" Oct 30 12:56:08.099350 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:08.099254 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wslfj\" (UniqueName: \"kubernetes.io/projected/a708ca03-fd35-4e4e-b2f2-ca29e42b5926-kube-api-access-wslfj\") pod \"lws-controller-manager-57fb7b744b-n88dm\" (UID: \"a708ca03-fd35-4e4e-b2f2-ca29e42b5926\") " pod="openshift-lws-operator/lws-controller-manager-57fb7b744b-n88dm" Oct 30 12:56:08.099350 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:08.099275 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a708ca03-fd35-4e4e-b2f2-ca29e42b5926-manager-config\") pod \"lws-controller-manager-57fb7b744b-n88dm\" (UID: \"a708ca03-fd35-4e4e-b2f2-ca29e42b5926\") " pod="openshift-lws-operator/lws-controller-manager-57fb7b744b-n88dm" Oct 30 12:56:08.099442 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:08.099396 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a708ca03-fd35-4e4e-b2f2-ca29e42b5926-metrics-cert\") pod \"lws-controller-manager-57fb7b744b-n88dm\" (UID: \"a708ca03-fd35-4e4e-b2f2-ca29e42b5926\") " pod="openshift-lws-operator/lws-controller-manager-57fb7b744b-n88dm" Oct 30 12:56:08.200100 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:08.200074 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wslfj\" (UniqueName: \"kubernetes.io/projected/a708ca03-fd35-4e4e-b2f2-ca29e42b5926-kube-api-access-wslfj\") pod \"lws-controller-manager-57fb7b744b-n88dm\" (UID: \"a708ca03-fd35-4e4e-b2f2-ca29e42b5926\") " pod="openshift-lws-operator/lws-controller-manager-57fb7b744b-n88dm" Oct 30 12:56:08.200188 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:08.200106 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a708ca03-fd35-4e4e-b2f2-ca29e42b5926-manager-config\") pod \"lws-controller-manager-57fb7b744b-n88dm\" (UID: \"a708ca03-fd35-4e4e-b2f2-ca29e42b5926\") " pod="openshift-lws-operator/lws-controller-manager-57fb7b744b-n88dm" Oct 30 12:56:08.200188 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:08.200141 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a708ca03-fd35-4e4e-b2f2-ca29e42b5926-metrics-cert\") pod \"lws-controller-manager-57fb7b744b-n88dm\" (UID: \"a708ca03-fd35-4e4e-b2f2-ca29e42b5926\") " pod="openshift-lws-operator/lws-controller-manager-57fb7b744b-n88dm" Oct 30 12:56:08.200188 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:08.200169 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a708ca03-fd35-4e4e-b2f2-ca29e42b5926-cert\") pod \"lws-controller-manager-57fb7b744b-n88dm\" (UID: \"a708ca03-fd35-4e4e-b2f2-ca29e42b5926\") " pod="openshift-lws-operator/lws-controller-manager-57fb7b744b-n88dm" Oct 30 12:56:08.200768 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:08.200748 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a708ca03-fd35-4e4e-b2f2-ca29e42b5926-manager-config\") pod \"lws-controller-manager-57fb7b744b-n88dm\" (UID: \"a708ca03-fd35-4e4e-b2f2-ca29e42b5926\") " pod="openshift-lws-operator/lws-controller-manager-57fb7b744b-n88dm" Oct 30 12:56:08.202486 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:08.202465 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a708ca03-fd35-4e4e-b2f2-ca29e42b5926-cert\") pod \"lws-controller-manager-57fb7b744b-n88dm\" (UID: \"a708ca03-fd35-4e4e-b2f2-ca29e42b5926\") " pod="openshift-lws-operator/lws-controller-manager-57fb7b744b-n88dm" Oct 30 12:56:08.202564 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:08.202527 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a708ca03-fd35-4e4e-b2f2-ca29e42b5926-metrics-cert\") pod \"lws-controller-manager-57fb7b744b-n88dm\" (UID: \"a708ca03-fd35-4e4e-b2f2-ca29e42b5926\") " pod="openshift-lws-operator/lws-controller-manager-57fb7b744b-n88dm" Oct 30 12:56:08.224477 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:08.224452 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wslfj\" (UniqueName: \"kubernetes.io/projected/a708ca03-fd35-4e4e-b2f2-ca29e42b5926-kube-api-access-wslfj\") pod \"lws-controller-manager-57fb7b744b-n88dm\" (UID: \"a708ca03-fd35-4e4e-b2f2-ca29e42b5926\") " pod="openshift-lws-operator/lws-controller-manager-57fb7b744b-n88dm" Oct 30 12:56:08.304150 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:08.304100 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-57fb7b744b-n88dm" Oct 30 12:56:08.597502 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:08.597428 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-57fb7b744b-n88dm"] Oct 30 12:56:08.602035 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:56:08.602008 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda708ca03_fd35_4e4e_b2f2_ca29e42b5926.slice/crio-5c3a7b5388c6d125d4e362b56c17fe255892fb74196777f292689d19551f2a93 WatchSource:0}: Error finding container 5c3a7b5388c6d125d4e362b56c17fe255892fb74196777f292689d19551f2a93: Status 404 returned error can't find the container with id 5c3a7b5388c6d125d4e362b56c17fe255892fb74196777f292689d19551f2a93 Oct 30 12:56:08.700380 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:08.700349 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:09.018074 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:09.018038 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:09.371978 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:09.371895 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:09.510417 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:09.510390 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-57fb7b744b-n88dm" event={"ID":"a708ca03-fd35-4e4e-b2f2-ca29e42b5926","Type":"ContainerStarted","Data":"5c3a7b5388c6d125d4e362b56c17fe255892fb74196777f292689d19551f2a93"} Oct 30 12:56:11.265366 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:11.265330 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-d59dd5dc7-rc4c8"] Oct 30 12:56:11.268798 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:11.268774 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d59dd5dc7-rc4c8" Oct 30 12:56:11.269456 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:11.269431 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:11.296425 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:11.296398 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d59dd5dc7-rc4c8"] Oct 30 12:56:11.330784 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:11.330762 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6qgz\" (UniqueName: \"kubernetes.io/projected/3108a560-3472-4283-b17e-2099fd70bd78-kube-api-access-h6qgz\") pod \"kube-auth-proxy-d59dd5dc7-rc4c8\" (UID: \"3108a560-3472-4283-b17e-2099fd70bd78\") " pod="openshift-ingress/kube-auth-proxy-d59dd5dc7-rc4c8" Oct 30 12:56:11.330911 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:11.330800 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3108a560-3472-4283-b17e-2099fd70bd78-tmp\") pod \"kube-auth-proxy-d59dd5dc7-rc4c8\" (UID: \"3108a560-3472-4283-b17e-2099fd70bd78\") " pod="openshift-ingress/kube-auth-proxy-d59dd5dc7-rc4c8" Oct 30 12:56:11.330911 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:11.330828 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3108a560-3472-4283-b17e-2099fd70bd78-tls-certs\") pod \"kube-auth-proxy-d59dd5dc7-rc4c8\" (UID: \"3108a560-3472-4283-b17e-2099fd70bd78\") " pod="openshift-ingress/kube-auth-proxy-d59dd5dc7-rc4c8" Oct 30 12:56:11.375420 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:11.375404 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 12:56:11.375576 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:11.375563 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-9fp8r\"" Oct 30 12:56:11.375656 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:11.375601 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 12:56:11.375791 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:11.375773 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 12:56:11.375857 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:11.375773 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 12:56:11.394338 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:11.394309 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:11.432025 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:11.432002 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6qgz\" (UniqueName: \"kubernetes.io/projected/3108a560-3472-4283-b17e-2099fd70bd78-kube-api-access-h6qgz\") pod \"kube-auth-proxy-d59dd5dc7-rc4c8\" (UID: \"3108a560-3472-4283-b17e-2099fd70bd78\") " pod="openshift-ingress/kube-auth-proxy-d59dd5dc7-rc4c8" Oct 30 12:56:11.432134 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:11.432043 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3108a560-3472-4283-b17e-2099fd70bd78-tmp\") pod \"kube-auth-proxy-d59dd5dc7-rc4c8\" (UID: \"3108a560-3472-4283-b17e-2099fd70bd78\") " pod="openshift-ingress/kube-auth-proxy-d59dd5dc7-rc4c8" Oct 30 12:56:11.432134 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:11.432065 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3108a560-3472-4283-b17e-2099fd70bd78-tls-certs\") pod \"kube-auth-proxy-d59dd5dc7-rc4c8\" (UID: \"3108a560-3472-4283-b17e-2099fd70bd78\") " pod="openshift-ingress/kube-auth-proxy-d59dd5dc7-rc4c8" Oct 30 12:56:11.434340 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:11.434316 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3108a560-3472-4283-b17e-2099fd70bd78-tmp\") pod \"kube-auth-proxy-d59dd5dc7-rc4c8\" (UID: \"3108a560-3472-4283-b17e-2099fd70bd78\") " pod="openshift-ingress/kube-auth-proxy-d59dd5dc7-rc4c8" Oct 30 12:56:11.434512 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:11.434496 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3108a560-3472-4283-b17e-2099fd70bd78-tls-certs\") pod \"kube-auth-proxy-d59dd5dc7-rc4c8\" (UID: \"3108a560-3472-4283-b17e-2099fd70bd78\") " pod="openshift-ingress/kube-auth-proxy-d59dd5dc7-rc4c8" Oct 30 12:56:11.456689 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:11.456671 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6qgz\" (UniqueName: \"kubernetes.io/projected/3108a560-3472-4283-b17e-2099fd70bd78-kube-api-access-h6qgz\") pod \"kube-auth-proxy-d59dd5dc7-rc4c8\" (UID: \"3108a560-3472-4283-b17e-2099fd70bd78\") " pod="openshift-ingress/kube-auth-proxy-d59dd5dc7-rc4c8" Oct 30 12:56:11.519482 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:11.519412 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-57fb7b744b-n88dm" event={"ID":"a708ca03-fd35-4e4e-b2f2-ca29e42b5926","Type":"ContainerStarted","Data":"2cf62081614bd176407346c96f179100d23f1546c56e4ede8532fe69ea5ab69f"} Oct 30 12:56:11.519593 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:11.519577 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-57fb7b744b-n88dm" Oct 30 12:56:11.571560 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:11.571515 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-57fb7b744b-n88dm" podStartSLOduration=1.816866202 podStartE2EDuration="4.571503169s" totalImagesPullingTime="2.754636967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:11.571262042 +0000 UTC m=+228.040677787" watchObservedRunningTime="2026-10-30 12:56:11.571503169 +0000 UTC m=+228.040918914" Oct 30 12:56:11.584479 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:11.584457 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d59dd5dc7-rc4c8" Oct 30 12:56:11.884830 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:11.884762 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d59dd5dc7-rc4c8"] Oct 30 12:56:11.888845 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:56:11.888819 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3108a560_3472_4283_b17e_2099fd70bd78.slice/crio-fb2595b1edf62c9d94bb549ba4341d38c22498328a5606a295b5f11c43d2fbc6 WatchSource:0}: Error finding container fb2595b1edf62c9d94bb549ba4341d38c22498328a5606a295b5f11c43d2fbc6: Status 404 returned error can't find the container with id fb2595b1edf62c9d94bb549ba4341d38c22498328a5606a295b5f11c43d2fbc6 Oct 30 12:56:12.524169 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:12.524141 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d59dd5dc7-rc4c8" event={"ID":"3108a560-3472-4283-b17e-2099fd70bd78","Type":"ContainerStarted","Data":"fb2595b1edf62c9d94bb549ba4341d38c22498328a5606a295b5f11c43d2fbc6"} Oct 30 12:56:12.936160 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:12.936084 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:13.113813 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:13.113780 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:14.506944 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:14.506913 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:14.533513 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:14.533487 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d59dd5dc7-rc4c8" event={"ID":"3108a560-3472-4283-b17e-2099fd70bd78","Type":"ContainerStarted","Data":"8b28cda6fabb95a643c546d8a11349c0f3f306344cf015e10c744c5ed8e4179a"} Oct 30 12:56:14.589448 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:14.589371 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-d59dd5dc7-rc4c8" podStartSLOduration=1.175917972 podStartE2EDuration="3.589358294s" totalImagesPullingTime="2.413440322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:14.588338992 +0000 UTC m=+231.057754736" watchObservedRunningTime="2026-10-30 12:56:14.589358294 +0000 UTC m=+231.058774059" Oct 30 12:56:16.776808 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:16.776770 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:16.898822 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:16.898792 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:17.215082 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:17.215051 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:17.336202 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:17.336168 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:18.049144 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:18.049108 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:22.526134 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:22.526079 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-57fb7b744b-n88dm" Oct 30 12:56:23.333718 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:23.333688 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:23.549348 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:23.549313 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:23.551068 ip-10-0-137-21 kubenswrapper[5922]: E1030 12:56:23.551039 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 12:56:31.210707 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:56:31.210673 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:53.550990 ip-10-0-137-21 kubenswrapper[5922]: E1030 12:56:53.550951 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 12:57:10.781233 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:57:10.781136 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:10.925865 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:57:10.925830 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:11.077300 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:57:11.077218 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:11.263414 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:57:11.263379 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:11.414432 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:57:11.414359 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-dmsjc"] Oct 30 12:57:11.416638 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:57:11.416607 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-dmsjc" Oct 30 12:57:11.444017 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:57:11.443988 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-dmsjc"] Oct 30 12:57:11.501035 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:57:11.501005 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 30 12:57:11.501201 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:57:11.501170 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 30 12:57:11.501201 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:57:11.501170 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-dgv65\"" Oct 30 12:57:11.541503 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:57:11.541464 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:11.582705 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:57:11.582683 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/18db6c7c-bd7b-4342-aa25-b184aca06d96-cert\") pod \"odh-model-controller-f87fcf56c-dmsjc\" (UID: \"18db6c7c-bd7b-4342-aa25-b184aca06d96\") " pod="opendatahub/odh-model-controller-f87fcf56c-dmsjc" Oct 30 12:57:11.582777 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:57:11.582719 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18db6c7c-bd7b-4342-aa25-b184aca06d96-metrics-certs\") pod \"odh-model-controller-f87fcf56c-dmsjc\" (UID: \"18db6c7c-bd7b-4342-aa25-b184aca06d96\") " pod="opendatahub/odh-model-controller-f87fcf56c-dmsjc" Oct 30 12:57:11.582813 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:57:11.582801 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqr2n\" (UniqueName: \"kubernetes.io/projected/18db6c7c-bd7b-4342-aa25-b184aca06d96-kube-api-access-qqr2n\") pod \"odh-model-controller-f87fcf56c-dmsjc\" (UID: \"18db6c7c-bd7b-4342-aa25-b184aca06d96\") " pod="opendatahub/odh-model-controller-f87fcf56c-dmsjc" Oct 30 12:57:11.674961 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:57:11.674891 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:11.683981 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:57:11.683958 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/18db6c7c-bd7b-4342-aa25-b184aca06d96-cert\") pod \"odh-model-controller-f87fcf56c-dmsjc\" (UID: \"18db6c7c-bd7b-4342-aa25-b184aca06d96\") " pod="opendatahub/odh-model-controller-f87fcf56c-dmsjc" Oct 30 12:57:11.684081 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:57:11.683997 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18db6c7c-bd7b-4342-aa25-b184aca06d96-metrics-certs\") pod \"odh-model-controller-f87fcf56c-dmsjc\" (UID: \"18db6c7c-bd7b-4342-aa25-b184aca06d96\") " pod="opendatahub/odh-model-controller-f87fcf56c-dmsjc" Oct 30 12:57:11.684081 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:57:11.684036 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqr2n\" (UniqueName: \"kubernetes.io/projected/18db6c7c-bd7b-4342-aa25-b184aca06d96-kube-api-access-qqr2n\") pod \"odh-model-controller-f87fcf56c-dmsjc\" (UID: \"18db6c7c-bd7b-4342-aa25-b184aca06d96\") " pod="opendatahub/odh-model-controller-f87fcf56c-dmsjc" Oct 30 12:57:11.686577 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:57:11.686553 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18db6c7c-bd7b-4342-aa25-b184aca06d96-metrics-certs\") pod \"odh-model-controller-f87fcf56c-dmsjc\" (UID: \"18db6c7c-bd7b-4342-aa25-b184aca06d96\") " pod="opendatahub/odh-model-controller-f87fcf56c-dmsjc" Oct 30 12:57:11.686694 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:57:11.686553 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/18db6c7c-bd7b-4342-aa25-b184aca06d96-cert\") pod \"odh-model-controller-f87fcf56c-dmsjc\" (UID: \"18db6c7c-bd7b-4342-aa25-b184aca06d96\") " pod="opendatahub/odh-model-controller-f87fcf56c-dmsjc" Oct 30 12:57:11.715732 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:57:11.715706 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqr2n\" (UniqueName: \"kubernetes.io/projected/18db6c7c-bd7b-4342-aa25-b184aca06d96-kube-api-access-qqr2n\") pod \"odh-model-controller-f87fcf56c-dmsjc\" (UID: \"18db6c7c-bd7b-4342-aa25-b184aca06d96\") " pod="opendatahub/odh-model-controller-f87fcf56c-dmsjc" Oct 30 12:57:11.727556 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:57:11.727537 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-dmsjc" Oct 30 12:57:11.996899 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:57:11.996868 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:12.020476 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:57:12.020444 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-dmsjc"] Oct 30 12:57:12.024343 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:57:12.024318 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18db6c7c_bd7b_4342_aa25_b184aca06d96.slice/crio-25a759f2c177c398d80a08f173a637e71288d81d0016335861aec583459614ee WatchSource:0}: Error finding container 25a759f2c177c398d80a08f173a637e71288d81d0016335861aec583459614ee: Status 404 returned error can't find the container with id 25a759f2c177c398d80a08f173a637e71288d81d0016335861aec583459614ee Oct 30 12:57:12.107949 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:57:12.107919 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:12.753788 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:57:12.753715 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-dmsjc" event={"ID":"18db6c7c-bd7b-4342-aa25-b184aca06d96","Type":"ContainerStarted","Data":"25a759f2c177c398d80a08f173a637e71288d81d0016335861aec583459614ee"} Oct 30 12:57:13.608626 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:57:13.608578 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:13.722982 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:57:13.722891 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:13.850249 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:57:13.850214 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:13.975561 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:57:13.975524 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:14.105744 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:57:14.105704 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:14.763731 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:57:14.763652 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-dmsjc" event={"ID":"18db6c7c-bd7b-4342-aa25-b184aca06d96","Type":"ContainerStarted","Data":"5246833c68b518f8c5d2b67a2e1b6658baeb0baeb7a769b308eef553dc2401d3"} Oct 30 12:57:14.764082 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:57:14.763736 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-f87fcf56c-dmsjc" Oct 30 12:57:14.805673 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:57:14.805604 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-f87fcf56c-dmsjc" podStartSLOduration=1.338774056 podStartE2EDuration="3.805589218s" totalImagesPullingTime="2.466815162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:14.805383051 +0000 UTC m=+291.274798797" watchObservedRunningTime="2026-10-30 12:57:14.805589218 +0000 UTC m=+291.275004962" Oct 30 12:57:23.550824 ip-10-0-137-21 kubenswrapper[5922]: E1030 12:57:23.550790 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 12:57:25.770000 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:57:25.769968 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-f87fcf56c-dmsjc" Oct 30 12:57:31.351394 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:57:31.351361 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:53.551558 ip-10-0-137-21 kubenswrapper[5922]: E1030 12:57:53.551523 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 12:57:58.176241 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:57:58.176207 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:58.299908 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:57:58.299876 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:10.299193 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:10.299154 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:12.907566 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:12.907531 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:13.289047 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:13.289015 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:13.685201 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:13.685131 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:16.536973 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:16.536940 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:17.322282 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:17.322248 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:17.582673 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:17.582575 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:18.774310 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:18.774268 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-6v28d"] Oct 30 12:58:18.784695 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:18.784660 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-6v28d" Oct 30 12:58:18.812857 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:18.812831 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 12:58:18.812857 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:18.812835 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 12:58:18.813035 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:18.812879 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-kpggb\"" Oct 30 12:58:18.813035 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:18.812896 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 12:58:18.813839 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:18.813816 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-6v28d"] Oct 30 12:58:18.914398 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:18.914369 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmfqf\" (UniqueName: \"kubernetes.io/projected/8ab008c5-fd4d-42b2-8f7e-67b35e9aae15-kube-api-access-mmfqf\") pod \"dns-operator-controller-manager-65bfd79598-6v28d\" (UID: \"8ab008c5-fd4d-42b2-8f7e-67b35e9aae15\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-6v28d" Oct 30 12:58:18.921091 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:18.921065 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:19.015508 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:19.015481 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmfqf\" (UniqueName: \"kubernetes.io/projected/8ab008c5-fd4d-42b2-8f7e-67b35e9aae15-kube-api-access-mmfqf\") pod \"dns-operator-controller-manager-65bfd79598-6v28d\" (UID: \"8ab008c5-fd4d-42b2-8f7e-67b35e9aae15\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-6v28d" Oct 30 12:58:19.042040 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:19.041983 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmfqf\" (UniqueName: \"kubernetes.io/projected/8ab008c5-fd4d-42b2-8f7e-67b35e9aae15-kube-api-access-mmfqf\") pod \"dns-operator-controller-manager-65bfd79598-6v28d\" (UID: \"8ab008c5-fd4d-42b2-8f7e-67b35e9aae15\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-6v28d" Oct 30 12:58:19.098683 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:19.098659 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-6v28d" Oct 30 12:58:19.155873 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:19.155822 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:19.277813 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:19.277781 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:19.405653 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:19.405566 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-6v28d"] Oct 30 12:58:19.409173 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:58:19.409149 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ab008c5_fd4d_42b2_8f7e_67b35e9aae15.slice/crio-0ae6e4055b7f8e5afa1c792ad4050689ce28932f9881cf363da4dbfca087be50 WatchSource:0}: Error finding container 0ae6e4055b7f8e5afa1c792ad4050689ce28932f9881cf363da4dbfca087be50: Status 404 returned error can't find the container with id 0ae6e4055b7f8e5afa1c792ad4050689ce28932f9881cf363da4dbfca087be50 Oct 30 12:58:19.455379 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:19.455354 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:20.012005 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:20.011975 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-6v28d" event={"ID":"8ab008c5-fd4d-42b2-8f7e-67b35e9aae15","Type":"ContainerStarted","Data":"0ae6e4055b7f8e5afa1c792ad4050689ce28932f9881cf363da4dbfca087be50"} Oct 30 12:58:20.542765 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:20.542681 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:21.983991 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:21.983958 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-dn2k2"] Oct 30 12:58:21.986910 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:21.986895 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-dn2k2" Oct 30 12:58:22.004836 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:22.004816 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-zzv9l\"" Oct 30 12:58:22.015771 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:22.015750 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-dn2k2"] Oct 30 12:58:22.039194 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:22.039171 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swtp7\" (UniqueName: \"kubernetes.io/projected/4c349358-62be-42fe-a686-ec417fd3d841-kube-api-access-swtp7\") pod \"authorino-operator-68d6dd76dc-dn2k2\" (UID: \"4c349358-62be-42fe-a686-ec417fd3d841\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-dn2k2" Oct 30 12:58:22.140013 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:22.139986 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swtp7\" (UniqueName: \"kubernetes.io/projected/4c349358-62be-42fe-a686-ec417fd3d841-kube-api-access-swtp7\") pod \"authorino-operator-68d6dd76dc-dn2k2\" (UID: \"4c349358-62be-42fe-a686-ec417fd3d841\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-dn2k2" Oct 30 12:58:22.165427 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:22.165404 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swtp7\" (UniqueName: \"kubernetes.io/projected/4c349358-62be-42fe-a686-ec417fd3d841-kube-api-access-swtp7\") pod \"authorino-operator-68d6dd76dc-dn2k2\" (UID: \"4c349358-62be-42fe-a686-ec417fd3d841\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-dn2k2" Oct 30 12:58:22.299746 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:22.299676 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-dn2k2" Oct 30 12:58:22.603115 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:22.603044 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-dn2k2"] Oct 30 12:58:22.607684 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:58:22.607655 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c349358_62be_42fe_a686_ec417fd3d841.slice/crio-0edcba076075f68537cceddacc3ce23bd23c54575a1ec8288787e5baae7fe146 WatchSource:0}: Error finding container 0edcba076075f68537cceddacc3ce23bd23c54575a1ec8288787e5baae7fe146: Status 404 returned error can't find the container with id 0edcba076075f68537cceddacc3ce23bd23c54575a1ec8288787e5baae7fe146 Oct 30 12:58:22.923025 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:22.922956 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:23.025911 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:23.025880 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-6v28d" event={"ID":"8ab008c5-fd4d-42b2-8f7e-67b35e9aae15","Type":"ContainerStarted","Data":"5f287270a48b03c4923760c955489a514e2eb68c91dca10766b2595e577aa8da"} Oct 30 12:58:23.026271 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:23.026005 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-6v28d" Oct 30 12:58:23.027193 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:23.027170 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-dn2k2" event={"ID":"4c349358-62be-42fe-a686-ec417fd3d841","Type":"ContainerStarted","Data":"0edcba076075f68537cceddacc3ce23bd23c54575a1ec8288787e5baae7fe146"} Oct 30 12:58:23.044051 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:23.044027 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:23.067912 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:23.067868 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-6v28d" podStartSLOduration=2.473357661 podStartE2EDuration="5.067857495s" totalImagesPullingTime="2.594499834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:23.067138472 +0000 UTC m=+359.536554230" watchObservedRunningTime="2026-10-30 12:58:23.067857495 +0000 UTC m=+359.537273244" Oct 30 12:58:23.554508 ip-10-0-137-21 kubenswrapper[5922]: E1030 12:58:23.554470 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 12:58:24.032319 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:24.032281 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-dn2k2" event={"ID":"4c349358-62be-42fe-a686-ec417fd3d841","Type":"ContainerStarted","Data":"909b754f4c6056f5c6a75921a3b55b45c7d787cc66fad20f567bf46414811367"} Oct 30 12:58:24.032789 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:24.032384 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-dn2k2" Oct 30 12:58:24.624583 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:24.624545 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:24.908825 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:24.908744 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:27.287978 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:27.287938 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:27.418618 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:27.418590 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:27.553152 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:27.553077 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:27.689078 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:27.689048 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:31.498580 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:31.498544 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:33.667862 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:33.667829 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:33.823757 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:33.823722 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:34.034406 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:34.034377 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-6v28d" Oct 30 12:58:34.080335 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:34.080290 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d6dd76dc-dn2k2" podStartSLOduration=11.948602391 podStartE2EDuration="13.080279264s" totalImagesPullingTime="1.131676873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:24.087231792 +0000 UTC m=+360.556647537" watchObservedRunningTime="2026-10-30 12:58:34.080279264 +0000 UTC m=+370.549695006" Oct 30 12:58:35.040988 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:35.040914 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-dn2k2" Oct 30 12:58:40.097595 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:40.097565 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:40.249374 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:40.249345 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:40.391269 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:40.391201 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:42.845783 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:42.845748 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:43.018330 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:43.018297 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:46.309721 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:46.309679 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:46.502515 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:46.502475 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:46.623970 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:46.623894 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:46.761557 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:46.761525 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:46.886295 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:46.886230 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:47.011072 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:47.011045 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:47.152256 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:47.152189 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:52.957458 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:52.957419 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:53.076029 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:53.075998 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:53.553099 ip-10-0-137-21 kubenswrapper[5922]: E1030 12:58:53.553064 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 12:58:54.840872 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:54.840837 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:54.958393 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:54.958353 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:59.498656 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:59.498602 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:59.620412 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:58:59.620380 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:07.577939 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:07.577899 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:07.722031 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:07.721994 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:10.851129 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:10.851096 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:10.970339 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:10.970306 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:11.090228 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:11.090172 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:11.662550 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:11.662519 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 12:59:11.664772 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:11.664758 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-5rxtm" Oct 30 12:59:11.697564 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:11.697541 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 12:59:11.708078 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:11.708056 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 12:59:11.708193 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:11.708096 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-chvlr\"" Oct 30 12:59:11.747093 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:11.747068 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 12:59:11.826541 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:11.826511 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fw4c\" (UniqueName: \"kubernetes.io/projected/4480c6ba-b6e6-4f0d-bc25-439bb9dcde74-kube-api-access-8fw4c\") pod \"limitador-limitador-7c9b87d574-5rxtm\" (UID: \"4480c6ba-b6e6-4f0d-bc25-439bb9dcde74\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-5rxtm" Oct 30 12:59:11.826669 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:11.826645 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4480c6ba-b6e6-4f0d-bc25-439bb9dcde74-config-file\") pod \"limitador-limitador-7c9b87d574-5rxtm\" (UID: \"4480c6ba-b6e6-4f0d-bc25-439bb9dcde74\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-5rxtm" Oct 30 12:59:11.927564 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:11.927498 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4480c6ba-b6e6-4f0d-bc25-439bb9dcde74-config-file\") pod \"limitador-limitador-7c9b87d574-5rxtm\" (UID: \"4480c6ba-b6e6-4f0d-bc25-439bb9dcde74\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-5rxtm" Oct 30 12:59:11.927564 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:11.927554 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fw4c\" (UniqueName: \"kubernetes.io/projected/4480c6ba-b6e6-4f0d-bc25-439bb9dcde74-kube-api-access-8fw4c\") pod \"limitador-limitador-7c9b87d574-5rxtm\" (UID: \"4480c6ba-b6e6-4f0d-bc25-439bb9dcde74\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-5rxtm" Oct 30 12:59:11.928159 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:11.928140 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4480c6ba-b6e6-4f0d-bc25-439bb9dcde74-config-file\") pod \"limitador-limitador-7c9b87d574-5rxtm\" (UID: \"4480c6ba-b6e6-4f0d-bc25-439bb9dcde74\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-5rxtm" Oct 30 12:59:11.955557 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:11.955533 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fw4c\" (UniqueName: \"kubernetes.io/projected/4480c6ba-b6e6-4f0d-bc25-439bb9dcde74-kube-api-access-8fw4c\") pod \"limitador-limitador-7c9b87d574-5rxtm\" (UID: \"4480c6ba-b6e6-4f0d-bc25-439bb9dcde74\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-5rxtm" Oct 30 12:59:11.976074 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:11.976054 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-5rxtm" Oct 30 12:59:11.997528 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:11.997493 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:12.114937 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:12.114898 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:12.281301 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:12.281273 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 12:59:12.285133 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:59:12.285103 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4480c6ba_b6e6_4f0d_bc25_439bb9dcde74.slice/crio-c88c1131ce9f22c2395e0bf476e1fe1723354a130d6ed5567226e0c925fc4e5b WatchSource:0}: Error finding container c88c1131ce9f22c2395e0bf476e1fe1723354a130d6ed5567226e0c925fc4e5b: Status 404 returned error can't find the container with id c88c1131ce9f22c2395e0bf476e1fe1723354a130d6ed5567226e0c925fc4e5b Oct 30 12:59:13.223053 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:13.223011 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-5rxtm" event={"ID":"4480c6ba-b6e6-4f0d-bc25-439bb9dcde74","Type":"ContainerStarted","Data":"c88c1131ce9f22c2395e0bf476e1fe1723354a130d6ed5567226e0c925fc4e5b"} Oct 30 12:59:14.228846 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:14.228814 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-5rxtm" event={"ID":"4480c6ba-b6e6-4f0d-bc25-439bb9dcde74","Type":"ContainerStarted","Data":"5d8e3eea2c1c20c2ab56907ba757177205013604f599b5a9c68849a60154cdab"} Oct 30 12:59:14.229274 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:14.228958 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-5rxtm" Oct 30 12:59:14.271591 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:14.271543 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-5rxtm" podStartSLOduration=2.253539393 podStartE2EDuration="3.271528633s" totalImagesPullingTime="1.01798924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:14.271367365 +0000 UTC m=+410.740783110" watchObservedRunningTime="2026-10-30 12:59:14.271528633 +0000 UTC m=+410.740944436" Oct 30 12:59:23.551743 ip-10-0-137-21 kubenswrapper[5922]: E1030 12:59:23.551706 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 12:59:23.663975 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:23.663948 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:23.779793 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:23.779747 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:25.235387 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:25.235361 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-5rxtm" Oct 30 12:59:27.081741 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:27.081704 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:27.218075 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:27.218008 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:27.845561 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:27.845525 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:31.615809 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:31.615778 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:39.996338 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:39.996301 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:40.117889 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:40.117857 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:40.539826 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:40.539795 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-svdg6"] Oct 30 12:59:40.542052 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:40.542027 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-svdg6" Oct 30 12:59:40.551910 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:40.551887 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c6mv\" (UniqueName: \"kubernetes.io/projected/248c6c3f-e2a0-4091-be0a-d89ff0e8bcb9-kube-api-access-4c6mv\") pod \"authorino-6889cbd8c4-svdg6\" (UID: \"248c6c3f-e2a0-4091-be0a-d89ff0e8bcb9\") " pod="kuadrant-system/authorino-6889cbd8c4-svdg6" Oct 30 12:59:40.552018 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:40.551923 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/248c6c3f-e2a0-4091-be0a-d89ff0e8bcb9-tls-cert\") pod \"authorino-6889cbd8c4-svdg6\" (UID: \"248c6c3f-e2a0-4091-be0a-d89ff0e8bcb9\") " pod="kuadrant-system/authorino-6889cbd8c4-svdg6" Oct 30 12:59:40.564867 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:40.564845 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-svdg6"] Oct 30 12:59:40.571963 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:40.571941 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 12:59:40.572063 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:40.572022 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jv989\"" Oct 30 12:59:40.643869 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:40.643847 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-svdg6"] Oct 30 12:59:40.644025 ip-10-0-137-21 kubenswrapper[5922]: E1030 12:59:40.644009 5922 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-4c6mv tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-svdg6" podUID="248c6c3f-e2a0-4091-be0a-d89ff0e8bcb9" Oct 30 12:59:40.652477 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:40.652454 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c6mv\" (UniqueName: \"kubernetes.io/projected/248c6c3f-e2a0-4091-be0a-d89ff0e8bcb9-kube-api-access-4c6mv\") pod \"authorino-6889cbd8c4-svdg6\" (UID: \"248c6c3f-e2a0-4091-be0a-d89ff0e8bcb9\") " pod="kuadrant-system/authorino-6889cbd8c4-svdg6" Oct 30 12:59:40.652557 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:40.652490 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/248c6c3f-e2a0-4091-be0a-d89ff0e8bcb9-tls-cert\") pod \"authorino-6889cbd8c4-svdg6\" (UID: \"248c6c3f-e2a0-4091-be0a-d89ff0e8bcb9\") " pod="kuadrant-system/authorino-6889cbd8c4-svdg6" Oct 30 12:59:40.654745 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:40.654727 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/248c6c3f-e2a0-4091-be0a-d89ff0e8bcb9-tls-cert\") pod \"authorino-6889cbd8c4-svdg6\" (UID: \"248c6c3f-e2a0-4091-be0a-d89ff0e8bcb9\") " pod="kuadrant-system/authorino-6889cbd8c4-svdg6" Oct 30 12:59:40.683325 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:40.683307 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c6mv\" (UniqueName: \"kubernetes.io/projected/248c6c3f-e2a0-4091-be0a-d89ff0e8bcb9-kube-api-access-4c6mv\") pod \"authorino-6889cbd8c4-svdg6\" (UID: \"248c6c3f-e2a0-4091-be0a-d89ff0e8bcb9\") " pod="kuadrant-system/authorino-6889cbd8c4-svdg6" Oct 30 12:59:40.697604 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:40.697579 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-54cd9c6d8d-979bp"] Oct 30 12:59:40.699909 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:40.699891 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54cd9c6d8d-979bp" Oct 30 12:59:40.719997 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:40.719977 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-54cd9c6d8d-979bp"] Oct 30 12:59:40.753271 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:40.753248 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9zwb\" (UniqueName: \"kubernetes.io/projected/3ae6e702-4a08-4335-b546-fda8d04e466b-kube-api-access-q9zwb\") pod \"authorino-54cd9c6d8d-979bp\" (UID: \"3ae6e702-4a08-4335-b546-fda8d04e466b\") " pod="kuadrant-system/authorino-54cd9c6d8d-979bp" Oct 30 12:59:40.753371 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:40.753296 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3ae6e702-4a08-4335-b546-fda8d04e466b-tls-cert\") pod \"authorino-54cd9c6d8d-979bp\" (UID: \"3ae6e702-4a08-4335-b546-fda8d04e466b\") " pod="kuadrant-system/authorino-54cd9c6d8d-979bp" Oct 30 12:59:40.853815 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:40.853765 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3ae6e702-4a08-4335-b546-fda8d04e466b-tls-cert\") pod \"authorino-54cd9c6d8d-979bp\" (UID: \"3ae6e702-4a08-4335-b546-fda8d04e466b\") " pod="kuadrant-system/authorino-54cd9c6d8d-979bp" Oct 30 12:59:40.853910 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:40.853832 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9zwb\" (UniqueName: \"kubernetes.io/projected/3ae6e702-4a08-4335-b546-fda8d04e466b-kube-api-access-q9zwb\") pod \"authorino-54cd9c6d8d-979bp\" (UID: \"3ae6e702-4a08-4335-b546-fda8d04e466b\") " pod="kuadrant-system/authorino-54cd9c6d8d-979bp" Oct 30 12:59:40.856154 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:40.856133 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3ae6e702-4a08-4335-b546-fda8d04e466b-tls-cert\") pod \"authorino-54cd9c6d8d-979bp\" (UID: \"3ae6e702-4a08-4335-b546-fda8d04e466b\") " pod="kuadrant-system/authorino-54cd9c6d8d-979bp" Oct 30 12:59:40.885151 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:40.885131 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9zwb\" (UniqueName: \"kubernetes.io/projected/3ae6e702-4a08-4335-b546-fda8d04e466b-kube-api-access-q9zwb\") pod \"authorino-54cd9c6d8d-979bp\" (UID: \"3ae6e702-4a08-4335-b546-fda8d04e466b\") " pod="kuadrant-system/authorino-54cd9c6d8d-979bp" Oct 30 12:59:41.010263 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:41.010244 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54cd9c6d8d-979bp" Oct 30 12:59:41.304303 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:41.304277 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-54cd9c6d8d-979bp"] Oct 30 12:59:41.308901 ip-10-0-137-21 kubenswrapper[5922]: W1030 12:59:41.308874 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ae6e702_4a08_4335_b546_fda8d04e466b.slice/crio-d5848aee3c9ac2bcdf297df4ee5c403f38f6b73ce7106d99384f65e06ab185e5 WatchSource:0}: Error finding container d5848aee3c9ac2bcdf297df4ee5c403f38f6b73ce7106d99384f65e06ab185e5: Status 404 returned error can't find the container with id d5848aee3c9ac2bcdf297df4ee5c403f38f6b73ce7106d99384f65e06ab185e5 Oct 30 12:59:41.342678 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:41.342651 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54cd9c6d8d-979bp" event={"ID":"3ae6e702-4a08-4335-b546-fda8d04e466b","Type":"ContainerStarted","Data":"d5848aee3c9ac2bcdf297df4ee5c403f38f6b73ce7106d99384f65e06ab185e5"} Oct 30 12:59:41.342797 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:41.342683 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-svdg6" Oct 30 12:59:41.349230 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:41.349210 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-svdg6" Oct 30 12:59:41.356676 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:41.356659 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/248c6c3f-e2a0-4091-be0a-d89ff0e8bcb9-kube-api-access-4c6mv\" (UniqueName: \"kubernetes.io/projected/248c6c3f-e2a0-4091-be0a-d89ff0e8bcb9-kube-api-access-4c6mv\") pod \"248c6c3f-e2a0-4091-be0a-d89ff0e8bcb9\" (UID: \"248c6c3f-e2a0-4091-be0a-d89ff0e8bcb9\") " Oct 30 12:59:41.356761 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:41.356705 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/248c6c3f-e2a0-4091-be0a-d89ff0e8bcb9-tls-cert\" (UniqueName: \"kubernetes.io/secret/248c6c3f-e2a0-4091-be0a-d89ff0e8bcb9-tls-cert\") pod \"248c6c3f-e2a0-4091-be0a-d89ff0e8bcb9\" (UID: \"248c6c3f-e2a0-4091-be0a-d89ff0e8bcb9\") " Oct 30 12:59:41.358536 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:41.358514 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/248c6c3f-e2a0-4091-be0a-d89ff0e8bcb9-kube-api-access-4c6mv" pod "248c6c3f-e2a0-4091-be0a-d89ff0e8bcb9" (UID: "248c6c3f-e2a0-4091-be0a-d89ff0e8bcb9"). InnerVolumeSpecName "kube-api-access-4c6mv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:59:41.358728 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:41.358708 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/248c6c3f-e2a0-4091-be0a-d89ff0e8bcb9-tls-cert" pod "248c6c3f-e2a0-4091-be0a-d89ff0e8bcb9" (UID: "248c6c3f-e2a0-4091-be0a-d89ff0e8bcb9"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:59:41.457167 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:41.457145 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c6mv\" (UniqueName: \"kubernetes.io/projected/248c6c3f-e2a0-4091-be0a-d89ff0e8bcb9-kube-api-access-4c6mv\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:59:41.457167 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:41.457166 5922 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/248c6c3f-e2a0-4091-be0a-d89ff0e8bcb9-tls-cert\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 12:59:42.347292 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:42.347256 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-svdg6" Oct 30 12:59:42.425994 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:42.425961 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-svdg6"] Oct 30 12:59:42.431859 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:42.431830 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-svdg6"] Oct 30 12:59:43.353449 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:43.353415 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54cd9c6d8d-979bp" event={"ID":"3ae6e702-4a08-4335-b546-fda8d04e466b","Type":"ContainerStarted","Data":"b153fbbb16d1716285745bc84b2ecc273361f45c5cdbe2e83f336b85f1404bb4"} Oct 30 12:59:43.394051 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:43.394006 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-54cd9c6d8d-979bp" podStartSLOduration=2.137108891 podStartE2EDuration="3.393993884s" totalImagesPullingTime="1.256884993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:43.393753587 +0000 UTC m=+439.863169331" watchObservedRunningTime="2026-10-30 12:59:43.393993884 +0000 UTC m=+439.863409623" Oct 30 12:59:43.642549 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:43.642481 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="248c6c3f-e2a0-4091-be0a-d89ff0e8bcb9" path="/var/lib/kubelet/pods/248c6c3f-e2a0-4091-be0a-d89ff0e8bcb9/volumes" Oct 30 12:59:50.576314 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:50.576260 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:50.700693 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:50.700659 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:50.854852 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:50.854769 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:50.965954 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:50.965928 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:53.551326 ip-10-0-137-21 kubenswrapper[5922]: E1030 12:59:53.551286 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 12:59:54.306017 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:54.305975 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:54.417264 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:54.417229 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:57.836850 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:57.836819 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:57.956622 ip-10-0-137-21 kubenswrapper[5922]: I1030 12:59:57.956593 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:23.552190 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:00:23.552156 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:00:25.049165 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:00:25.049119 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:25.165980 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:00:25.165951 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:26.600388 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:00:26.600357 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:26.770526 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:00:26.770495 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:26.886458 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:00:26.886390 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:28.703386 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:00:28.703350 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:00:31.736741 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:00:31.736703 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:53.551136 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:00:53.551096 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:01:11.345201 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:11.345169 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:12.759251 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:12.759213 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:14.173393 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:14.173355 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:14.300676 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:14.300643 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:23.550971 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:01:23.550933 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:01:30.953685 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:30.953644 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:31.884486 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:31.884453 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:32.018392 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:32.018360 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:01:33.469718 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:33.469666 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:35.219786 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:35.219706 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz"] Oct 30 13:01:35.222426 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:35.222410 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" Oct 30 13:01:35.245850 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:35.245824 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-7g8ld\"" Oct 30 13:01:35.245850 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:35.245824 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 13:01:35.258011 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:35.257989 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 13:01:35.258011 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:35.258009 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 13:01:35.259036 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:35.259022 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz"] Oct 30 13:01:35.292838 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:35.292809 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:35.355979 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:35.355938 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14ae0aea-473f-4ca0-bcc0-f617b9839c8c-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz\" (UID: \"14ae0aea-473f-4ca0-bcc0-f617b9839c8c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" Oct 30 13:01:35.355979 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:35.355971 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14ae0aea-473f-4ca0-bcc0-f617b9839c8c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz\" (UID: \"14ae0aea-473f-4ca0-bcc0-f617b9839c8c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" Oct 30 13:01:35.356223 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:35.356003 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfxdh\" (UniqueName: \"kubernetes.io/projected/14ae0aea-473f-4ca0-bcc0-f617b9839c8c-kube-api-access-lfxdh\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz\" (UID: \"14ae0aea-473f-4ca0-bcc0-f617b9839c8c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" Oct 30 13:01:35.356223 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:35.356141 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14ae0aea-473f-4ca0-bcc0-f617b9839c8c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz\" (UID: \"14ae0aea-473f-4ca0-bcc0-f617b9839c8c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" Oct 30 13:01:35.356223 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:35.356179 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14ae0aea-473f-4ca0-bcc0-f617b9839c8c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz\" (UID: \"14ae0aea-473f-4ca0-bcc0-f617b9839c8c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" Oct 30 13:01:35.356388 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:35.356279 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14ae0aea-473f-4ca0-bcc0-f617b9839c8c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz\" (UID: \"14ae0aea-473f-4ca0-bcc0-f617b9839c8c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" Oct 30 13:01:35.456840 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:35.456803 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14ae0aea-473f-4ca0-bcc0-f617b9839c8c-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz\" (UID: \"14ae0aea-473f-4ca0-bcc0-f617b9839c8c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" Oct 30 13:01:35.457111 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:35.457081 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14ae0aea-473f-4ca0-bcc0-f617b9839c8c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz\" (UID: \"14ae0aea-473f-4ca0-bcc0-f617b9839c8c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" Oct 30 13:01:35.457270 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:35.457146 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfxdh\" (UniqueName: \"kubernetes.io/projected/14ae0aea-473f-4ca0-bcc0-f617b9839c8c-kube-api-access-lfxdh\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz\" (UID: \"14ae0aea-473f-4ca0-bcc0-f617b9839c8c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" Oct 30 13:01:35.457270 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:35.457182 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14ae0aea-473f-4ca0-bcc0-f617b9839c8c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz\" (UID: \"14ae0aea-473f-4ca0-bcc0-f617b9839c8c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" Oct 30 13:01:35.457270 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:35.457213 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14ae0aea-473f-4ca0-bcc0-f617b9839c8c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz\" (UID: \"14ae0aea-473f-4ca0-bcc0-f617b9839c8c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" Oct 30 13:01:35.457467 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:35.457272 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14ae0aea-473f-4ca0-bcc0-f617b9839c8c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz\" (UID: \"14ae0aea-473f-4ca0-bcc0-f617b9839c8c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" Oct 30 13:01:35.457467 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:35.457341 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14ae0aea-473f-4ca0-bcc0-f617b9839c8c-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz\" (UID: \"14ae0aea-473f-4ca0-bcc0-f617b9839c8c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" Oct 30 13:01:35.457559 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:35.457537 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14ae0aea-473f-4ca0-bcc0-f617b9839c8c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz\" (UID: \"14ae0aea-473f-4ca0-bcc0-f617b9839c8c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" Oct 30 13:01:35.457829 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:35.457810 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14ae0aea-473f-4ca0-bcc0-f617b9839c8c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz\" (UID: \"14ae0aea-473f-4ca0-bcc0-f617b9839c8c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" Oct 30 13:01:35.459520 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:35.459495 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14ae0aea-473f-4ca0-bcc0-f617b9839c8c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz\" (UID: \"14ae0aea-473f-4ca0-bcc0-f617b9839c8c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" Oct 30 13:01:35.460173 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:35.460152 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14ae0aea-473f-4ca0-bcc0-f617b9839c8c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz\" (UID: \"14ae0aea-473f-4ca0-bcc0-f617b9839c8c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" Oct 30 13:01:35.482664 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:35.482578 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfxdh\" (UniqueName: \"kubernetes.io/projected/14ae0aea-473f-4ca0-bcc0-f617b9839c8c-kube-api-access-lfxdh\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz\" (UID: \"14ae0aea-473f-4ca0-bcc0-f617b9839c8c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" Oct 30 13:01:35.533318 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:35.533293 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" Oct 30 13:01:35.832507 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:35.832426 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz"] Oct 30 13:01:35.836966 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:01:35.836943 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14ae0aea_473f_4ca0_bcc0_f617b9839c8c.slice/crio-6b885fb5f6308d17d280eac8280e396f3472161a05a23f8308d56e16690ab571 WatchSource:0}: Error finding container 6b885fb5f6308d17d280eac8280e396f3472161a05a23f8308d56e16690ab571: Status 404 returned error can't find the container with id 6b885fb5f6308d17d280eac8280e396f3472161a05a23f8308d56e16690ab571 Oct 30 13:01:35.838162 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:35.838142 5922 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:01:36.825443 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:36.825384 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" event={"ID":"14ae0aea-473f-4ca0-bcc0-f617b9839c8c","Type":"ContainerStarted","Data":"6b885fb5f6308d17d280eac8280e396f3472161a05a23f8308d56e16690ab571"} Oct 30 13:01:36.827921 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:36.827894 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:01:37.088747 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:37.088658 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:37.830679 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:37.830642 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" event={"ID":"14ae0aea-473f-4ca0-bcc0-f617b9839c8c","Type":"ContainerStarted","Data":"884ae3577fd94388d6d06ca68f9cdd8fd5282ee16c1b7953f1d8c313dcd9cdc5"} Oct 30 13:01:37.873697 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:37.873656 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" podStartSLOduration=1.8006551050000001 podStartE2EDuration="2.87364332s" totalImagesPullingTime="1.072988215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:37.873356069 +0000 UTC m=+554.342771813" watchObservedRunningTime="2026-10-30 13:01:37.87364332 +0000 UTC m=+554.343059064" Oct 30 13:01:38.534637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:38.534230 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" Oct 30 13:01:39.156577 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:39.156541 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:40.421587 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:40.421558 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-p22vk"] Oct 30 13:01:40.424087 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:40.424063 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p22vk" Oct 30 13:01:40.440227 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:40.440206 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 30 13:01:40.452471 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:40.452445 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-p22vk"] Oct 30 13:01:40.486530 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:40.486500 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:40.602412 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:40.602372 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6870ecd9-9f61-4a6f-a753-1ca41fe77166-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p22vk\" (UID: \"6870ecd9-9f61-4a6f-a753-1ca41fe77166\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p22vk" Oct 30 13:01:40.602601 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:40.602424 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6870ecd9-9f61-4a6f-a753-1ca41fe77166-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p22vk\" (UID: \"6870ecd9-9f61-4a6f-a753-1ca41fe77166\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p22vk" Oct 30 13:01:40.602601 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:40.602503 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6870ecd9-9f61-4a6f-a753-1ca41fe77166-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p22vk\" (UID: \"6870ecd9-9f61-4a6f-a753-1ca41fe77166\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p22vk" Oct 30 13:01:40.602601 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:40.602550 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6870ecd9-9f61-4a6f-a753-1ca41fe77166-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p22vk\" (UID: \"6870ecd9-9f61-4a6f-a753-1ca41fe77166\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p22vk" Oct 30 13:01:40.602821 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:40.602607 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6870ecd9-9f61-4a6f-a753-1ca41fe77166-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p22vk\" (UID: \"6870ecd9-9f61-4a6f-a753-1ca41fe77166\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p22vk" Oct 30 13:01:40.602821 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:40.602663 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7cth\" (UniqueName: \"kubernetes.io/projected/6870ecd9-9f61-4a6f-a753-1ca41fe77166-kube-api-access-z7cth\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p22vk\" (UID: \"6870ecd9-9f61-4a6f-a753-1ca41fe77166\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p22vk" Oct 30 13:01:40.703776 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:40.703743 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6870ecd9-9f61-4a6f-a753-1ca41fe77166-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p22vk\" (UID: \"6870ecd9-9f61-4a6f-a753-1ca41fe77166\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p22vk" Oct 30 13:01:40.703939 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:40.703785 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6870ecd9-9f61-4a6f-a753-1ca41fe77166-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p22vk\" (UID: \"6870ecd9-9f61-4a6f-a753-1ca41fe77166\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p22vk" Oct 30 13:01:40.703939 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:40.703914 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6870ecd9-9f61-4a6f-a753-1ca41fe77166-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p22vk\" (UID: \"6870ecd9-9f61-4a6f-a753-1ca41fe77166\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p22vk" Oct 30 13:01:40.704046 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:40.703967 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6870ecd9-9f61-4a6f-a753-1ca41fe77166-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p22vk\" (UID: \"6870ecd9-9f61-4a6f-a753-1ca41fe77166\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p22vk" Oct 30 13:01:40.704046 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:40.704024 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7cth\" (UniqueName: \"kubernetes.io/projected/6870ecd9-9f61-4a6f-a753-1ca41fe77166-kube-api-access-z7cth\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p22vk\" (UID: \"6870ecd9-9f61-4a6f-a753-1ca41fe77166\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p22vk" Oct 30 13:01:40.704151 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:40.704108 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6870ecd9-9f61-4a6f-a753-1ca41fe77166-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p22vk\" (UID: \"6870ecd9-9f61-4a6f-a753-1ca41fe77166\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p22vk" Oct 30 13:01:40.704315 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:40.704287 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6870ecd9-9f61-4a6f-a753-1ca41fe77166-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p22vk\" (UID: \"6870ecd9-9f61-4a6f-a753-1ca41fe77166\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p22vk" Oct 30 13:01:40.704452 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:40.704370 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6870ecd9-9f61-4a6f-a753-1ca41fe77166-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p22vk\" (UID: \"6870ecd9-9f61-4a6f-a753-1ca41fe77166\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p22vk" Oct 30 13:01:40.704738 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:40.704712 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6870ecd9-9f61-4a6f-a753-1ca41fe77166-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p22vk\" (UID: \"6870ecd9-9f61-4a6f-a753-1ca41fe77166\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p22vk" Oct 30 13:01:40.706205 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:40.706180 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6870ecd9-9f61-4a6f-a753-1ca41fe77166-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p22vk\" (UID: \"6870ecd9-9f61-4a6f-a753-1ca41fe77166\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p22vk" Oct 30 13:01:40.706812 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:40.706792 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6870ecd9-9f61-4a6f-a753-1ca41fe77166-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p22vk\" (UID: \"6870ecd9-9f61-4a6f-a753-1ca41fe77166\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p22vk" Oct 30 13:01:40.728946 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:40.728912 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7cth\" (UniqueName: \"kubernetes.io/projected/6870ecd9-9f61-4a6f-a753-1ca41fe77166-kube-api-access-z7cth\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p22vk\" (UID: \"6870ecd9-9f61-4a6f-a753-1ca41fe77166\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p22vk" Oct 30 13:01:40.737757 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:40.737734 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p22vk" Oct 30 13:01:40.942304 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:40.942272 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:01:41.078161 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:41.078073 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-p22vk"] Oct 30 13:01:41.082166 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:01:41.082135 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6870ecd9_9f61_4a6f_a753_1ca41fe77166.slice/crio-b48c728da7ac4d027094e81138d089b9d3634680c91c3b7c1f3b9e130bc834f4 WatchSource:0}: Error finding container b48c728da7ac4d027094e81138d089b9d3634680c91c3b7c1f3b9e130bc834f4: Status 404 returned error can't find the container with id b48c728da7ac4d027094e81138d089b9d3634680c91c3b7c1f3b9e130bc834f4 Oct 30 13:01:41.850087 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:41.850054 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p22vk" event={"ID":"6870ecd9-9f61-4a6f-a753-1ca41fe77166","Type":"ContainerStarted","Data":"f4dce091bee54db6f70bda383d743c236e9a7785e6bbd8c109b99e478f763490"} Oct 30 13:01:41.850461 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:41.850101 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p22vk" event={"ID":"6870ecd9-9f61-4a6f-a753-1ca41fe77166","Type":"ContainerStarted","Data":"b48c728da7ac4d027094e81138d089b9d3634680c91c3b7c1f3b9e130bc834f4"} Oct 30 13:01:41.901220 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:41.901171 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p22vk" podStartSLOduration=1.688867975 podStartE2EDuration="1.901156912s" totalImagesPullingTime="212.288937ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:41.900412537 +0000 UTC m=+558.369828282" watchObservedRunningTime="2026-10-30 13:01:41.901156912 +0000 UTC m=+558.370572657" Oct 30 13:01:42.682079 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:42.682041 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:42.738470 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:42.738442 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p22vk" Oct 30 13:01:45.533649 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:45.533590 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" Oct 30 13:01:45.551656 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:45.551634 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" Oct 30 13:01:45.878044 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:45.878016 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" Oct 30 13:01:48.147773 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:48.147739 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:01:49.331452 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:49.331426 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c755c8f75-7njmp"] Oct 30 13:01:49.333881 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:49.333865 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c755c8f75-7njmp" Oct 30 13:01:49.352212 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:49.352191 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c755c8f75-7njmp"] Oct 30 13:01:49.484991 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:49.484964 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdqpq\" (UniqueName: \"kubernetes.io/projected/578b67e5-39d7-44dc-b8e6-39c364ba1c5c-kube-api-access-wdqpq\") pod \"authorino-c755c8f75-7njmp\" (UID: \"578b67e5-39d7-44dc-b8e6-39c364ba1c5c\") " pod="kuadrant-system/authorino-c755c8f75-7njmp" Oct 30 13:01:49.485111 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:49.484994 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/578b67e5-39d7-44dc-b8e6-39c364ba1c5c-tls-cert\") pod \"authorino-c755c8f75-7njmp\" (UID: \"578b67e5-39d7-44dc-b8e6-39c364ba1c5c\") " pod="kuadrant-system/authorino-c755c8f75-7njmp" Oct 30 13:01:49.585778 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:49.585721 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdqpq\" (UniqueName: \"kubernetes.io/projected/578b67e5-39d7-44dc-b8e6-39c364ba1c5c-kube-api-access-wdqpq\") pod \"authorino-c755c8f75-7njmp\" (UID: \"578b67e5-39d7-44dc-b8e6-39c364ba1c5c\") " pod="kuadrant-system/authorino-c755c8f75-7njmp" Oct 30 13:01:49.585778 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:49.585754 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/578b67e5-39d7-44dc-b8e6-39c364ba1c5c-tls-cert\") pod \"authorino-c755c8f75-7njmp\" (UID: \"578b67e5-39d7-44dc-b8e6-39c364ba1c5c\") " pod="kuadrant-system/authorino-c755c8f75-7njmp" Oct 30 13:01:49.588092 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:49.588071 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/578b67e5-39d7-44dc-b8e6-39c364ba1c5c-tls-cert\") pod \"authorino-c755c8f75-7njmp\" (UID: \"578b67e5-39d7-44dc-b8e6-39c364ba1c5c\") " pod="kuadrant-system/authorino-c755c8f75-7njmp" Oct 30 13:01:49.610106 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:49.610081 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdqpq\" (UniqueName: \"kubernetes.io/projected/578b67e5-39d7-44dc-b8e6-39c364ba1c5c-kube-api-access-wdqpq\") pod \"authorino-c755c8f75-7njmp\" (UID: \"578b67e5-39d7-44dc-b8e6-39c364ba1c5c\") " pod="kuadrant-system/authorino-c755c8f75-7njmp" Oct 30 13:01:49.645215 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:49.645195 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c755c8f75-7njmp" Oct 30 13:01:49.968826 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:49.968788 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c755c8f75-7njmp"] Oct 30 13:01:49.973926 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:01:49.973895 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod578b67e5_39d7_44dc_b8e6_39c364ba1c5c.slice/crio-38dc8acdff7dbbae340f16127ca9f67626e775e3ec3ed95caeb12ffd94f007a5 WatchSource:0}: Error finding container 38dc8acdff7dbbae340f16127ca9f67626e775e3ec3ed95caeb12ffd94f007a5: Status 404 returned error can't find the container with id 38dc8acdff7dbbae340f16127ca9f67626e775e3ec3ed95caeb12ffd94f007a5 Oct 30 13:01:50.738933 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:50.738901 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p22vk" Oct 30 13:01:50.751608 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:50.751582 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p22vk" Oct 30 13:01:50.889131 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:50.889048 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c755c8f75-7njmp" event={"ID":"578b67e5-39d7-44dc-b8e6-39c364ba1c5c","Type":"ContainerStarted","Data":"5a272efed13f4d4db85e4b28462736126309b2e03ee4c44919298c61818bbaae"} Oct 30 13:01:50.889131 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:50.889087 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c755c8f75-7njmp" event={"ID":"578b67e5-39d7-44dc-b8e6-39c364ba1c5c","Type":"ContainerStarted","Data":"38dc8acdff7dbbae340f16127ca9f67626e775e3ec3ed95caeb12ffd94f007a5"} Oct 30 13:01:50.905784 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:50.905757 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p22vk" Oct 30 13:01:50.929827 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:50.929787 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-c755c8f75-7njmp" podStartSLOduration=1.3420797850000001 podStartE2EDuration="1.929776475s" totalImagesPullingTime="587.69669ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:50.929604136 +0000 UTC m=+567.399019885" watchObservedRunningTime="2026-10-30 13:01:50.929776475 +0000 UTC m=+567.399192217" Oct 30 13:01:50.957174 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:50.957135 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-54cd9c6d8d-979bp"] Oct 30 13:01:50.957371 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:50.957347 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-54cd9c6d8d-979bp" podUID="3ae6e702-4a08-4335-b546-fda8d04e466b" containerName="authorino" containerID="cri-o://b153fbbb16d1716285745bc84b2ecc273361f45c5cdbe2e83f336b85f1404bb4" gracePeriod=30 Oct 30 13:01:51.213843 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:51.213819 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54cd9c6d8d-979bp" Oct 30 13:01:51.403543 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:51.403513 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ae6e702-4a08-4335-b546-fda8d04e466b-kube-api-access-q9zwb\" (UniqueName: \"kubernetes.io/projected/3ae6e702-4a08-4335-b546-fda8d04e466b-kube-api-access-q9zwb\") pod \"3ae6e702-4a08-4335-b546-fda8d04e466b\" (UID: \"3ae6e702-4a08-4335-b546-fda8d04e466b\") " Oct 30 13:01:51.403730 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:51.403690 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ae6e702-4a08-4335-b546-fda8d04e466b-tls-cert\" (UniqueName: \"kubernetes.io/secret/3ae6e702-4a08-4335-b546-fda8d04e466b-tls-cert\") pod \"3ae6e702-4a08-4335-b546-fda8d04e466b\" (UID: \"3ae6e702-4a08-4335-b546-fda8d04e466b\") " Oct 30 13:01:51.405824 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:51.405789 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ae6e702-4a08-4335-b546-fda8d04e466b-kube-api-access-q9zwb" pod "3ae6e702-4a08-4335-b546-fda8d04e466b" (UID: "3ae6e702-4a08-4335-b546-fda8d04e466b"). InnerVolumeSpecName "kube-api-access-q9zwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:01:51.413362 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:51.413338 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ae6e702-4a08-4335-b546-fda8d04e466b-tls-cert" pod "3ae6e702-4a08-4335-b546-fda8d04e466b" (UID: "3ae6e702-4a08-4335-b546-fda8d04e466b"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:01:51.505312 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:51.505239 5922 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3ae6e702-4a08-4335-b546-fda8d04e466b-tls-cert\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:01:51.505312 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:51.505270 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9zwb\" (UniqueName: \"kubernetes.io/projected/3ae6e702-4a08-4335-b546-fda8d04e466b-kube-api-access-q9zwb\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:01:51.557700 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:51.557296 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:51.755995 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:51.755890 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:51.895853 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:51.895823 5922 generic.go:356] "Generic (PLEG): container finished" podID="3ae6e702-4a08-4335-b546-fda8d04e466b" containerID="b153fbbb16d1716285745bc84b2ecc273361f45c5cdbe2e83f336b85f1404bb4" exitCode=0 Oct 30 13:01:51.896012 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:51.895877 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54cd9c6d8d-979bp" Oct 30 13:01:51.896012 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:51.895909 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54cd9c6d8d-979bp" event={"ID":"3ae6e702-4a08-4335-b546-fda8d04e466b","Type":"ContainerDied","Data":"b153fbbb16d1716285745bc84b2ecc273361f45c5cdbe2e83f336b85f1404bb4"} Oct 30 13:01:51.896012 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:51.895953 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54cd9c6d8d-979bp" event={"ID":"3ae6e702-4a08-4335-b546-fda8d04e466b","Type":"ContainerDied","Data":"d5848aee3c9ac2bcdf297df4ee5c403f38f6b73ce7106d99384f65e06ab185e5"} Oct 30 13:01:51.896012 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:51.895969 5922 scope.go:122] "RemoveContainer" containerID="b153fbbb16d1716285745bc84b2ecc273361f45c5cdbe2e83f336b85f1404bb4" Oct 30 13:01:51.919541 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:51.919520 5922 scope.go:122] "RemoveContainer" containerID="b153fbbb16d1716285745bc84b2ecc273361f45c5cdbe2e83f336b85f1404bb4" Oct 30 13:01:51.920116 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:01:51.920093 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b153fbbb16d1716285745bc84b2ecc273361f45c5cdbe2e83f336b85f1404bb4\": container with ID starting with b153fbbb16d1716285745bc84b2ecc273361f45c5cdbe2e83f336b85f1404bb4 not found: ID does not exist" containerID="b153fbbb16d1716285745bc84b2ecc273361f45c5cdbe2e83f336b85f1404bb4" Oct 30 13:01:51.920284 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:51.920259 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b153fbbb16d1716285745bc84b2ecc273361f45c5cdbe2e83f336b85f1404bb4"} err="failed to get container status \"b153fbbb16d1716285745bc84b2ecc273361f45c5cdbe2e83f336b85f1404bb4\": rpc error: code = NotFound desc = could not find container \"b153fbbb16d1716285745bc84b2ecc273361f45c5cdbe2e83f336b85f1404bb4\": container with ID starting with b153fbbb16d1716285745bc84b2ecc273361f45c5cdbe2e83f336b85f1404bb4 not found: ID does not exist" Oct 30 13:01:51.950167 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:51.950136 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-54cd9c6d8d-979bp"] Oct 30 13:01:51.955905 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:51.955885 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-54cd9c6d8d-979bp"] Oct 30 13:01:53.552692 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:01:53.552651 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:01:53.643918 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:53.643880 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ae6e702-4a08-4335-b546-fda8d04e466b" path="/var/lib/kubelet/pods/3ae6e702-4a08-4335-b546-fda8d04e466b/volumes" Oct 30 13:01:56.960468 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:56.960426 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:57.122917 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:57.122871 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:57.157012 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:57.156986 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-rh4bz"] Oct 30 13:01:57.157379 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:57.157367 5922 cpu_manager.go:414] "containerMap: removing container" podUID="3ae6e702-4a08-4335-b546-fda8d04e466b" containerName="authorino" Oct 30 13:01:57.157417 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:57.157381 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ae6e702-4a08-4335-b546-fda8d04e466b" containerName="authorino" Oct 30 13:01:57.157449 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:57.157442 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ae6e702-4a08-4335-b546-fda8d04e466b" containerName="authorino" Oct 30 13:01:57.160369 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:57.160353 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rh4bz" Oct 30 13:01:57.171670 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:57.171646 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj7fq\" (UniqueName: \"kubernetes.io/projected/bd96022a-32a7-4b5d-a59b-b3678075d1f6-kube-api-access-kj7fq\") pod \"payload-pre-processing-7b98bb678c-rh4bz\" (UID: \"bd96022a-32a7-4b5d-a59b-b3678075d1f6\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-rh4bz" Oct 30 13:01:57.171810 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:57.171789 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd96022a-32a7-4b5d-a59b-b3678075d1f6-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-rh4bz\" (UID: \"bd96022a-32a7-4b5d-a59b-b3678075d1f6\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-rh4bz" Oct 30 13:01:57.182514 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:57.182489 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-rh4bz"] Oct 30 13:01:57.194212 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:57.194193 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 13:01:57.225702 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:57.225650 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-wdr7j"] Oct 30 13:01:57.227910 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:57.227894 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-wdr7j" Oct 30 13:01:57.244386 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:57.244366 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-42bzq\"" Oct 30 13:01:57.248826 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:57.248805 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-wdr7j"] Oct 30 13:01:57.272464 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:57.272443 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj7fq\" (UniqueName: \"kubernetes.io/projected/bd96022a-32a7-4b5d-a59b-b3678075d1f6-kube-api-access-kj7fq\") pod \"payload-pre-processing-7b98bb678c-rh4bz\" (UID: \"bd96022a-32a7-4b5d-a59b-b3678075d1f6\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-rh4bz" Oct 30 13:01:57.272571 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:57.272481 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj4rs\" (UniqueName: \"kubernetes.io/projected/a103895f-e01d-4661-b522-73301acf6e22-kube-api-access-tj4rs\") pod \"payload-processing-5f78687d5b-wdr7j\" (UID: \"a103895f-e01d-4661-b522-73301acf6e22\") " pod="openshift-ingress/payload-processing-5f78687d5b-wdr7j" Oct 30 13:01:57.272571 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:57.272527 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a103895f-e01d-4661-b522-73301acf6e22-plugins-config-volume\") pod \"payload-processing-5f78687d5b-wdr7j\" (UID: \"a103895f-e01d-4661-b522-73301acf6e22\") " pod="openshift-ingress/payload-processing-5f78687d5b-wdr7j" Oct 30 13:01:57.272571 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:57.272557 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd96022a-32a7-4b5d-a59b-b3678075d1f6-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-rh4bz\" (UID: \"bd96022a-32a7-4b5d-a59b-b3678075d1f6\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-rh4bz" Oct 30 13:01:57.273254 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:57.273236 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd96022a-32a7-4b5d-a59b-b3678075d1f6-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-rh4bz\" (UID: \"bd96022a-32a7-4b5d-a59b-b3678075d1f6\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-rh4bz" Oct 30 13:01:57.297186 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:57.297160 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj7fq\" (UniqueName: \"kubernetes.io/projected/bd96022a-32a7-4b5d-a59b-b3678075d1f6-kube-api-access-kj7fq\") pod \"payload-pre-processing-7b98bb678c-rh4bz\" (UID: \"bd96022a-32a7-4b5d-a59b-b3678075d1f6\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-rh4bz" Oct 30 13:01:57.373382 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:57.373356 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj4rs\" (UniqueName: \"kubernetes.io/projected/a103895f-e01d-4661-b522-73301acf6e22-kube-api-access-tj4rs\") pod \"payload-processing-5f78687d5b-wdr7j\" (UID: \"a103895f-e01d-4661-b522-73301acf6e22\") " pod="openshift-ingress/payload-processing-5f78687d5b-wdr7j" Oct 30 13:01:57.373502 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:57.373427 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a103895f-e01d-4661-b522-73301acf6e22-plugins-config-volume\") pod \"payload-processing-5f78687d5b-wdr7j\" (UID: \"a103895f-e01d-4661-b522-73301acf6e22\") " pod="openshift-ingress/payload-processing-5f78687d5b-wdr7j" Oct 30 13:01:57.374067 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:57.374045 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a103895f-e01d-4661-b522-73301acf6e22-plugins-config-volume\") pod \"payload-processing-5f78687d5b-wdr7j\" (UID: \"a103895f-e01d-4661-b522-73301acf6e22\") " pod="openshift-ingress/payload-processing-5f78687d5b-wdr7j" Oct 30 13:01:57.434132 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:57.434110 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj4rs\" (UniqueName: \"kubernetes.io/projected/a103895f-e01d-4661-b522-73301acf6e22-kube-api-access-tj4rs\") pod \"payload-processing-5f78687d5b-wdr7j\" (UID: \"a103895f-e01d-4661-b522-73301acf6e22\") " pod="openshift-ingress/payload-processing-5f78687d5b-wdr7j" Oct 30 13:01:57.471437 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:57.471418 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rh4bz" Oct 30 13:01:57.538943 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:57.538916 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-wdr7j" Oct 30 13:01:57.764864 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:57.764794 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-rh4bz"] Oct 30 13:01:57.768354 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:01:57.768325 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd96022a_32a7_4b5d_a59b_b3678075d1f6.slice/crio-9debe8a9a76e5ce42e4f972b2a996fd2c1b62ceac9c1a88479d195987e0f9462 WatchSource:0}: Error finding container 9debe8a9a76e5ce42e4f972b2a996fd2c1b62ceac9c1a88479d195987e0f9462: Status 404 returned error can't find the container with id 9debe8a9a76e5ce42e4f972b2a996fd2c1b62ceac9c1a88479d195987e0f9462 Oct 30 13:01:57.836106 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:57.836073 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-wdr7j"] Oct 30 13:01:57.839699 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:01:57.839674 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda103895f_e01d_4661_b522_73301acf6e22.slice/crio-bc2271102af035ac8d694c06d444fb519afa34631479304d019f1dbe8b3fab4d WatchSource:0}: Error finding container bc2271102af035ac8d694c06d444fb519afa34631479304d019f1dbe8b3fab4d: Status 404 returned error can't find the container with id bc2271102af035ac8d694c06d444fb519afa34631479304d019f1dbe8b3fab4d Oct 30 13:01:57.929764 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:57.929737 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-wdr7j" event={"ID":"a103895f-e01d-4661-b522-73301acf6e22","Type":"ContainerStarted","Data":"bc2271102af035ac8d694c06d444fb519afa34631479304d019f1dbe8b3fab4d"} Oct 30 13:01:57.930842 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:01:57.930820 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rh4bz" event={"ID":"bd96022a-32a7-4b5d-a59b-b3678075d1f6","Type":"ContainerStarted","Data":"9debe8a9a76e5ce42e4f972b2a996fd2c1b62ceac9c1a88479d195987e0f9462"} Oct 30 13:02:00.954045 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:00.953995 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rh4bz" event={"ID":"bd96022a-32a7-4b5d-a59b-b3678075d1f6","Type":"ContainerStarted","Data":"240077d97d37350b2ceb24ccad39de3db056a8c474ed3c4823c018457951e714"} Oct 30 13:02:00.954727 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:00.954701 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rh4bz" Oct 30 13:02:00.956524 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:00.956499 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-wdr7j" event={"ID":"a103895f-e01d-4661-b522-73301acf6e22","Type":"ContainerStarted","Data":"62c3b3628ca9d64a836a1fde723c8150467e3b3f7afccefa8d067ac62c3ba45c"} Oct 30 13:02:00.956658 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:00.956551 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-wdr7j" Oct 30 13:02:01.000318 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:01.000275 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rh4bz" podStartSLOduration=1.744924602 podStartE2EDuration="4.00026344s" totalImagesPullingTime="2.255338838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:00.999471865 +0000 UTC m=+577.468887628" watchObservedRunningTime="2026-10-30 13:02:01.00026344 +0000 UTC m=+577.469679186" Oct 30 13:02:01.041320 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:01.041282 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-wdr7j" podStartSLOduration=1.925479224 podStartE2EDuration="4.041272948s" totalImagesPullingTime="2.115793724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:01.040997803 +0000 UTC m=+577.510413593" watchObservedRunningTime="2026-10-30 13:02:01.041272948 +0000 UTC m=+577.510688695" Oct 30 13:02:06.503795 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:06.503764 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6f5bb998fd-4c485"] Oct 30 13:02:06.506190 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:06.506173 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6f5bb998fd-4c485" Oct 30 13:02:06.534885 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:06.534863 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-m7t6c\"" Oct 30 13:02:06.535042 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:06.535027 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 13:02:06.535118 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:06.535101 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 13:02:06.536234 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:06.536218 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6f5bb998fd-4c485"] Oct 30 13:02:06.550986 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:06.550960 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24gw5\" (UniqueName: \"kubernetes.io/projected/8ec441fb-e6b7-471b-bc1e-d418d1a516b1-kube-api-access-24gw5\") pod \"ai-gateway-controller-6f5bb998fd-4c485\" (UID: \"8ec441fb-e6b7-471b-bc1e-d418d1a516b1\") " pod="opendatahub/ai-gateway-controller-6f5bb998fd-4c485" Oct 30 13:02:06.551077 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:06.551041 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec441fb-e6b7-471b-bc1e-d418d1a516b1-metrics-certs\") pod \"ai-gateway-controller-6f5bb998fd-4c485\" (UID: \"8ec441fb-e6b7-471b-bc1e-d418d1a516b1\") " pod="opendatahub/ai-gateway-controller-6f5bb998fd-4c485" Oct 30 13:02:06.651705 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:06.651684 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24gw5\" (UniqueName: \"kubernetes.io/projected/8ec441fb-e6b7-471b-bc1e-d418d1a516b1-kube-api-access-24gw5\") pod \"ai-gateway-controller-6f5bb998fd-4c485\" (UID: \"8ec441fb-e6b7-471b-bc1e-d418d1a516b1\") " pod="opendatahub/ai-gateway-controller-6f5bb998fd-4c485" Oct 30 13:02:06.651797 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:06.651743 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec441fb-e6b7-471b-bc1e-d418d1a516b1-metrics-certs\") pod \"ai-gateway-controller-6f5bb998fd-4c485\" (UID: \"8ec441fb-e6b7-471b-bc1e-d418d1a516b1\") " pod="opendatahub/ai-gateway-controller-6f5bb998fd-4c485" Oct 30 13:02:06.654047 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:06.654029 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec441fb-e6b7-471b-bc1e-d418d1a516b1-metrics-certs\") pod \"ai-gateway-controller-6f5bb998fd-4c485\" (UID: \"8ec441fb-e6b7-471b-bc1e-d418d1a516b1\") " pod="opendatahub/ai-gateway-controller-6f5bb998fd-4c485" Oct 30 13:02:06.676456 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:06.676430 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24gw5\" (UniqueName: \"kubernetes.io/projected/8ec441fb-e6b7-471b-bc1e-d418d1a516b1-kube-api-access-24gw5\") pod \"ai-gateway-controller-6f5bb998fd-4c485\" (UID: \"8ec441fb-e6b7-471b-bc1e-d418d1a516b1\") " pod="opendatahub/ai-gateway-controller-6f5bb998fd-4c485" Oct 30 13:02:06.817827 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:06.817777 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6f5bb998fd-4c485" Oct 30 13:02:07.113748 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:07.113680 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6f5bb998fd-4c485"] Oct 30 13:02:07.117134 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:02:07.117106 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ec441fb_e6b7_471b_bc1e_d418d1a516b1.slice/crio-6ccf707bf22264d9ab51c362948bc87a4b7b4502d26b4039712f4314c23a4a45 WatchSource:0}: Error finding container 6ccf707bf22264d9ab51c362948bc87a4b7b4502d26b4039712f4314c23a4a45: Status 404 returned error can't find the container with id 6ccf707bf22264d9ab51c362948bc87a4b7b4502d26b4039712f4314c23a4a45 Oct 30 13:02:07.988136 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:07.988100 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6f5bb998fd-4c485" event={"ID":"8ec441fb-e6b7-471b-bc1e-d418d1a516b1","Type":"ContainerStarted","Data":"6ccf707bf22264d9ab51c362948bc87a4b7b4502d26b4039712f4314c23a4a45"} Oct 30 13:02:08.631657 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:08.631607 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5"] Oct 30 13:02:08.638244 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:08.638225 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5" Oct 30 13:02:08.661975 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:08.661948 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5"] Oct 30 13:02:08.669344 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:08.669311 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-g8svl\"" Oct 30 13:02:08.669460 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:08.669348 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 13:02:08.674318 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:08.674290 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3584258d-ca3c-4574-a39e-a70cc78d0ce6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5x6r5\" (UID: \"3584258d-ca3c-4574-a39e-a70cc78d0ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5" Oct 30 13:02:08.674431 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:08.674351 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3584258d-ca3c-4574-a39e-a70cc78d0ce6-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5x6r5\" (UID: \"3584258d-ca3c-4574-a39e-a70cc78d0ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5" Oct 30 13:02:08.674491 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:08.674471 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3584258d-ca3c-4574-a39e-a70cc78d0ce6-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5x6r5\" (UID: \"3584258d-ca3c-4574-a39e-a70cc78d0ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5" Oct 30 13:02:08.674547 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:08.674501 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3584258d-ca3c-4574-a39e-a70cc78d0ce6-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5x6r5\" (UID: \"3584258d-ca3c-4574-a39e-a70cc78d0ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5" Oct 30 13:02:08.674658 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:08.674604 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3584258d-ca3c-4574-a39e-a70cc78d0ce6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5x6r5\" (UID: \"3584258d-ca3c-4574-a39e-a70cc78d0ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5" Oct 30 13:02:08.674780 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:08.674664 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3584258d-ca3c-4574-a39e-a70cc78d0ce6-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5x6r5\" (UID: \"3584258d-ca3c-4574-a39e-a70cc78d0ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5" Oct 30 13:02:08.674780 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:08.674681 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgw7j\" (UniqueName: \"kubernetes.io/projected/3584258d-ca3c-4574-a39e-a70cc78d0ce6-kube-api-access-pgw7j\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5x6r5\" (UID: \"3584258d-ca3c-4574-a39e-a70cc78d0ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5" Oct 30 13:02:08.674780 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:08.674726 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3584258d-ca3c-4574-a39e-a70cc78d0ce6-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5x6r5\" (UID: \"3584258d-ca3c-4574-a39e-a70cc78d0ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5" Oct 30 13:02:08.674780 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:08.674757 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3584258d-ca3c-4574-a39e-a70cc78d0ce6-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5x6r5\" (UID: \"3584258d-ca3c-4574-a39e-a70cc78d0ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5" Oct 30 13:02:08.776014 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:08.775975 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3584258d-ca3c-4574-a39e-a70cc78d0ce6-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5x6r5\" (UID: \"3584258d-ca3c-4574-a39e-a70cc78d0ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5" Oct 30 13:02:08.776014 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:08.776015 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgw7j\" (UniqueName: \"kubernetes.io/projected/3584258d-ca3c-4574-a39e-a70cc78d0ce6-kube-api-access-pgw7j\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5x6r5\" (UID: \"3584258d-ca3c-4574-a39e-a70cc78d0ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5" Oct 30 13:02:08.776268 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:08.776090 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3584258d-ca3c-4574-a39e-a70cc78d0ce6-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5x6r5\" (UID: \"3584258d-ca3c-4574-a39e-a70cc78d0ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5" Oct 30 13:02:08.776268 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:08.776157 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3584258d-ca3c-4574-a39e-a70cc78d0ce6-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5x6r5\" (UID: \"3584258d-ca3c-4574-a39e-a70cc78d0ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5" Oct 30 13:02:08.776268 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:08.776210 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3584258d-ca3c-4574-a39e-a70cc78d0ce6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5x6r5\" (UID: \"3584258d-ca3c-4574-a39e-a70cc78d0ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5" Oct 30 13:02:08.776268 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:08.776228 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3584258d-ca3c-4574-a39e-a70cc78d0ce6-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5x6r5\" (UID: \"3584258d-ca3c-4574-a39e-a70cc78d0ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5" Oct 30 13:02:08.776428 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:08.776283 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3584258d-ca3c-4574-a39e-a70cc78d0ce6-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5x6r5\" (UID: \"3584258d-ca3c-4574-a39e-a70cc78d0ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5" Oct 30 13:02:08.776428 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:08.776302 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3584258d-ca3c-4574-a39e-a70cc78d0ce6-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5x6r5\" (UID: \"3584258d-ca3c-4574-a39e-a70cc78d0ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5" Oct 30 13:02:08.776428 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:08.776351 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3584258d-ca3c-4574-a39e-a70cc78d0ce6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5x6r5\" (UID: \"3584258d-ca3c-4574-a39e-a70cc78d0ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5" Oct 30 13:02:08.776530 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:08.776485 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3584258d-ca3c-4574-a39e-a70cc78d0ce6-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5x6r5\" (UID: \"3584258d-ca3c-4574-a39e-a70cc78d0ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5" Oct 30 13:02:08.776800 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:08.776775 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3584258d-ca3c-4574-a39e-a70cc78d0ce6-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5x6r5\" (UID: \"3584258d-ca3c-4574-a39e-a70cc78d0ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5" Oct 30 13:02:08.777075 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:08.777051 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3584258d-ca3c-4574-a39e-a70cc78d0ce6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5x6r5\" (UID: \"3584258d-ca3c-4574-a39e-a70cc78d0ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5" Oct 30 13:02:08.777221 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:08.777202 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3584258d-ca3c-4574-a39e-a70cc78d0ce6-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5x6r5\" (UID: \"3584258d-ca3c-4574-a39e-a70cc78d0ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5" Oct 30 13:02:08.777322 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:08.777238 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3584258d-ca3c-4574-a39e-a70cc78d0ce6-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5x6r5\" (UID: \"3584258d-ca3c-4574-a39e-a70cc78d0ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5" Oct 30 13:02:08.779154 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:08.779129 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3584258d-ca3c-4574-a39e-a70cc78d0ce6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5x6r5\" (UID: \"3584258d-ca3c-4574-a39e-a70cc78d0ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5" Oct 30 13:02:08.779355 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:08.779337 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3584258d-ca3c-4574-a39e-a70cc78d0ce6-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5x6r5\" (UID: \"3584258d-ca3c-4574-a39e-a70cc78d0ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5" Oct 30 13:02:08.802242 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:08.802213 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3584258d-ca3c-4574-a39e-a70cc78d0ce6-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5x6r5\" (UID: \"3584258d-ca3c-4574-a39e-a70cc78d0ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5" Oct 30 13:02:08.802692 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:08.802668 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgw7j\" (UniqueName: \"kubernetes.io/projected/3584258d-ca3c-4574-a39e-a70cc78d0ce6-kube-api-access-pgw7j\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5x6r5\" (UID: \"3584258d-ca3c-4574-a39e-a70cc78d0ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5" Oct 30 13:02:08.952868 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:08.952832 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5" Oct 30 13:02:09.277517 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:09.277490 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5"] Oct 30 13:02:09.278741 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:02:09.278711 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3584258d_ca3c_4574_a39e_a70cc78d0ce6.slice/crio-408289f0d67ca84d5da5db87712158d2a499cc85d2ba5c7d6f60918fc61b2bea WatchSource:0}: Error finding container 408289f0d67ca84d5da5db87712158d2a499cc85d2ba5c7d6f60918fc61b2bea: Status 404 returned error can't find the container with id 408289f0d67ca84d5da5db87712158d2a499cc85d2ba5c7d6f60918fc61b2bea Oct 30 13:02:09.997918 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:09.997885 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6f5bb998fd-4c485" event={"ID":"8ec441fb-e6b7-471b-bc1e-d418d1a516b1","Type":"ContainerStarted","Data":"53b325c2d027f04fc4c4c6bf4b5fc0f23466f92f79b5840bed318513809868a3"} Oct 30 13:02:09.998090 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:09.997942 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6f5bb998fd-4c485" Oct 30 13:02:09.999152 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:09.999122 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5" event={"ID":"3584258d-ca3c-4574-a39e-a70cc78d0ce6","Type":"ContainerStarted","Data":"408289f0d67ca84d5da5db87712158d2a499cc85d2ba5c7d6f60918fc61b2bea"} Oct 30 13:02:10.054439 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:10.054389 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6f5bb998fd-4c485" podStartSLOduration=1.826351744 podStartE2EDuration="4.054375858s" totalImagesPullingTime="2.228024114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:10.053459615 +0000 UTC m=+586.522875362" watchObservedRunningTime="2026-10-30 13:02:10.054375858 +0000 UTC m=+586.523791603" Oct 30 13:02:11.964081 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:11.964049 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-wdr7j" Oct 30 13:02:12.024821 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:12.024786 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:02:12.024975 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:12.024848 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:02:12.024975 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:12.024877 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:02:12.987673 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:12.987638 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rh4bz" Oct 30 13:02:13.014768 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:13.014743 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5" event={"ID":"3584258d-ca3c-4574-a39e-a70cc78d0ce6","Type":"ContainerStarted","Data":"79ea971f2c6bfe32c890978e585892207ab60c57f17d909d54acb8b91fb372ac"} Oct 30 13:02:13.953341 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:13.953306 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5" Oct 30 13:02:13.954769 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:13.954743 5922 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5x6r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 13:02:13.954890 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:13.954791 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5" podUID="3584258d-ca3c-4574-a39e-a70cc78d0ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 13:02:14.953985 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:14.953952 5922 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5x6r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 13:02:14.954356 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:14.954003 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5" podUID="3584258d-ca3c-4574-a39e-a70cc78d0ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 13:02:15.953537 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:15.953499 5922 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5x6r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 13:02:15.953760 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:15.953557 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5" podUID="3584258d-ca3c-4574-a39e-a70cc78d0ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 13:02:16.953391 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:16.953358 5922 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5x6r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 13:02:16.953792 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:16.953416 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5" podUID="3584258d-ca3c-4574-a39e-a70cc78d0ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 13:02:17.953950 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:17.953908 5922 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5x6r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 13:02:17.954465 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:17.953961 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5" podUID="3584258d-ca3c-4574-a39e-a70cc78d0ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 13:02:18.953955 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:18.953925 5922 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5x6r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 13:02:18.954458 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:18.953971 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5" podUID="3584258d-ca3c-4574-a39e-a70cc78d0ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 13:02:20.239351 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:20.239317 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5" Oct 30 13:02:20.239884 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:20.239713 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5" Oct 30 13:02:20.240753 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:20.240729 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5" Oct 30 13:02:20.321606 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:20.321552 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5" podStartSLOduration=9.578709188 podStartE2EDuration="12.321539098s" totalImagesPullingTime="2.74282991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:13.089739515 +0000 UTC m=+589.559155272" watchObservedRunningTime="2026-10-30 13:02:20.321539098 +0000 UTC m=+596.790954842" Oct 30 13:02:21.007347 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:21.007310 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6f5bb998fd-4c485" Oct 30 13:02:22.834209 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:22.834177 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:23.281949 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:23.281908 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:23.552123 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:02:23.552051 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:02:23.685440 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:23.685410 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl"] Oct 30 13:02:23.690574 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:23.690550 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl" Oct 30 13:02:23.722185 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:23.722154 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl"] Oct 30 13:02:23.802635 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:23.802394 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c29858c0-d396-4092-9e19-b76069383a82-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cxwwl\" (UID: \"c29858c0-d396-4092-9e19-b76069383a82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl" Oct 30 13:02:23.802839 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:23.802815 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29858c0-d396-4092-9e19-b76069383a82-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cxwwl\" (UID: \"c29858c0-d396-4092-9e19-b76069383a82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl" Oct 30 13:02:23.802922 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:23.802891 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c29858c0-d396-4092-9e19-b76069383a82-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cxwwl\" (UID: \"c29858c0-d396-4092-9e19-b76069383a82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl" Oct 30 13:02:23.802992 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:23.802957 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c29858c0-d396-4092-9e19-b76069383a82-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cxwwl\" (UID: \"c29858c0-d396-4092-9e19-b76069383a82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl" Oct 30 13:02:23.803047 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:23.803031 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c29858c0-d396-4092-9e19-b76069383a82-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cxwwl\" (UID: \"c29858c0-d396-4092-9e19-b76069383a82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl" Oct 30 13:02:23.803100 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:23.803069 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c29858c0-d396-4092-9e19-b76069383a82-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cxwwl\" (UID: \"c29858c0-d396-4092-9e19-b76069383a82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl" Oct 30 13:02:23.803150 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:23.803103 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c29858c0-d396-4092-9e19-b76069383a82-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cxwwl\" (UID: \"c29858c0-d396-4092-9e19-b76069383a82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl" Oct 30 13:02:23.803150 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:23.803141 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d84w4\" (UniqueName: \"kubernetes.io/projected/c29858c0-d396-4092-9e19-b76069383a82-kube-api-access-d84w4\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cxwwl\" (UID: \"c29858c0-d396-4092-9e19-b76069383a82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl" Oct 30 13:02:23.803268 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:23.803183 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29858c0-d396-4092-9e19-b76069383a82-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cxwwl\" (UID: \"c29858c0-d396-4092-9e19-b76069383a82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl" Oct 30 13:02:23.903553 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:23.903526 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d84w4\" (UniqueName: \"kubernetes.io/projected/c29858c0-d396-4092-9e19-b76069383a82-kube-api-access-d84w4\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cxwwl\" (UID: \"c29858c0-d396-4092-9e19-b76069383a82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl" Oct 30 13:02:23.904008 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:23.903568 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29858c0-d396-4092-9e19-b76069383a82-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cxwwl\" (UID: \"c29858c0-d396-4092-9e19-b76069383a82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl" Oct 30 13:02:23.904008 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:23.903633 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c29858c0-d396-4092-9e19-b76069383a82-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cxwwl\" (UID: \"c29858c0-d396-4092-9e19-b76069383a82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl" Oct 30 13:02:23.904008 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:23.903715 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29858c0-d396-4092-9e19-b76069383a82-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cxwwl\" (UID: \"c29858c0-d396-4092-9e19-b76069383a82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl" Oct 30 13:02:23.904008 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:23.903743 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c29858c0-d396-4092-9e19-b76069383a82-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cxwwl\" (UID: \"c29858c0-d396-4092-9e19-b76069383a82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl" Oct 30 13:02:23.904008 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:23.903776 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c29858c0-d396-4092-9e19-b76069383a82-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cxwwl\" (UID: \"c29858c0-d396-4092-9e19-b76069383a82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl" Oct 30 13:02:23.904008 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:23.903957 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c29858c0-d396-4092-9e19-b76069383a82-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cxwwl\" (UID: \"c29858c0-d396-4092-9e19-b76069383a82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl" Oct 30 13:02:23.904335 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:23.904011 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c29858c0-d396-4092-9e19-b76069383a82-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cxwwl\" (UID: \"c29858c0-d396-4092-9e19-b76069383a82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl" Oct 30 13:02:23.904335 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:23.904023 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c29858c0-d396-4092-9e19-b76069383a82-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cxwwl\" (UID: \"c29858c0-d396-4092-9e19-b76069383a82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl" Oct 30 13:02:23.904335 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:23.904042 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c29858c0-d396-4092-9e19-b76069383a82-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cxwwl\" (UID: \"c29858c0-d396-4092-9e19-b76069383a82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl" Oct 30 13:02:23.904335 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:23.904046 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29858c0-d396-4092-9e19-b76069383a82-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cxwwl\" (UID: \"c29858c0-d396-4092-9e19-b76069383a82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl" Oct 30 13:02:23.904335 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:23.904300 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c29858c0-d396-4092-9e19-b76069383a82-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cxwwl\" (UID: \"c29858c0-d396-4092-9e19-b76069383a82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl" Oct 30 13:02:23.904596 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:23.904540 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29858c0-d396-4092-9e19-b76069383a82-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cxwwl\" (UID: \"c29858c0-d396-4092-9e19-b76069383a82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl" Oct 30 13:02:23.904766 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:23.904748 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c29858c0-d396-4092-9e19-b76069383a82-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cxwwl\" (UID: \"c29858c0-d396-4092-9e19-b76069383a82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl" Oct 30 13:02:23.906594 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:23.906573 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c29858c0-d396-4092-9e19-b76069383a82-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cxwwl\" (UID: \"c29858c0-d396-4092-9e19-b76069383a82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl" Oct 30 13:02:23.906795 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:23.906772 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c29858c0-d396-4092-9e19-b76069383a82-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cxwwl\" (UID: \"c29858c0-d396-4092-9e19-b76069383a82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl" Oct 30 13:02:23.943692 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:23.943663 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c29858c0-d396-4092-9e19-b76069383a82-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cxwwl\" (UID: \"c29858c0-d396-4092-9e19-b76069383a82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl" Oct 30 13:02:23.943792 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:23.943776 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d84w4\" (UniqueName: \"kubernetes.io/projected/c29858c0-d396-4092-9e19-b76069383a82-kube-api-access-d84w4\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cxwwl\" (UID: \"c29858c0-d396-4092-9e19-b76069383a82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl" Oct 30 13:02:24.007164 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:24.007132 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl" Oct 30 13:02:24.322922 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:24.322855 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl"] Oct 30 13:02:24.326159 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:02:24.326133 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc29858c0_d396_4092_9e19_b76069383a82.slice/crio-961c0c1fe0759de65bda20e2c7f3977c3a7970d259599784d6b63dd2a380e167 WatchSource:0}: Error finding container 961c0c1fe0759de65bda20e2c7f3977c3a7970d259599784d6b63dd2a380e167: Status 404 returned error can't find the container with id 961c0c1fe0759de65bda20e2c7f3977c3a7970d259599784d6b63dd2a380e167 Oct 30 13:02:24.328297 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:24.328267 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:02:24.328382 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:24.328329 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:02:24.328382 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:24.328359 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:02:24.887659 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:24.882484 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:25.049343 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:25.049309 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:25.091381 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:25.091345 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl" event={"ID":"c29858c0-d396-4092-9e19-b76069383a82","Type":"ContainerStarted","Data":"a104f46db5f933237ee5694be98eef80b20ef0b88517207264f15ccf42f97ca5"} Oct 30 13:02:25.091578 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:25.091396 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl" event={"ID":"c29858c0-d396-4092-9e19-b76069383a82","Type":"ContainerStarted","Data":"961c0c1fe0759de65bda20e2c7f3977c3a7970d259599784d6b63dd2a380e167"} Oct 30 13:02:25.140456 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:25.140369 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl" podStartSLOduration=2.140356525 podStartE2EDuration="2.140356525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:25.139186539 +0000 UTC m=+601.608602286" watchObservedRunningTime="2026-10-30 13:02:25.140356525 +0000 UTC m=+601.609772271" Oct 30 13:02:26.007995 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:26.007962 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl" Oct 30 13:02:26.009369 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:26.009348 5922 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-cxwwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 13:02:26.009428 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:26.009405 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl" podUID="c29858c0-d396-4092-9e19-b76069383a82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 13:02:26.387410 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:26.387319 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:26.537966 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:26.537931 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:27.008151 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:27.008120 5922 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-cxwwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 13:02:27.008325 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:27.008171 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl" podUID="c29858c0-d396-4092-9e19-b76069383a82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 13:02:28.008019 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:28.007979 5922 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-cxwwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 13:02:28.008472 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:28.008037 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl" podUID="c29858c0-d396-4092-9e19-b76069383a82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 13:02:29.007886 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:29.007851 5922 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-cxwwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 13:02:29.008075 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:29.007919 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl" podUID="c29858c0-d396-4092-9e19-b76069383a82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 13:02:30.007528 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:30.007488 5922 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-cxwwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 13:02:30.007754 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:30.007545 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl" podUID="c29858c0-d396-4092-9e19-b76069383a82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 13:02:31.007529 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:31.007495 5922 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-cxwwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 13:02:31.008044 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:31.007548 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl" podUID="c29858c0-d396-4092-9e19-b76069383a82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 13:02:31.999161 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:31.999123 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:32.567013 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:32.566981 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl" Oct 30 13:02:32.567482 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:32.567360 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl" Oct 30 13:02:32.568261 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:32.568240 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl" Oct 30 13:02:48.156878 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:48.156840 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:48.265506 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:48.265468 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:48.609847 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:48.605694 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47"] Oct 30 13:02:48.616494 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:48.616472 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" Oct 30 13:02:48.632508 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:48.632477 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47"] Oct 30 13:02:48.643008 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:48.642986 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-484607-openshift-default-dockercfg-ggcps\"" Oct 30 13:02:48.656122 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:48.656096 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:48.723428 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:48.723400 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78zlh\" (UniqueName: \"kubernetes.io/projected/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-kube-api-access-78zlh\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-pdf47\" (UID: \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" Oct 30 13:02:48.723549 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:48.723439 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-istiod-ca-cert\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-pdf47\" (UID: \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" Oct 30 13:02:48.723549 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:48.723472 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-istio-envoy\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-pdf47\" (UID: \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" Oct 30 13:02:48.723679 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:48.723634 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-istio-podinfo\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-pdf47\" (UID: \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" Oct 30 13:02:48.723742 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:48.723723 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-credential-socket\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-pdf47\" (UID: \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" Oct 30 13:02:48.723797 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:48.723775 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-istio-token\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-pdf47\" (UID: \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" Oct 30 13:02:48.723850 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:48.723827 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-workload-socket\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-pdf47\" (UID: \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" Oct 30 13:02:48.723914 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:48.723895 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-workload-certs\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-pdf47\" (UID: \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" Oct 30 13:02:48.723966 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:48.723928 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-istio-data\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-pdf47\" (UID: \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" Oct 30 13:02:48.824488 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:48.824459 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-istio-podinfo\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-pdf47\" (UID: \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" Oct 30 13:02:48.824603 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:48.824500 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-credential-socket\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-pdf47\" (UID: \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" Oct 30 13:02:48.824603 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:48.824536 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-istio-token\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-pdf47\" (UID: \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" Oct 30 13:02:48.824603 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:48.824579 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-workload-socket\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-pdf47\" (UID: \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" Oct 30 13:02:48.824805 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:48.824661 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-workload-certs\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-pdf47\" (UID: \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" Oct 30 13:02:48.824805 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:48.824687 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-istio-data\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-pdf47\" (UID: \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" Oct 30 13:02:48.824805 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:48.824724 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78zlh\" (UniqueName: \"kubernetes.io/projected/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-kube-api-access-78zlh\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-pdf47\" (UID: \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" Oct 30 13:02:48.824805 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:48.824757 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-istiod-ca-cert\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-pdf47\" (UID: \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" Oct 30 13:02:48.824805 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:48.824789 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-istio-envoy\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-pdf47\" (UID: \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" Oct 30 13:02:48.825057 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:48.824951 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-credential-socket\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-pdf47\" (UID: \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" Oct 30 13:02:48.825140 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:48.825108 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-workload-socket\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-pdf47\" (UID: \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" Oct 30 13:02:48.825197 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:48.825118 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-workload-certs\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-pdf47\" (UID: \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" Oct 30 13:02:48.825277 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:48.825260 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-istio-data\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-pdf47\" (UID: \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" Oct 30 13:02:48.825517 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:48.825495 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-istiod-ca-cert\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-pdf47\" (UID: \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" Oct 30 13:02:48.826934 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:48.826913 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-istio-envoy\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-pdf47\" (UID: \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" Oct 30 13:02:48.827044 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:48.827027 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-istio-podinfo\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-pdf47\" (UID: \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" Oct 30 13:02:48.859603 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:48.859575 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78zlh\" (UniqueName: \"kubernetes.io/projected/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-kube-api-access-78zlh\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-pdf47\" (UID: \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" Oct 30 13:02:48.865207 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:48.865158 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-istio-token\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-pdf47\" (UID: \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" Oct 30 13:02:48.882284 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:48.882259 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:48.929121 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:48.929095 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" Oct 30 13:02:48.999787 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:48.999756 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k"] Oct 30 13:02:49.004349 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.004324 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" Oct 30 13:02:49.007921 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.007893 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:49.034581 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.034499 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-175831-openshift-default-dockercfg-whpw8\"" Oct 30 13:02:49.036586 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.036559 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k"] Oct 30 13:02:49.119815 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.119741 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:49.127980 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.127954 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aced22f0-8827-4cad-b04b-85ddb56490af-credential-socket\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k\" (UID: \"aced22f0-8827-4cad-b04b-85ddb56490af\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" Oct 30 13:02:49.128095 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.128076 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7mrp\" (UniqueName: \"kubernetes.io/projected/aced22f0-8827-4cad-b04b-85ddb56490af-kube-api-access-r7mrp\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k\" (UID: \"aced22f0-8827-4cad-b04b-85ddb56490af\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" Oct 30 13:02:49.128148 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.128126 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aced22f0-8827-4cad-b04b-85ddb56490af-istio-envoy\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k\" (UID: \"aced22f0-8827-4cad-b04b-85ddb56490af\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" Oct 30 13:02:49.128191 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.128151 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aced22f0-8827-4cad-b04b-85ddb56490af-istio-podinfo\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k\" (UID: \"aced22f0-8827-4cad-b04b-85ddb56490af\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" Oct 30 13:02:49.128191 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.128181 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aced22f0-8827-4cad-b04b-85ddb56490af-istio-token\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k\" (UID: \"aced22f0-8827-4cad-b04b-85ddb56490af\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" Oct 30 13:02:49.128294 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.128275 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aced22f0-8827-4cad-b04b-85ddb56490af-istio-data\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k\" (UID: \"aced22f0-8827-4cad-b04b-85ddb56490af\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" Oct 30 13:02:49.128353 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.128334 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aced22f0-8827-4cad-b04b-85ddb56490af-workload-certs\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k\" (UID: \"aced22f0-8827-4cad-b04b-85ddb56490af\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" Oct 30 13:02:49.128428 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.128412 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aced22f0-8827-4cad-b04b-85ddb56490af-workload-socket\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k\" (UID: \"aced22f0-8827-4cad-b04b-85ddb56490af\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" Oct 30 13:02:49.128483 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.128444 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aced22f0-8827-4cad-b04b-85ddb56490af-istiod-ca-cert\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k\" (UID: \"aced22f0-8827-4cad-b04b-85ddb56490af\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" Oct 30 13:02:49.229669 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.229605 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aced22f0-8827-4cad-b04b-85ddb56490af-credential-socket\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k\" (UID: \"aced22f0-8827-4cad-b04b-85ddb56490af\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" Oct 30 13:02:49.230112 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.229710 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7mrp\" (UniqueName: \"kubernetes.io/projected/aced22f0-8827-4cad-b04b-85ddb56490af-kube-api-access-r7mrp\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k\" (UID: \"aced22f0-8827-4cad-b04b-85ddb56490af\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" Oct 30 13:02:49.230112 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.229738 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aced22f0-8827-4cad-b04b-85ddb56490af-istio-envoy\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k\" (UID: \"aced22f0-8827-4cad-b04b-85ddb56490af\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" Oct 30 13:02:49.230112 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.229757 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aced22f0-8827-4cad-b04b-85ddb56490af-istio-podinfo\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k\" (UID: \"aced22f0-8827-4cad-b04b-85ddb56490af\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" Oct 30 13:02:49.230112 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.229786 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aced22f0-8827-4cad-b04b-85ddb56490af-istio-token\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k\" (UID: \"aced22f0-8827-4cad-b04b-85ddb56490af\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" Oct 30 13:02:49.230112 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.229806 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aced22f0-8827-4cad-b04b-85ddb56490af-istio-data\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k\" (UID: \"aced22f0-8827-4cad-b04b-85ddb56490af\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" Oct 30 13:02:49.230112 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.229823 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aced22f0-8827-4cad-b04b-85ddb56490af-workload-certs\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k\" (UID: \"aced22f0-8827-4cad-b04b-85ddb56490af\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" Oct 30 13:02:49.230112 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.229936 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aced22f0-8827-4cad-b04b-85ddb56490af-workload-socket\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k\" (UID: \"aced22f0-8827-4cad-b04b-85ddb56490af\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" Oct 30 13:02:49.230112 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.229967 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aced22f0-8827-4cad-b04b-85ddb56490af-istiod-ca-cert\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k\" (UID: \"aced22f0-8827-4cad-b04b-85ddb56490af\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" Oct 30 13:02:49.230501 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.230223 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aced22f0-8827-4cad-b04b-85ddb56490af-istio-data\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k\" (UID: \"aced22f0-8827-4cad-b04b-85ddb56490af\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" Oct 30 13:02:49.230501 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.230314 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aced22f0-8827-4cad-b04b-85ddb56490af-credential-socket\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k\" (UID: \"aced22f0-8827-4cad-b04b-85ddb56490af\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" Oct 30 13:02:49.230501 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.230385 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aced22f0-8827-4cad-b04b-85ddb56490af-workload-socket\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k\" (UID: \"aced22f0-8827-4cad-b04b-85ddb56490af\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" Oct 30 13:02:49.230825 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.230791 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aced22f0-8827-4cad-b04b-85ddb56490af-istiod-ca-cert\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k\" (UID: \"aced22f0-8827-4cad-b04b-85ddb56490af\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" Oct 30 13:02:49.230933 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.230911 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aced22f0-8827-4cad-b04b-85ddb56490af-workload-certs\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k\" (UID: \"aced22f0-8827-4cad-b04b-85ddb56490af\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" Oct 30 13:02:49.232211 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.232186 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aced22f0-8827-4cad-b04b-85ddb56490af-istio-envoy\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k\" (UID: \"aced22f0-8827-4cad-b04b-85ddb56490af\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" Oct 30 13:02:49.232282 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.232269 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aced22f0-8827-4cad-b04b-85ddb56490af-istio-podinfo\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k\" (UID: \"aced22f0-8827-4cad-b04b-85ddb56490af\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" Oct 30 13:02:49.236975 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.236955 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:49.245454 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.245417 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47"] Oct 30 13:02:49.248974 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:02:49.248952 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd62b6ccd_e8a1_4320_87d0_0b12377ad8ee.slice/crio-57a86d45e02c20e3126d7de12b1bce5bf2d0c67c254ba69c2d540ff75e68af83 WatchSource:0}: Error finding container 57a86d45e02c20e3126d7de12b1bce5bf2d0c67c254ba69c2d540ff75e68af83: Status 404 returned error can't find the container with id 57a86d45e02c20e3126d7de12b1bce5bf2d0c67c254ba69c2d540ff75e68af83 Oct 30 13:02:49.251035 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.251005 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:02:49.251111 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.251080 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:02:49.251150 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.251110 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:02:49.254916 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.254896 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aced22f0-8827-4cad-b04b-85ddb56490af-istio-token\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k\" (UID: \"aced22f0-8827-4cad-b04b-85ddb56490af\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" Oct 30 13:02:49.255125 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.255103 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7mrp\" (UniqueName: \"kubernetes.io/projected/aced22f0-8827-4cad-b04b-85ddb56490af-kube-api-access-r7mrp\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k\" (UID: \"aced22f0-8827-4cad-b04b-85ddb56490af\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" Oct 30 13:02:49.284188 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.284164 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b"] Oct 30 13:02:49.288021 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.288004 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" Oct 30 13:02:49.313339 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.313302 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b"] Oct 30 13:02:49.319280 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.319260 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" Oct 30 13:02:49.350039 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.350006 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:49.361504 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.360749 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-3048f9-openshift-default-dockercfg-b5fdc\"" Oct 30 13:02:49.445716 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.443817 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8"] Oct 30 13:02:49.459670 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.459630 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8"] Oct 30 13:02:49.459826 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.459701 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" Oct 30 13:02:49.460124 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.460103 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecdab1d-a927-4d7a-b270-ba7908c88d52-credential-socket\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b\" (UID: \"cecdab1d-a927-4d7a-b270-ba7908c88d52\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" Oct 30 13:02:49.460262 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.460245 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbb5n\" (UniqueName: \"kubernetes.io/projected/cecdab1d-a927-4d7a-b270-ba7908c88d52-kube-api-access-vbb5n\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b\" (UID: \"cecdab1d-a927-4d7a-b270-ba7908c88d52\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" Oct 30 13:02:49.460381 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.460366 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cecdab1d-a927-4d7a-b270-ba7908c88d52-istiod-ca-cert\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b\" (UID: \"cecdab1d-a927-4d7a-b270-ba7908c88d52\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" Oct 30 13:02:49.465241 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.460530 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cecdab1d-a927-4d7a-b270-ba7908c88d52-istio-podinfo\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b\" (UID: \"cecdab1d-a927-4d7a-b270-ba7908c88d52\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" Oct 30 13:02:49.465417 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.465403 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cecdab1d-a927-4d7a-b270-ba7908c88d52-istio-token\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b\" (UID: \"cecdab1d-a927-4d7a-b270-ba7908c88d52\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" Oct 30 13:02:49.465509 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.465499 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cecdab1d-a927-4d7a-b270-ba7908c88d52-istio-data\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b\" (UID: \"cecdab1d-a927-4d7a-b270-ba7908c88d52\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" Oct 30 13:02:49.465657 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.465641 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecdab1d-a927-4d7a-b270-ba7908c88d52-workload-socket\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b\" (UID: \"cecdab1d-a927-4d7a-b270-ba7908c88d52\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" Oct 30 13:02:49.465808 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.465797 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cecdab1d-a927-4d7a-b270-ba7908c88d52-workload-certs\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b\" (UID: \"cecdab1d-a927-4d7a-b270-ba7908c88d52\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" Oct 30 13:02:49.468401 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.465921 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cecdab1d-a927-4d7a-b270-ba7908c88d52-istio-envoy\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b\" (UID: \"cecdab1d-a927-4d7a-b270-ba7908c88d52\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" Oct 30 13:02:49.476237 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.476213 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:49.476493 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.476476 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-bb71e2-openshift-default-dockercfg-qmj6k\"" Oct 30 13:02:49.566582 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.566536 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecdab1d-a927-4d7a-b270-ba7908c88d52-workload-socket\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b\" (UID: \"cecdab1d-a927-4d7a-b270-ba7908c88d52\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" Oct 30 13:02:49.566840 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.566648 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a68a48f7-d090-4eb7-af40-df304a84b319-istio-data\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8\" (UID: \"a68a48f7-d090-4eb7-af40-df304a84b319\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" Oct 30 13:02:49.566840 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.566722 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cecdab1d-a927-4d7a-b270-ba7908c88d52-workload-certs\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b\" (UID: \"cecdab1d-a927-4d7a-b270-ba7908c88d52\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" Oct 30 13:02:49.566840 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.566756 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cecdab1d-a927-4d7a-b270-ba7908c88d52-istio-envoy\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b\" (UID: \"cecdab1d-a927-4d7a-b270-ba7908c88d52\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" Oct 30 13:02:49.566840 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.566787 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a68a48f7-d090-4eb7-af40-df304a84b319-workload-certs\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8\" (UID: \"a68a48f7-d090-4eb7-af40-df304a84b319\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" Oct 30 13:02:49.566840 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.566819 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecdab1d-a927-4d7a-b270-ba7908c88d52-credential-socket\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b\" (UID: \"cecdab1d-a927-4d7a-b270-ba7908c88d52\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" Oct 30 13:02:49.567177 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.566867 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbb5n\" (UniqueName: \"kubernetes.io/projected/cecdab1d-a927-4d7a-b270-ba7908c88d52-kube-api-access-vbb5n\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b\" (UID: \"cecdab1d-a927-4d7a-b270-ba7908c88d52\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" Oct 30 13:02:49.567177 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.566912 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cecdab1d-a927-4d7a-b270-ba7908c88d52-istiod-ca-cert\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b\" (UID: \"cecdab1d-a927-4d7a-b270-ba7908c88d52\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" Oct 30 13:02:49.567177 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.566957 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a68a48f7-d090-4eb7-af40-df304a84b319-credential-socket\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8\" (UID: \"a68a48f7-d090-4eb7-af40-df304a84b319\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" Oct 30 13:02:49.567177 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.567005 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cecdab1d-a927-4d7a-b270-ba7908c88d52-istio-podinfo\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b\" (UID: \"cecdab1d-a927-4d7a-b270-ba7908c88d52\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" Oct 30 13:02:49.567177 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.567057 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqbv7\" (UniqueName: \"kubernetes.io/projected/a68a48f7-d090-4eb7-af40-df304a84b319-kube-api-access-rqbv7\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8\" (UID: \"a68a48f7-d090-4eb7-af40-df304a84b319\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" Oct 30 13:02:49.567177 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.567083 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecdab1d-a927-4d7a-b270-ba7908c88d52-workload-socket\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b\" (UID: \"cecdab1d-a927-4d7a-b270-ba7908c88d52\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" Oct 30 13:02:49.567177 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.567140 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a68a48f7-d090-4eb7-af40-df304a84b319-istio-podinfo\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8\" (UID: \"a68a48f7-d090-4eb7-af40-df304a84b319\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" Oct 30 13:02:49.567522 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.567178 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cecdab1d-a927-4d7a-b270-ba7908c88d52-workload-certs\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b\" (UID: \"cecdab1d-a927-4d7a-b270-ba7908c88d52\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" Oct 30 13:02:49.567522 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.567208 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cecdab1d-a927-4d7a-b270-ba7908c88d52-istio-token\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b\" (UID: \"cecdab1d-a927-4d7a-b270-ba7908c88d52\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" Oct 30 13:02:49.567522 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.567335 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a68a48f7-d090-4eb7-af40-df304a84b319-istio-token\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8\" (UID: \"a68a48f7-d090-4eb7-af40-df304a84b319\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" Oct 30 13:02:49.567522 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.567419 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a68a48f7-d090-4eb7-af40-df304a84b319-istiod-ca-cert\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8\" (UID: \"a68a48f7-d090-4eb7-af40-df304a84b319\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" Oct 30 13:02:49.567522 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.567516 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cecdab1d-a927-4d7a-b270-ba7908c88d52-istio-data\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b\" (UID: \"cecdab1d-a927-4d7a-b270-ba7908c88d52\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" Oct 30 13:02:49.567812 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.567549 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a68a48f7-d090-4eb7-af40-df304a84b319-istio-envoy\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8\" (UID: \"a68a48f7-d090-4eb7-af40-df304a84b319\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" Oct 30 13:02:49.568051 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.568019 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a68a48f7-d090-4eb7-af40-df304a84b319-workload-socket\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8\" (UID: \"a68a48f7-d090-4eb7-af40-df304a84b319\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" Oct 30 13:02:49.568179 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.568148 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cecdab1d-a927-4d7a-b270-ba7908c88d52-istio-data\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b\" (UID: \"cecdab1d-a927-4d7a-b270-ba7908c88d52\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" Oct 30 13:02:49.568286 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.568258 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cecdab1d-a927-4d7a-b270-ba7908c88d52-istiod-ca-cert\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b\" (UID: \"cecdab1d-a927-4d7a-b270-ba7908c88d52\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" Oct 30 13:02:49.568484 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.568464 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecdab1d-a927-4d7a-b270-ba7908c88d52-credential-socket\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b\" (UID: \"cecdab1d-a927-4d7a-b270-ba7908c88d52\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" Oct 30 13:02:49.570408 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.570384 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cecdab1d-a927-4d7a-b270-ba7908c88d52-istio-podinfo\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b\" (UID: \"cecdab1d-a927-4d7a-b270-ba7908c88d52\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" Oct 30 13:02:49.570515 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.570475 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cecdab1d-a927-4d7a-b270-ba7908c88d52-istio-envoy\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b\" (UID: \"cecdab1d-a927-4d7a-b270-ba7908c88d52\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" Oct 30 13:02:49.603139 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.603112 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbb5n\" (UniqueName: \"kubernetes.io/projected/cecdab1d-a927-4d7a-b270-ba7908c88d52-kube-api-access-vbb5n\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b\" (UID: \"cecdab1d-a927-4d7a-b270-ba7908c88d52\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" Oct 30 13:02:49.603951 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.603928 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cecdab1d-a927-4d7a-b270-ba7908c88d52-istio-token\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b\" (UID: \"cecdab1d-a927-4d7a-b270-ba7908c88d52\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" Oct 30 13:02:49.610258 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.610219 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:49.610718 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.610696 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" Oct 30 13:02:49.646372 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.646291 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k"] Oct 30 13:02:49.652277 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:02:49.652248 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaced22f0_8827_4cad_b04b_85ddb56490af.slice/crio-bafb68be112012c37e6e86b30c59964d6f555e02150bbfde796f4be7a44323db WatchSource:0}: Error finding container bafb68be112012c37e6e86b30c59964d6f555e02150bbfde796f4be7a44323db: Status 404 returned error can't find the container with id bafb68be112012c37e6e86b30c59964d6f555e02150bbfde796f4be7a44323db Oct 30 13:02:49.654733 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.654698 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:02:49.654836 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.654777 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:02:49.654836 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.654826 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:02:49.668711 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.668684 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a68a48f7-d090-4eb7-af40-df304a84b319-workload-certs\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8\" (UID: \"a68a48f7-d090-4eb7-af40-df304a84b319\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" Oct 30 13:02:49.668825 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.668747 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a68a48f7-d090-4eb7-af40-df304a84b319-credential-socket\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8\" (UID: \"a68a48f7-d090-4eb7-af40-df304a84b319\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" Oct 30 13:02:49.668825 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.668813 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqbv7\" (UniqueName: \"kubernetes.io/projected/a68a48f7-d090-4eb7-af40-df304a84b319-kube-api-access-rqbv7\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8\" (UID: \"a68a48f7-d090-4eb7-af40-df304a84b319\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" Oct 30 13:02:49.668961 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.668860 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a68a48f7-d090-4eb7-af40-df304a84b319-istio-podinfo\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8\" (UID: \"a68a48f7-d090-4eb7-af40-df304a84b319\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" Oct 30 13:02:49.668961 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.668892 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a68a48f7-d090-4eb7-af40-df304a84b319-istio-token\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8\" (UID: \"a68a48f7-d090-4eb7-af40-df304a84b319\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" Oct 30 13:02:49.668961 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.668920 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a68a48f7-d090-4eb7-af40-df304a84b319-istiod-ca-cert\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8\" (UID: \"a68a48f7-d090-4eb7-af40-df304a84b319\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" Oct 30 13:02:49.668961 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.668959 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a68a48f7-d090-4eb7-af40-df304a84b319-istio-envoy\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8\" (UID: \"a68a48f7-d090-4eb7-af40-df304a84b319\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" Oct 30 13:02:49.669169 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.669005 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a68a48f7-d090-4eb7-af40-df304a84b319-workload-socket\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8\" (UID: \"a68a48f7-d090-4eb7-af40-df304a84b319\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" Oct 30 13:02:49.669169 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.669076 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a68a48f7-d090-4eb7-af40-df304a84b319-istio-data\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8\" (UID: \"a68a48f7-d090-4eb7-af40-df304a84b319\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" Oct 30 13:02:49.670000 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.669975 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a68a48f7-d090-4eb7-af40-df304a84b319-istio-data\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8\" (UID: \"a68a48f7-d090-4eb7-af40-df304a84b319\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" Oct 30 13:02:49.670783 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.670758 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a68a48f7-d090-4eb7-af40-df304a84b319-credential-socket\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8\" (UID: \"a68a48f7-d090-4eb7-af40-df304a84b319\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" Oct 30 13:02:49.671196 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.671174 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a68a48f7-d090-4eb7-af40-df304a84b319-workload-certs\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8\" (UID: \"a68a48f7-d090-4eb7-af40-df304a84b319\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" Oct 30 13:02:49.671316 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.671296 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a68a48f7-d090-4eb7-af40-df304a84b319-workload-socket\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8\" (UID: \"a68a48f7-d090-4eb7-af40-df304a84b319\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" Oct 30 13:02:49.671532 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.671505 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a68a48f7-d090-4eb7-af40-df304a84b319-istiod-ca-cert\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8\" (UID: \"a68a48f7-d090-4eb7-af40-df304a84b319\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" Oct 30 13:02:49.676362 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.676339 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a68a48f7-d090-4eb7-af40-df304a84b319-istio-envoy\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8\" (UID: \"a68a48f7-d090-4eb7-af40-df304a84b319\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" Oct 30 13:02:49.686529 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.686507 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a68a48f7-d090-4eb7-af40-df304a84b319-istio-podinfo\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8\" (UID: \"a68a48f7-d090-4eb7-af40-df304a84b319\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" Oct 30 13:02:49.696030 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.695993 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a68a48f7-d090-4eb7-af40-df304a84b319-istio-token\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8\" (UID: \"a68a48f7-d090-4eb7-af40-df304a84b319\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" Oct 30 13:02:49.703600 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.703573 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqbv7\" (UniqueName: \"kubernetes.io/projected/a68a48f7-d090-4eb7-af40-df304a84b319-kube-api-access-rqbv7\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8\" (UID: \"a68a48f7-d090-4eb7-af40-df304a84b319\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" Oct 30 13:02:49.779840 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.779810 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" Oct 30 13:02:49.783321 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.783289 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:49.937196 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.937110 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b"] Oct 30 13:02:49.944054 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:02:49.944018 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcecdab1d_a927_4d7a_b270_ba7908c88d52.slice/crio-4ffef3b1a64685463f41f47001fec79cc78d2bd36a2f6d0c2d2cdf2235938383 WatchSource:0}: Error finding container 4ffef3b1a64685463f41f47001fec79cc78d2bd36a2f6d0c2d2cdf2235938383: Status 404 returned error can't find the container with id 4ffef3b1a64685463f41f47001fec79cc78d2bd36a2f6d0c2d2cdf2235938383 Oct 30 13:02:49.947225 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.947184 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:02:49.947435 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.947403 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:02:49.947568 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.947462 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:02:49.948768 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:49.948713 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:50.091292 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:50.091255 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8"] Oct 30 13:02:50.100470 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:50.100433 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:02:50.100576 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:50.100519 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:02:50.100576 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:50.100557 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:02:50.211489 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:50.211452 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" event={"ID":"aced22f0-8827-4cad-b04b-85ddb56490af","Type":"ContainerStarted","Data":"622569e4d5eaebd803a4ee6adfb75551939cb04face002ebb9dd9a900633ff33"} Oct 30 13:02:50.211686 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:50.211518 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" event={"ID":"aced22f0-8827-4cad-b04b-85ddb56490af","Type":"ContainerStarted","Data":"bafb68be112012c37e6e86b30c59964d6f555e02150bbfde796f4be7a44323db"} Oct 30 13:02:50.213642 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:50.213422 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" event={"ID":"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee","Type":"ContainerStarted","Data":"5af4f5e7c7ee7cea6851d1ac6622767bda512d8d889d9f19d691bf996d53c70f"} Oct 30 13:02:50.213642 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:50.213478 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" event={"ID":"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee","Type":"ContainerStarted","Data":"57a86d45e02c20e3126d7de12b1bce5bf2d0c67c254ba69c2d540ff75e68af83"} Oct 30 13:02:50.215293 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:50.215256 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" event={"ID":"a68a48f7-d090-4eb7-af40-df304a84b319","Type":"ContainerStarted","Data":"933f6ee0846072e68eb36ab006c8cc87cea85081333069c8fb6816d7f3650b59"} Oct 30 13:02:50.215363 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:50.215310 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" event={"ID":"a68a48f7-d090-4eb7-af40-df304a84b319","Type":"ContainerStarted","Data":"35ec412147d64b21fd80049f3fbea856135a8b313e2235dedd48bc1c3f0bb5c7"} Oct 30 13:02:50.217710 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:50.217686 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" event={"ID":"cecdab1d-a927-4d7a-b270-ba7908c88d52","Type":"ContainerStarted","Data":"39fe7c325a40de5d43ff7303c8776ba7b385000c66965a74ea286879561694a6"} Oct 30 13:02:50.217826 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:50.217731 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" event={"ID":"cecdab1d-a927-4d7a-b270-ba7908c88d52","Type":"ContainerStarted","Data":"4ffef3b1a64685463f41f47001fec79cc78d2bd36a2f6d0c2d2cdf2235938383"} Oct 30 13:02:50.261384 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:50.261340 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" podStartSLOduration=2.261322503 podStartE2EDuration="2.261322503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:50.258035616 +0000 UTC m=+626.727451384" watchObservedRunningTime="2026-10-30 13:02:50.261322503 +0000 UTC m=+626.730738249" Oct 30 13:02:50.310116 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:50.310065 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" podStartSLOduration=1.310051066 podStartE2EDuration="1.310051066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:50.309770004 +0000 UTC m=+626.779185797" watchObservedRunningTime="2026-10-30 13:02:50.310051066 +0000 UTC m=+626.779466805" Oct 30 13:02:50.320002 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:50.319972 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" Oct 30 13:02:50.325162 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:50.325140 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" Oct 30 13:02:50.356919 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:50.356864 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" podStartSLOduration=1.356845833 podStartE2EDuration="1.356845833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:50.355516303 +0000 UTC m=+626.824932100" watchObservedRunningTime="2026-10-30 13:02:50.356845833 +0000 UTC m=+626.826261579" Oct 30 13:02:50.404029 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:50.403984 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" podStartSLOduration=2.403970922 podStartE2EDuration="2.403970922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:50.403454264 +0000 UTC m=+626.872870057" watchObservedRunningTime="2026-10-30 13:02:50.403970922 +0000 UTC m=+626.873386667" Oct 30 13:02:50.611520 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:50.611450 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" Oct 30 13:02:50.780777 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:50.780749 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" Oct 30 13:02:50.929641 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:50.929553 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" Oct 30 13:02:50.934423 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:50.934398 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" Oct 30 13:02:51.222757 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:51.222727 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" Oct 30 13:02:51.222949 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:51.222778 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" Oct 30 13:02:51.223567 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:51.223536 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" Oct 30 13:02:51.223711 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:51.223603 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" Oct 30 13:02:51.617074 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:51.617000 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" Oct 30 13:02:51.785966 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:51.785937 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" Oct 30 13:02:52.227532 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:52.227496 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" Oct 30 13:02:52.227705 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:52.227554 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" Oct 30 13:02:52.228066 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:52.228044 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" Oct 30 13:02:52.228271 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:52.228255 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" Oct 30 13:02:52.910573 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:52.910538 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:53.030157 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:53.030120 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:53.154659 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:53.154626 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:53.280246 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:53.280208 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:53.394630 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:53.394583 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:53.553478 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:02:53.553397 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:02:53.592016 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:53.591989 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:53.733288 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:53.733254 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:53.996662 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:53.996631 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:54.158503 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:54.158465 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:54.279948 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:54.279843 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:54.632509 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:54.632432 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:54.757426 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:54.757390 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:54.872947 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:54.872905 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:54.995880 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:54.995847 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:55.110505 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:55.110472 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:55.110903 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:55.110807 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-3048f9-fc55696f5-j6t4d"] Oct 30 13:02:55.113483 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:55.113467 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3048f9-fc55696f5-j6t4d" Oct 30 13:02:55.132251 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:55.132231 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-3048f9-fc55696f5-j6t4d"] Oct 30 13:02:55.142853 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:55.142828 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-3048f9-dockercfg-szzbs\"" Oct 30 13:02:55.142853 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:55.142828 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-3048f9\"" Oct 30 13:02:55.227285 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:55.227242 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfr42\" (UniqueName: \"kubernetes.io/projected/68a59f39-95e5-4e6c-b589-bf0cb560a12c-kube-api-access-bfr42\") pod \"payload-pre-processing-e2e-worker-w3-3048f9-fc55696f5-j6t4d\" (UID: \"68a59f39-95e5-4e6c-b589-bf0cb560a12c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3048f9-fc55696f5-j6t4d" Oct 30 13:02:55.227468 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:55.227373 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68a59f39-95e5-4e6c-b589-bf0cb560a12c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-3048f9-fc55696f5-j6t4d\" (UID: \"68a59f39-95e5-4e6c-b589-bf0cb560a12c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3048f9-fc55696f5-j6t4d" Oct 30 13:02:55.328070 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:55.328003 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68a59f39-95e5-4e6c-b589-bf0cb560a12c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-3048f9-fc55696f5-j6t4d\" (UID: \"68a59f39-95e5-4e6c-b589-bf0cb560a12c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3048f9-fc55696f5-j6t4d" Oct 30 13:02:55.328204 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:55.328163 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfr42\" (UniqueName: \"kubernetes.io/projected/68a59f39-95e5-4e6c-b589-bf0cb560a12c-kube-api-access-bfr42\") pod \"payload-pre-processing-e2e-worker-w3-3048f9-fc55696f5-j6t4d\" (UID: \"68a59f39-95e5-4e6c-b589-bf0cb560a12c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3048f9-fc55696f5-j6t4d" Oct 30 13:02:55.328828 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:55.328803 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68a59f39-95e5-4e6c-b589-bf0cb560a12c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-3048f9-fc55696f5-j6t4d\" (UID: \"68a59f39-95e5-4e6c-b589-bf0cb560a12c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3048f9-fc55696f5-j6t4d" Oct 30 13:02:55.352762 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:55.352735 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfr42\" (UniqueName: \"kubernetes.io/projected/68a59f39-95e5-4e6c-b589-bf0cb560a12c-kube-api-access-bfr42\") pod \"payload-pre-processing-e2e-worker-w3-3048f9-fc55696f5-j6t4d\" (UID: \"68a59f39-95e5-4e6c-b589-bf0cb560a12c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3048f9-fc55696f5-j6t4d" Oct 30 13:02:55.425432 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:55.425399 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3048f9-fc55696f5-j6t4d" Oct 30 13:02:55.721076 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:55.721047 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-3048f9-fc55696f5-j6t4d"] Oct 30 13:02:55.724182 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:02:55.724155 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68a59f39_95e5_4e6c_b589_bf0cb560a12c.slice/crio-c46aefdc6d111ed4ecd575dff499ce76c506ba7b6bac1ba399d5830fd2b97b7a WatchSource:0}: Error finding container c46aefdc6d111ed4ecd575dff499ce76c506ba7b6bac1ba399d5830fd2b97b7a: Status 404 returned error can't find the container with id c46aefdc6d111ed4ecd575dff499ce76c506ba7b6bac1ba399d5830fd2b97b7a Oct 30 13:02:56.249129 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:56.249096 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3048f9-fc55696f5-j6t4d" event={"ID":"68a59f39-95e5-4e6c-b589-bf0cb560a12c","Type":"ContainerStarted","Data":"ff02d73c2f9c94e9703af8ddbb47c989cf3b32dfb6f5220da125202b5c940293"} Oct 30 13:02:56.249526 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:56.249144 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3048f9-fc55696f5-j6t4d" event={"ID":"68a59f39-95e5-4e6c-b589-bf0cb560a12c","Type":"ContainerStarted","Data":"c46aefdc6d111ed4ecd575dff499ce76c506ba7b6bac1ba399d5830fd2b97b7a"} Oct 30 13:02:56.249526 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:56.249208 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3048f9-fc55696f5-j6t4d" Oct 30 13:02:56.299886 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:56.299840 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3048f9-fc55696f5-j6t4d" podStartSLOduration=1.299828882 podStartE2EDuration="1.299828882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:56.29935175 +0000 UTC m=+632.768767498" watchObservedRunningTime="2026-10-30 13:02:56.299828882 +0000 UTC m=+632.769244642" Oct 30 13:02:56.704644 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:56.704593 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:56.818670 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:56.818634 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:56.904019 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:56.903992 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-bb71e2-59f79f4f99-xh9lj"] Oct 30 13:02:56.906758 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:56.906744 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bb71e2-59f79f4f99-xh9lj" Oct 30 13:02:56.927476 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:56.927453 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-bb71e2-59f79f4f99-xh9lj"] Oct 30 13:02:56.934927 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:56.934908 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-bb71e2-dockercfg-xxlbt\"" Oct 30 13:02:56.934927 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:56.934924 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-bb71e2\"" Oct 30 13:02:56.943169 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:56.943145 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72a0ed5e-1d31-4695-9dc5-92a304c0dc69-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-bb71e2-59f79f4f99-xh9lj\" (UID: \"72a0ed5e-1d31-4695-9dc5-92a304c0dc69\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bb71e2-59f79f4f99-xh9lj" Oct 30 13:02:56.943292 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:56.943187 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw8rz\" (UniqueName: \"kubernetes.io/projected/72a0ed5e-1d31-4695-9dc5-92a304c0dc69-kube-api-access-cw8rz\") pod \"payload-pre-processing-e2e-worker-w2-bb71e2-59f79f4f99-xh9lj\" (UID: \"72a0ed5e-1d31-4695-9dc5-92a304c0dc69\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bb71e2-59f79f4f99-xh9lj" Oct 30 13:02:57.044749 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:57.044675 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw8rz\" (UniqueName: \"kubernetes.io/projected/72a0ed5e-1d31-4695-9dc5-92a304c0dc69-kube-api-access-cw8rz\") pod \"payload-pre-processing-e2e-worker-w2-bb71e2-59f79f4f99-xh9lj\" (UID: \"72a0ed5e-1d31-4695-9dc5-92a304c0dc69\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bb71e2-59f79f4f99-xh9lj" Oct 30 13:02:57.044911 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:57.044850 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72a0ed5e-1d31-4695-9dc5-92a304c0dc69-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-bb71e2-59f79f4f99-xh9lj\" (UID: \"72a0ed5e-1d31-4695-9dc5-92a304c0dc69\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bb71e2-59f79f4f99-xh9lj" Oct 30 13:02:57.045514 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:57.045486 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72a0ed5e-1d31-4695-9dc5-92a304c0dc69-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-bb71e2-59f79f4f99-xh9lj\" (UID: \"72a0ed5e-1d31-4695-9dc5-92a304c0dc69\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bb71e2-59f79f4f99-xh9lj" Oct 30 13:02:57.070264 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:57.070234 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw8rz\" (UniqueName: \"kubernetes.io/projected/72a0ed5e-1d31-4695-9dc5-92a304c0dc69-kube-api-access-cw8rz\") pod \"payload-pre-processing-e2e-worker-w2-bb71e2-59f79f4f99-xh9lj\" (UID: \"72a0ed5e-1d31-4695-9dc5-92a304c0dc69\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bb71e2-59f79f4f99-xh9lj" Oct 30 13:02:57.218814 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:57.218784 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bb71e2-59f79f4f99-xh9lj" Oct 30 13:02:57.521768 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:57.521740 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-bb71e2-59f79f4f99-xh9lj"] Oct 30 13:02:57.524585 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:02:57.524558 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72a0ed5e_1d31_4695_9dc5_92a304c0dc69.slice/crio-cd3ca241f8386b317f249ae483ef74711a80293ab66e396b6155d4a474c1f175 WatchSource:0}: Error finding container cd3ca241f8386b317f249ae483ef74711a80293ab66e396b6155d4a474c1f175: Status 404 returned error can't find the container with id cd3ca241f8386b317f249ae483ef74711a80293ab66e396b6155d4a474c1f175 Oct 30 13:02:57.693079 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:57.693046 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:57.817752 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:57.817662 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:57.864091 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:57.864064 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-484607-6d646d7567-hd7j6"] Oct 30 13:02:57.867229 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:57.867207 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-484607-6d646d7567-hd7j6" Oct 30 13:02:57.888649 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:57.888624 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-484607\"" Oct 30 13:02:57.889963 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:57.889936 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-484607-6d646d7567-hd7j6"] Oct 30 13:02:57.896587 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:57.896560 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-484607-dockercfg-ps7wr\"" Oct 30 13:02:57.924826 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:57.924800 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:02:57.953868 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:57.953839 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2j7d\" (UniqueName: \"kubernetes.io/projected/62a8727a-ad11-4c04-87e1-0b198af32183-kube-api-access-t2j7d\") pod \"payload-pre-processing-e2e-worker-w6-484607-6d646d7567-hd7j6\" (UID: \"62a8727a-ad11-4c04-87e1-0b198af32183\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-484607-6d646d7567-hd7j6" Oct 30 13:02:57.954060 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:57.954042 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62a8727a-ad11-4c04-87e1-0b198af32183-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-484607-6d646d7567-hd7j6\" (UID: \"62a8727a-ad11-4c04-87e1-0b198af32183\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-484607-6d646d7567-hd7j6" Oct 30 13:02:57.963528 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:57.963505 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-484607-5684dfc6c6-8z79j"] Oct 30 13:02:57.966025 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:57.966010 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-484607-5684dfc6c6-8z79j" Oct 30 13:02:57.986252 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:57.986230 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-484607-5684dfc6c6-8z79j"] Oct 30 13:02:58.054981 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:58.054950 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2j7d\" (UniqueName: \"kubernetes.io/projected/62a8727a-ad11-4c04-87e1-0b198af32183-kube-api-access-t2j7d\") pod \"payload-pre-processing-e2e-worker-w6-484607-6d646d7567-hd7j6\" (UID: \"62a8727a-ad11-4c04-87e1-0b198af32183\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-484607-6d646d7567-hd7j6" Oct 30 13:02:58.055171 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:58.055070 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62a8727a-ad11-4c04-87e1-0b198af32183-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-484607-6d646d7567-hd7j6\" (UID: \"62a8727a-ad11-4c04-87e1-0b198af32183\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-484607-6d646d7567-hd7j6" Oct 30 13:02:58.055171 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:58.055092 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhrdt\" (UniqueName: \"kubernetes.io/projected/82af0f22-cb2d-41c5-b4a8-adcecc306998-kube-api-access-mhrdt\") pod \"payload-processing-e2e-worker-w6-484607-5684dfc6c6-8z79j\" (UID: \"82af0f22-cb2d-41c5-b4a8-adcecc306998\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-484607-5684dfc6c6-8z79j" Oct 30 13:02:58.055171 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:58.055117 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82af0f22-cb2d-41c5-b4a8-adcecc306998-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-484607-5684dfc6c6-8z79j\" (UID: \"82af0f22-cb2d-41c5-b4a8-adcecc306998\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-484607-5684dfc6c6-8z79j" Oct 30 13:02:58.056033 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:58.056004 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62a8727a-ad11-4c04-87e1-0b198af32183-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-484607-6d646d7567-hd7j6\" (UID: \"62a8727a-ad11-4c04-87e1-0b198af32183\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-484607-6d646d7567-hd7j6" Oct 30 13:02:58.084046 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:58.083969 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2j7d\" (UniqueName: \"kubernetes.io/projected/62a8727a-ad11-4c04-87e1-0b198af32183-kube-api-access-t2j7d\") pod \"payload-pre-processing-e2e-worker-w6-484607-6d646d7567-hd7j6\" (UID: \"62a8727a-ad11-4c04-87e1-0b198af32183\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-484607-6d646d7567-hd7j6" Oct 30 13:02:58.156709 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:58.156668 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhrdt\" (UniqueName: \"kubernetes.io/projected/82af0f22-cb2d-41c5-b4a8-adcecc306998-kube-api-access-mhrdt\") pod \"payload-processing-e2e-worker-w6-484607-5684dfc6c6-8z79j\" (UID: \"82af0f22-cb2d-41c5-b4a8-adcecc306998\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-484607-5684dfc6c6-8z79j" Oct 30 13:02:58.156920 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:58.156731 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82af0f22-cb2d-41c5-b4a8-adcecc306998-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-484607-5684dfc6c6-8z79j\" (UID: \"82af0f22-cb2d-41c5-b4a8-adcecc306998\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-484607-5684dfc6c6-8z79j" Oct 30 13:02:58.157778 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:58.157745 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82af0f22-cb2d-41c5-b4a8-adcecc306998-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-484607-5684dfc6c6-8z79j\" (UID: \"82af0f22-cb2d-41c5-b4a8-adcecc306998\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-484607-5684dfc6c6-8z79j" Oct 30 13:02:58.179414 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:58.179385 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-484607-6d646d7567-hd7j6" Oct 30 13:02:58.183572 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:58.183536 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhrdt\" (UniqueName: \"kubernetes.io/projected/82af0f22-cb2d-41c5-b4a8-adcecc306998-kube-api-access-mhrdt\") pod \"payload-processing-e2e-worker-w6-484607-5684dfc6c6-8z79j\" (UID: \"82af0f22-cb2d-41c5-b4a8-adcecc306998\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-484607-5684dfc6c6-8z79j" Oct 30 13:02:58.266002 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:58.265964 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bb71e2-59f79f4f99-xh9lj" event={"ID":"72a0ed5e-1d31-4695-9dc5-92a304c0dc69","Type":"ContainerStarted","Data":"c39f7418cd4472879b43de64fe85b261a2cdd81d409b658f68edd9650c50375f"} Oct 30 13:02:58.266141 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:58.266032 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bb71e2-59f79f4f99-xh9lj" event={"ID":"72a0ed5e-1d31-4695-9dc5-92a304c0dc69","Type":"ContainerStarted","Data":"cd3ca241f8386b317f249ae483ef74711a80293ab66e396b6155d4a474c1f175"} Oct 30 13:02:58.266898 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:58.266875 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bb71e2-59f79f4f99-xh9lj" Oct 30 13:02:58.278318 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:58.278292 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-484607-5684dfc6c6-8z79j" Oct 30 13:02:58.312131 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:58.312075 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bb71e2-59f79f4f99-xh9lj" podStartSLOduration=2.312056087 podStartE2EDuration="2.312056087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:58.311934819 +0000 UTC m=+634.781350577" watchObservedRunningTime="2026-10-30 13:02:58.312056087 +0000 UTC m=+634.781471834" Oct 30 13:02:58.497307 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:58.497282 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-484607-6d646d7567-hd7j6"] Oct 30 13:02:58.500670 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:02:58.500640 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62a8727a_ad11_4c04_87e1_0b198af32183.slice/crio-c27c994131b6c9d5206cc9ecdc4ec0e6c70c0d4b9f416064dc15f9880b065d62 WatchSource:0}: Error finding container c27c994131b6c9d5206cc9ecdc4ec0e6c70c0d4b9f416064dc15f9880b065d62: Status 404 returned error can't find the container with id c27c994131b6c9d5206cc9ecdc4ec0e6c70c0d4b9f416064dc15f9880b065d62 Oct 30 13:02:58.597290 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:58.597218 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-484607-5684dfc6c6-8z79j"] Oct 30 13:02:58.601380 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:02:58.601353 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82af0f22_cb2d_41c5_b4a8_adcecc306998.slice/crio-5f128ba84869d6da51d064f08e27df44dfc1f4c312a1643484d160af1d3974bc WatchSource:0}: Error finding container 5f128ba84869d6da51d064f08e27df44dfc1f4c312a1643484d160af1d3974bc: Status 404 returned error can't find the container with id 5f128ba84869d6da51d064f08e27df44dfc1f4c312a1643484d160af1d3974bc Oct 30 13:02:58.630671 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:58.630630 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:58.749440 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:58.749404 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:58.843719 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:58.843689 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-175831-6f46fb6d59-ltklr"] Oct 30 13:02:58.846296 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:58.846275 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-175831-6f46fb6d59-ltklr" Oct 30 13:02:58.866806 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:58.866575 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-175831-6f46fb6d59-ltklr"] Oct 30 13:02:58.868054 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:58.868032 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-175831\"" Oct 30 13:02:58.871674 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:58.868033 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-175831-dockercfg-xv7qs\"" Oct 30 13:02:59.000146 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:59.000108 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81a0354c-9716-4cc3-a902-b81908879ab3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-175831-6f46fb6d59-ltklr\" (UID: \"81a0354c-9716-4cc3-a902-b81908879ab3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-175831-6f46fb6d59-ltklr" Oct 30 13:02:59.000361 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:59.000173 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwjww\" (UniqueName: \"kubernetes.io/projected/81a0354c-9716-4cc3-a902-b81908879ab3-kube-api-access-cwjww\") pod \"payload-pre-processing-e2e-worker-w5-175831-6f46fb6d59-ltklr\" (UID: \"81a0354c-9716-4cc3-a902-b81908879ab3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-175831-6f46fb6d59-ltklr" Oct 30 13:02:59.101147 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:59.101110 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81a0354c-9716-4cc3-a902-b81908879ab3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-175831-6f46fb6d59-ltklr\" (UID: \"81a0354c-9716-4cc3-a902-b81908879ab3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-175831-6f46fb6d59-ltklr" Oct 30 13:02:59.101311 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:59.101164 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwjww\" (UniqueName: \"kubernetes.io/projected/81a0354c-9716-4cc3-a902-b81908879ab3-kube-api-access-cwjww\") pod \"payload-pre-processing-e2e-worker-w5-175831-6f46fb6d59-ltklr\" (UID: \"81a0354c-9716-4cc3-a902-b81908879ab3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-175831-6f46fb6d59-ltklr" Oct 30 13:02:59.101842 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:59.101822 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81a0354c-9716-4cc3-a902-b81908879ab3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-175831-6f46fb6d59-ltklr\" (UID: \"81a0354c-9716-4cc3-a902-b81908879ab3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-175831-6f46fb6d59-ltklr" Oct 30 13:02:59.125781 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:59.125713 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwjww\" (UniqueName: \"kubernetes.io/projected/81a0354c-9716-4cc3-a902-b81908879ab3-kube-api-access-cwjww\") pod \"payload-pre-processing-e2e-worker-w5-175831-6f46fb6d59-ltklr\" (UID: \"81a0354c-9716-4cc3-a902-b81908879ab3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-175831-6f46fb6d59-ltklr" Oct 30 13:02:59.158380 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:59.158347 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-175831-6f46fb6d59-ltklr" Oct 30 13:02:59.295286 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:59.295189 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-484607-5684dfc6c6-8z79j" event={"ID":"82af0f22-cb2d-41c5-b4a8-adcecc306998","Type":"ContainerStarted","Data":"aa05590eeb7de7aaa57966fe245390736ced2c4977a13ccf7666622926e189b4"} Oct 30 13:02:59.295286 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:59.295237 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-484607-5684dfc6c6-8z79j" event={"ID":"82af0f22-cb2d-41c5-b4a8-adcecc306998","Type":"ContainerStarted","Data":"5f128ba84869d6da51d064f08e27df44dfc1f4c312a1643484d160af1d3974bc"} Oct 30 13:02:59.296057 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:59.296002 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-484607-5684dfc6c6-8z79j" Oct 30 13:02:59.298758 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:59.298658 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-484607-6d646d7567-hd7j6" event={"ID":"62a8727a-ad11-4c04-87e1-0b198af32183","Type":"ContainerStarted","Data":"5603efccb4fa9200a032236fd33f3c45f3cf5b097848295ac27a877bd8aa8874"} Oct 30 13:02:59.298758 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:59.298701 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-484607-6d646d7567-hd7j6" event={"ID":"62a8727a-ad11-4c04-87e1-0b198af32183","Type":"ContainerStarted","Data":"c27c994131b6c9d5206cc9ecdc4ec0e6c70c0d4b9f416064dc15f9880b065d62"} Oct 30 13:02:59.298758 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:59.298725 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-484607-6d646d7567-hd7j6" Oct 30 13:02:59.340567 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:59.340527 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-484607-5684dfc6c6-8z79j" podStartSLOduration=2.340515543 podStartE2EDuration="2.340515543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:59.339303113 +0000 UTC m=+635.808718869" watchObservedRunningTime="2026-10-30 13:02:59.340515543 +0000 UTC m=+635.809931287" Oct 30 13:02:59.389245 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:59.389141 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-484607-6d646d7567-hd7j6" podStartSLOduration=2.389126489 podStartE2EDuration="2.389126489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:59.387408151 +0000 UTC m=+635.856823921" watchObservedRunningTime="2026-10-30 13:02:59.389126489 +0000 UTC m=+635.858542234" Oct 30 13:02:59.483489 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:02:59.479672 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-175831-6f46fb6d59-ltklr"] Oct 30 13:03:00.303649 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:00.303588 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-175831-6f46fb6d59-ltklr" event={"ID":"81a0354c-9716-4cc3-a902-b81908879ab3","Type":"ContainerStarted","Data":"991f0f1fdb622d1e6103d8df2c36e47883f508b34bed8c4da8a818f6a5451191"} Oct 30 13:03:00.304111 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:00.303669 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-175831-6f46fb6d59-ltklr" event={"ID":"81a0354c-9716-4cc3-a902-b81908879ab3","Type":"ContainerStarted","Data":"f7acf4bb2de15bede613705a5ebdafe17b0e4be7d9f47a6f0d88ca4bf786068c"} Oct 30 13:03:00.304111 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:00.303831 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-175831-6f46fb6d59-ltklr" Oct 30 13:03:00.348060 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:00.348014 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-175831-6f46fb6d59-ltklr" podStartSLOduration=2.348001769 podStartE2EDuration="2.348001769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:00.34674681 +0000 UTC m=+636.816162560" watchObservedRunningTime="2026-10-30 13:03:00.348001769 +0000 UTC m=+636.817417514" Oct 30 13:03:00.485997 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:00.485738 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:00.603406 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:00.603329 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:00.655067 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:00.655039 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-54994c-75d97544d7-lvpzs"] Oct 30 13:03:00.658979 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:00.658946 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-54994c-75d97544d7-lvpzs" Oct 30 13:03:00.684016 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:00.683986 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-54994c-dockercfg-cpg6l\"" Oct 30 13:03:00.692442 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:00.692416 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-54994c-75d97544d7-lvpzs"] Oct 30 13:03:00.734994 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:00.734966 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-54994c-f69fc5d86-68fv5"] Oct 30 13:03:00.737662 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:00.737646 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-54994c-f69fc5d86-68fv5" Oct 30 13:03:00.754780 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:00.754757 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-54994c\"" Oct 30 13:03:00.795555 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:00.795521 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-54994c-f69fc5d86-68fv5"] Oct 30 13:03:00.816229 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:00.816167 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrzj8\" (UniqueName: \"kubernetes.io/projected/ac077401-1be7-4461-a1e5-bc4f186c9a94-kube-api-access-mrzj8\") pod \"payload-pre-processing-e2e-worker-w4-54994c-75d97544d7-lvpzs\" (UID: \"ac077401-1be7-4461-a1e5-bc4f186c9a94\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-54994c-75d97544d7-lvpzs" Oct 30 13:03:00.816441 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:00.816412 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/727a0820-c1f2-469e-827d-3f623e9df963-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-54994c-f69fc5d86-68fv5\" (UID: \"727a0820-c1f2-469e-827d-3f623e9df963\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-54994c-f69fc5d86-68fv5" Oct 30 13:03:00.816553 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:00.816468 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac077401-1be7-4461-a1e5-bc4f186c9a94-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-54994c-75d97544d7-lvpzs\" (UID: \"ac077401-1be7-4461-a1e5-bc4f186c9a94\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-54994c-75d97544d7-lvpzs" Oct 30 13:03:00.816697 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:00.816672 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bxsg\" (UniqueName: \"kubernetes.io/projected/727a0820-c1f2-469e-827d-3f623e9df963-kube-api-access-7bxsg\") pod \"payload-processing-e2e-worker-w4-54994c-f69fc5d86-68fv5\" (UID: \"727a0820-c1f2-469e-827d-3f623e9df963\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-54994c-f69fc5d86-68fv5" Oct 30 13:03:00.917463 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:00.917390 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bxsg\" (UniqueName: \"kubernetes.io/projected/727a0820-c1f2-469e-827d-3f623e9df963-kube-api-access-7bxsg\") pod \"payload-processing-e2e-worker-w4-54994c-f69fc5d86-68fv5\" (UID: \"727a0820-c1f2-469e-827d-3f623e9df963\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-54994c-f69fc5d86-68fv5" Oct 30 13:03:00.917463 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:00.917440 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrzj8\" (UniqueName: \"kubernetes.io/projected/ac077401-1be7-4461-a1e5-bc4f186c9a94-kube-api-access-mrzj8\") pod \"payload-pre-processing-e2e-worker-w4-54994c-75d97544d7-lvpzs\" (UID: \"ac077401-1be7-4461-a1e5-bc4f186c9a94\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-54994c-75d97544d7-lvpzs" Oct 30 13:03:00.917463 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:00.917463 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/727a0820-c1f2-469e-827d-3f623e9df963-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-54994c-f69fc5d86-68fv5\" (UID: \"727a0820-c1f2-469e-827d-3f623e9df963\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-54994c-f69fc5d86-68fv5" Oct 30 13:03:00.917772 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:00.917485 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac077401-1be7-4461-a1e5-bc4f186c9a94-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-54994c-75d97544d7-lvpzs\" (UID: \"ac077401-1be7-4461-a1e5-bc4f186c9a94\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-54994c-75d97544d7-lvpzs" Oct 30 13:03:00.918416 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:00.918392 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac077401-1be7-4461-a1e5-bc4f186c9a94-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-54994c-75d97544d7-lvpzs\" (UID: \"ac077401-1be7-4461-a1e5-bc4f186c9a94\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-54994c-75d97544d7-lvpzs" Oct 30 13:03:00.918448 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:00.918426 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/727a0820-c1f2-469e-827d-3f623e9df963-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-54994c-f69fc5d86-68fv5\" (UID: \"727a0820-c1f2-469e-827d-3f623e9df963\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-54994c-f69fc5d86-68fv5" Oct 30 13:03:00.942161 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:00.942134 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrzj8\" (UniqueName: \"kubernetes.io/projected/ac077401-1be7-4461-a1e5-bc4f186c9a94-kube-api-access-mrzj8\") pod \"payload-pre-processing-e2e-worker-w4-54994c-75d97544d7-lvpzs\" (UID: \"ac077401-1be7-4461-a1e5-bc4f186c9a94\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-54994c-75d97544d7-lvpzs" Oct 30 13:03:00.942292 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:00.942261 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bxsg\" (UniqueName: \"kubernetes.io/projected/727a0820-c1f2-469e-827d-3f623e9df963-kube-api-access-7bxsg\") pod \"payload-processing-e2e-worker-w4-54994c-f69fc5d86-68fv5\" (UID: \"727a0820-c1f2-469e-827d-3f623e9df963\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-54994c-f69fc5d86-68fv5" Oct 30 13:03:01.001331 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:01.001300 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-54994c-75d97544d7-lvpzs" Oct 30 13:03:01.051989 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:01.051737 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-54994c-f69fc5d86-68fv5" Oct 30 13:03:01.305683 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:01.305645 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-54994c-75d97544d7-lvpzs"] Oct 30 13:03:01.308697 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:03:01.308670 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac077401_1be7_4461_a1e5_bc4f186c9a94.slice/crio-5571a137a07f5b446beec8d0214346f54075c563083352fbdc497cc133c4d99d WatchSource:0}: Error finding container 5571a137a07f5b446beec8d0214346f54075c563083352fbdc497cc133c4d99d: Status 404 returned error can't find the container with id 5571a137a07f5b446beec8d0214346f54075c563083352fbdc497cc133c4d99d Oct 30 13:03:01.389531 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:01.389493 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-54994c-f69fc5d86-68fv5"] Oct 30 13:03:01.393834 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:03:01.393809 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod727a0820_c1f2_469e_827d_3f623e9df963.slice/crio-9f3fd069eaa512bfcba8a0b0e5aa09a3ca98917e4360ea1be9265d2211be760a WatchSource:0}: Error finding container 9f3fd069eaa512bfcba8a0b0e5aa09a3ca98917e4360ea1be9265d2211be760a: Status 404 returned error can't find the container with id 9f3fd069eaa512bfcba8a0b0e5aa09a3ca98917e4360ea1be9265d2211be760a Oct 30 13:03:01.592682 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:01.591448 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:01.705709 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:01.705672 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:01.716376 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:01.716348 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:03:01.856553 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:01.845924 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-42ee9d-5645d7744d-d6f7m"] Oct 30 13:03:01.880637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:01.878144 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-42ee9d-5645d7744d-d6f7m"] Oct 30 13:03:01.880637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:01.878272 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-42ee9d-5645d7744d-d6f7m" Oct 30 13:03:01.903171 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:01.901007 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-42ee9d\"" Oct 30 13:03:01.903171 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:01.901747 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-42ee9d-dockercfg-vm8lf\"" Oct 30 13:03:01.937125 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:01.937096 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8sc6\" (UniqueName: \"kubernetes.io/projected/85a299c7-b978-4b75-8aed-fff5da4d32f7-kube-api-access-k8sc6\") pod \"payload-processing-e2e-worker-w0-42ee9d-5645d7744d-d6f7m\" (UID: \"85a299c7-b978-4b75-8aed-fff5da4d32f7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-42ee9d-5645d7744d-d6f7m" Oct 30 13:03:01.937252 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:01.937227 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85a299c7-b978-4b75-8aed-fff5da4d32f7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-42ee9d-5645d7744d-d6f7m\" (UID: \"85a299c7-b978-4b75-8aed-fff5da4d32f7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-42ee9d-5645d7744d-d6f7m" Oct 30 13:03:02.038881 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:02.038850 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85a299c7-b978-4b75-8aed-fff5da4d32f7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-42ee9d-5645d7744d-d6f7m\" (UID: \"85a299c7-b978-4b75-8aed-fff5da4d32f7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-42ee9d-5645d7744d-d6f7m" Oct 30 13:03:02.039045 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:02.038985 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8sc6\" (UniqueName: \"kubernetes.io/projected/85a299c7-b978-4b75-8aed-fff5da4d32f7-kube-api-access-k8sc6\") pod \"payload-processing-e2e-worker-w0-42ee9d-5645d7744d-d6f7m\" (UID: \"85a299c7-b978-4b75-8aed-fff5da4d32f7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-42ee9d-5645d7744d-d6f7m" Oct 30 13:03:02.039553 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:02.039529 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85a299c7-b978-4b75-8aed-fff5da4d32f7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-42ee9d-5645d7744d-d6f7m\" (UID: \"85a299c7-b978-4b75-8aed-fff5da4d32f7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-42ee9d-5645d7744d-d6f7m" Oct 30 13:03:02.081699 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:02.072480 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8sc6\" (UniqueName: \"kubernetes.io/projected/85a299c7-b978-4b75-8aed-fff5da4d32f7-kube-api-access-k8sc6\") pod \"payload-processing-e2e-worker-w0-42ee9d-5645d7744d-d6f7m\" (UID: \"85a299c7-b978-4b75-8aed-fff5da4d32f7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-42ee9d-5645d7744d-d6f7m" Oct 30 13:03:02.207159 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:02.207122 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-42ee9d-5645d7744d-d6f7m" Oct 30 13:03:02.319333 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:02.317585 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-54994c-f69fc5d86-68fv5" event={"ID":"727a0820-c1f2-469e-827d-3f623e9df963","Type":"ContainerStarted","Data":"c316cebabc1a6f07a6e8b66c0b4e6cd81ef845b3954ca7962283e92872d1781a"} Oct 30 13:03:02.319333 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:02.317669 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-54994c-f69fc5d86-68fv5" event={"ID":"727a0820-c1f2-469e-827d-3f623e9df963","Type":"ContainerStarted","Data":"9f3fd069eaa512bfcba8a0b0e5aa09a3ca98917e4360ea1be9265d2211be760a"} Oct 30 13:03:02.319333 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:02.318529 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-54994c-f69fc5d86-68fv5" Oct 30 13:03:02.320238 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:02.320215 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-54994c-75d97544d7-lvpzs" event={"ID":"ac077401-1be7-4461-a1e5-bc4f186c9a94","Type":"ContainerStarted","Data":"fc5b24620981b6fad12282a9e22d5812e805032b934d1e30da4d192be0d4b6be"} Oct 30 13:03:02.320299 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:02.320253 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-54994c-75d97544d7-lvpzs" event={"ID":"ac077401-1be7-4461-a1e5-bc4f186c9a94","Type":"ContainerStarted","Data":"5571a137a07f5b446beec8d0214346f54075c563083352fbdc497cc133c4d99d"} Oct 30 13:03:02.320941 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:02.320753 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-54994c-75d97544d7-lvpzs" Oct 30 13:03:02.506684 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:02.506206 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-54994c-75d97544d7-lvpzs" podStartSLOduration=2.506186838 podStartE2EDuration="2.506186838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:02.505826362 +0000 UTC m=+638.975242110" watchObservedRunningTime="2026-10-30 13:03:02.506186838 +0000 UTC m=+638.975602585" Oct 30 13:03:02.514705 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:02.506990 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-54994c-f69fc5d86-68fv5" podStartSLOduration=2.506979203 podStartE2EDuration="2.506979203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:02.412891215 +0000 UTC m=+638.882306961" watchObservedRunningTime="2026-10-30 13:03:02.506979203 +0000 UTC m=+638.976394948" Oct 30 13:03:02.561165 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:02.560342 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-42ee9d-5645d7744d-d6f7m"] Oct 30 13:03:02.585635 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:02.580888 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:02.695675 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:02.692457 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:02.814719 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:02.809709 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:03.327892 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:03.327863 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-42ee9d-5645d7744d-d6f7m" event={"ID":"85a299c7-b978-4b75-8aed-fff5da4d32f7","Type":"ContainerStarted","Data":"0d2bf6b0fa8808b75c204fd58adfd598458a476a1b1238b82760ac012441cfb9"} Oct 30 13:03:03.328387 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:03.328366 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-42ee9d-5645d7744d-d6f7m" event={"ID":"85a299c7-b978-4b75-8aed-fff5da4d32f7","Type":"ContainerStarted","Data":"b50f6a645c50d434c6c1dcb6f69e4e559e83f51a8e469e09091a36b262885c2b"} Oct 30 13:03:03.328493 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:03.328484 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-42ee9d-5645d7744d-d6f7m" Oct 30 13:03:03.382637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:03.379664 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-42ee9d-5645d7744d-d6f7m" podStartSLOduration=2.379645154 podStartE2EDuration="2.379645154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:03.377493684 +0000 UTC m=+639.846909443" watchObservedRunningTime="2026-10-30 13:03:03.379645154 +0000 UTC m=+639.849060900" Oct 30 13:03:03.550014 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:03.549938 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:03.714694 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:03.714506 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:03.819476 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:03.819447 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-ac0c58-7df9968d48-46vcj"] Oct 30 13:03:03.828152 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:03.828126 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-ac0c58-7df9968d48-46vcj" Oct 30 13:03:03.851596 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:03.849519 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-ac0c58\"" Oct 30 13:03:03.851596 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:03.849818 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-ac0c58-dockercfg-cgdbx\"" Oct 30 13:03:03.886570 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:03.886534 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84fqm\" (UniqueName: \"kubernetes.io/projected/b4fd49fb-f890-446a-9d22-435e43054ea5-kube-api-access-84fqm\") pod \"payload-processing-e2e-worker-w1-ac0c58-7df9968d48-46vcj\" (UID: \"b4fd49fb-f890-446a-9d22-435e43054ea5\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-ac0c58-7df9968d48-46vcj" Oct 30 13:03:03.936987 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:03.936945 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4fd49fb-f890-446a-9d22-435e43054ea5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-ac0c58-7df9968d48-46vcj\" (UID: \"b4fd49fb-f890-446a-9d22-435e43054ea5\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-ac0c58-7df9968d48-46vcj" Oct 30 13:03:03.940690 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:03.940662 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-ac0c58-7df9968d48-46vcj"] Oct 30 13:03:04.038159 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:04.038080 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4fd49fb-f890-446a-9d22-435e43054ea5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-ac0c58-7df9968d48-46vcj\" (UID: \"b4fd49fb-f890-446a-9d22-435e43054ea5\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-ac0c58-7df9968d48-46vcj" Oct 30 13:03:04.038159 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:04.038136 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84fqm\" (UniqueName: \"kubernetes.io/projected/b4fd49fb-f890-446a-9d22-435e43054ea5-kube-api-access-84fqm\") pod \"payload-processing-e2e-worker-w1-ac0c58-7df9968d48-46vcj\" (UID: \"b4fd49fb-f890-446a-9d22-435e43054ea5\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-ac0c58-7df9968d48-46vcj" Oct 30 13:03:04.038831 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:04.038809 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4fd49fb-f890-446a-9d22-435e43054ea5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-ac0c58-7df9968d48-46vcj\" (UID: \"b4fd49fb-f890-446a-9d22-435e43054ea5\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-ac0c58-7df9968d48-46vcj" Oct 30 13:03:04.117315 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:04.117287 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84fqm\" (UniqueName: \"kubernetes.io/projected/b4fd49fb-f890-446a-9d22-435e43054ea5-kube-api-access-84fqm\") pod \"payload-processing-e2e-worker-w1-ac0c58-7df9968d48-46vcj\" (UID: \"b4fd49fb-f890-446a-9d22-435e43054ea5\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-ac0c58-7df9968d48-46vcj" Oct 30 13:03:04.146368 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:04.146341 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-ac0c58-7df9968d48-46vcj" Oct 30 13:03:04.490319 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:04.490279 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:04.513737 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:04.513682 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-ac0c58-7df9968d48-46vcj"] Oct 30 13:03:04.516652 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:03:04.516605 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4fd49fb_f890_446a_9d22_435e43054ea5.slice/crio-8b5272a22fba991f048881011de4712d580ffd0d639c542e3b4420f6a22e1a0d WatchSource:0}: Error finding container 8b5272a22fba991f048881011de4712d580ffd0d639c542e3b4420f6a22e1a0d: Status 404 returned error can't find the container with id 8b5272a22fba991f048881011de4712d580ffd0d639c542e3b4420f6a22e1a0d Oct 30 13:03:04.610397 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:04.610360 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:05.341685 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:05.341645 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-ac0c58-7df9968d48-46vcj" event={"ID":"b4fd49fb-f890-446a-9d22-435e43054ea5","Type":"ContainerStarted","Data":"d74bfad8de0eeede6269e4d45d87d57b58039f528e0fd5837bb3ef20c8d3ec0e"} Oct 30 13:03:05.341874 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:05.341709 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-ac0c58-7df9968d48-46vcj" event={"ID":"b4fd49fb-f890-446a-9d22-435e43054ea5","Type":"ContainerStarted","Data":"8b5272a22fba991f048881011de4712d580ffd0d639c542e3b4420f6a22e1a0d"} Oct 30 13:03:05.341874 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:05.341828 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-ac0c58-7df9968d48-46vcj" Oct 30 13:03:05.388433 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:05.388372 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-ac0c58-7df9968d48-46vcj" podStartSLOduration=2.388352787 podStartE2EDuration="2.388352787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:05.386998581 +0000 UTC m=+641.856414353" watchObservedRunningTime="2026-10-30 13:03:05.388352787 +0000 UTC m=+641.857768533" Oct 30 13:03:06.317368 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:06.317331 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:06.520628 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:06.520578 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:07.258483 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:07.258447 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3048f9-fc55696f5-j6t4d" Oct 30 13:03:08.131454 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:08.131414 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:08.253074 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:08.253041 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:08.276175 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:08.275804 5922 patch_prober.go:29] interesting pod/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:03:08.276175 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:08.275859 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" podUID="cecdab1d-a927-4d7a-b270-ba7908c88d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:03:08.417680 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:08.417594 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:08.636563 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:08.636443 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:09.246465 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:09.246416 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:03:10.044623 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:10.044567 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:10.319007 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:10.318922 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-484607-5684dfc6c6-8z79j" Oct 30 13:03:10.319497 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:10.319014 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-484607-6d646d7567-hd7j6" Oct 30 13:03:10.319497 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:10.319333 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bb71e2-59f79f4f99-xh9lj" Oct 30 13:03:11.318432 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:11.318395 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-175831-6f46fb6d59-ltklr" Oct 30 13:03:11.436075 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:11.435993 5922 generic.go:356] "Generic (PLEG): container finished" podID="cecdab1d-a927-4d7a-b270-ba7908c88d52" containerID="39fe7c325a40de5d43ff7303c8776ba7b385000c66965a74ea286879561694a6" exitCode=0 Oct 30 13:03:11.436544 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:11.436097 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" event={"ID":"cecdab1d-a927-4d7a-b270-ba7908c88d52","Type":"ContainerDied","Data":"39fe7c325a40de5d43ff7303c8776ba7b385000c66965a74ea286879561694a6"} Oct 30 13:03:11.436544 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:11.436466 5922 scope.go:122] "RemoveContainer" containerID="39fe7c325a40de5d43ff7303c8776ba7b385000c66965a74ea286879561694a6" Oct 30 13:03:11.437837 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:11.437805 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:03:11.437941 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:11.437878 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:03:11.437941 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:11.437911 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:03:12.323065 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:12.322933 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:12.447640 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:12.446711 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" event={"ID":"cecdab1d-a927-4d7a-b270-ba7908c88d52","Type":"ContainerStarted","Data":"3e878132d030716b2a26484e794a3815e7d2f9d535210bc5c095d4472e0dda0d"} Oct 30 13:03:12.611408 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:12.611300 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" Oct 30 13:03:12.611578 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:12.611517 5922 patch_prober.go:29] interesting pod/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 13:03:12.611578 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:12.611563 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" podUID="cecdab1d-a927-4d7a-b270-ba7908c88d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 13:03:12.908097 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:12.908004 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:13.520210 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:13.520166 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:13.612326 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:13.612285 5922 patch_prober.go:29] interesting pod/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 13:03:13.612519 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:13.612358 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" podUID="cecdab1d-a927-4d7a-b270-ba7908c88d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 13:03:13.654422 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:13.654379 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:13.859109 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:13.858977 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:14.340197 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:14.340082 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-54994c-f69fc5d86-68fv5" Oct 30 13:03:14.340410 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:14.340348 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-54994c-75d97544d7-lvpzs" Oct 30 13:03:14.612245 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:14.612158 5922 patch_prober.go:29] interesting pod/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 13:03:14.612245 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:14.612219 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" podUID="cecdab1d-a927-4d7a-b270-ba7908c88d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 13:03:14.848489 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:14.848455 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:15.439690 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:15.344835 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-42ee9d-5645d7744d-d6f7m" Oct 30 13:03:15.489682 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:15.470530 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j"] Oct 30 13:03:15.514181 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:15.514146 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j"] Oct 30 13:03:15.514351 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:15.514260 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" Oct 30 13:03:15.535470 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:15.535442 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-3048f9\"/\"kube-root-ca.crt\"" Oct 30 13:03:15.535674 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:15.535582 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-3048f9\"/\"openshift-service-ca.crt\"" Oct 30 13:03:15.535835 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:15.535812 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:15.544216 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:15.544194 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-3048f9\"/\"default-dockercfg-f7hxq\"" Oct 30 13:03:15.544357 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:15.544238 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-3048f9\"/\"simulator-w3-3048f9-kserve-self-signed-certs\"" Oct 30 13:03:15.605545 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:15.605508 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0ea7fde-2ac5-432d-bb9b-534024186ac9-tmp-dir\") pod \"simulator-w3-3048f9-kserve-5665c6ccb8-zt49j\" (UID: \"c0ea7fde-2ac5-432d-bb9b-534024186ac9\") " pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" Oct 30 13:03:15.605766 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:15.605689 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0ea7fde-2ac5-432d-bb9b-534024186ac9-dshm\") pod \"simulator-w3-3048f9-kserve-5665c6ccb8-zt49j\" (UID: \"c0ea7fde-2ac5-432d-bb9b-534024186ac9\") " pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" Oct 30 13:03:15.605766 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:15.605748 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf6g8\" (UniqueName: \"kubernetes.io/projected/c0ea7fde-2ac5-432d-bb9b-534024186ac9-kube-api-access-bf6g8\") pod \"simulator-w3-3048f9-kserve-5665c6ccb8-zt49j\" (UID: \"c0ea7fde-2ac5-432d-bb9b-534024186ac9\") " pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" Oct 30 13:03:15.605875 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:15.605805 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0ea7fde-2ac5-432d-bb9b-534024186ac9-model-cache\") pod \"simulator-w3-3048f9-kserve-5665c6ccb8-zt49j\" (UID: \"c0ea7fde-2ac5-432d-bb9b-534024186ac9\") " pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" Oct 30 13:03:15.605875 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:15.605851 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0ea7fde-2ac5-432d-bb9b-534024186ac9-home\") pod \"simulator-w3-3048f9-kserve-5665c6ccb8-zt49j\" (UID: \"c0ea7fde-2ac5-432d-bb9b-534024186ac9\") " pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" Oct 30 13:03:15.605976 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:15.605923 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0ea7fde-2ac5-432d-bb9b-534024186ac9-tls-certs\") pod \"simulator-w3-3048f9-kserve-5665c6ccb8-zt49j\" (UID: \"c0ea7fde-2ac5-432d-bb9b-534024186ac9\") " pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" Oct 30 13:03:15.611555 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:15.611521 5922 patch_prober.go:29] interesting pod/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 13:03:15.611712 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:15.611594 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" podUID="cecdab1d-a927-4d7a-b270-ba7908c88d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 13:03:15.706722 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:15.706680 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0ea7fde-2ac5-432d-bb9b-534024186ac9-tls-certs\") pod \"simulator-w3-3048f9-kserve-5665c6ccb8-zt49j\" (UID: \"c0ea7fde-2ac5-432d-bb9b-534024186ac9\") " pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" Oct 30 13:03:15.707232 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:15.706747 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0ea7fde-2ac5-432d-bb9b-534024186ac9-tmp-dir\") pod \"simulator-w3-3048f9-kserve-5665c6ccb8-zt49j\" (UID: \"c0ea7fde-2ac5-432d-bb9b-534024186ac9\") " pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" Oct 30 13:03:15.707232 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:15.706849 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0ea7fde-2ac5-432d-bb9b-534024186ac9-dshm\") pod \"simulator-w3-3048f9-kserve-5665c6ccb8-zt49j\" (UID: \"c0ea7fde-2ac5-432d-bb9b-534024186ac9\") " pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" Oct 30 13:03:15.707232 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:15.706901 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf6g8\" (UniqueName: \"kubernetes.io/projected/c0ea7fde-2ac5-432d-bb9b-534024186ac9-kube-api-access-bf6g8\") pod \"simulator-w3-3048f9-kserve-5665c6ccb8-zt49j\" (UID: \"c0ea7fde-2ac5-432d-bb9b-534024186ac9\") " pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" Oct 30 13:03:15.707232 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:15.706953 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0ea7fde-2ac5-432d-bb9b-534024186ac9-model-cache\") pod \"simulator-w3-3048f9-kserve-5665c6ccb8-zt49j\" (UID: \"c0ea7fde-2ac5-432d-bb9b-534024186ac9\") " pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" Oct 30 13:03:15.707232 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:15.706996 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0ea7fde-2ac5-432d-bb9b-534024186ac9-home\") pod \"simulator-w3-3048f9-kserve-5665c6ccb8-zt49j\" (UID: \"c0ea7fde-2ac5-432d-bb9b-534024186ac9\") " pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" Oct 30 13:03:15.707232 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:15.707167 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0ea7fde-2ac5-432d-bb9b-534024186ac9-tmp-dir\") pod \"simulator-w3-3048f9-kserve-5665c6ccb8-zt49j\" (UID: \"c0ea7fde-2ac5-432d-bb9b-534024186ac9\") " pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" Oct 30 13:03:15.707554 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:15.707353 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0ea7fde-2ac5-432d-bb9b-534024186ac9-home\") pod \"simulator-w3-3048f9-kserve-5665c6ccb8-zt49j\" (UID: \"c0ea7fde-2ac5-432d-bb9b-534024186ac9\") " pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" Oct 30 13:03:15.707554 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:15.707399 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0ea7fde-2ac5-432d-bb9b-534024186ac9-model-cache\") pod \"simulator-w3-3048f9-kserve-5665c6ccb8-zt49j\" (UID: \"c0ea7fde-2ac5-432d-bb9b-534024186ac9\") " pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" Oct 30 13:03:15.709343 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:15.709314 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0ea7fde-2ac5-432d-bb9b-534024186ac9-dshm\") pod \"simulator-w3-3048f9-kserve-5665c6ccb8-zt49j\" (UID: \"c0ea7fde-2ac5-432d-bb9b-534024186ac9\") " pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" Oct 30 13:03:15.709645 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:15.709601 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0ea7fde-2ac5-432d-bb9b-534024186ac9-tls-certs\") pod \"simulator-w3-3048f9-kserve-5665c6ccb8-zt49j\" (UID: \"c0ea7fde-2ac5-432d-bb9b-534024186ac9\") " pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" Oct 30 13:03:15.735987 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:15.735956 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf6g8\" (UniqueName: \"kubernetes.io/projected/c0ea7fde-2ac5-432d-bb9b-534024186ac9-kube-api-access-bf6g8\") pod \"simulator-w3-3048f9-kserve-5665c6ccb8-zt49j\" (UID: \"c0ea7fde-2ac5-432d-bb9b-534024186ac9\") " pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" Oct 30 13:03:15.851293 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:15.851261 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" Oct 30 13:03:16.206723 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:16.206690 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j"] Oct 30 13:03:16.211478 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:03:16.211452 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0ea7fde_2ac5_432d_bb9b_534024186ac9.slice/crio-84326493f218fbf8a55a33d5e321c499a00ac03a528793c784a87d7e19f9ad3b WatchSource:0}: Error finding container 84326493f218fbf8a55a33d5e321c499a00ac03a528793c784a87d7e19f9ad3b: Status 404 returned error can't find the container with id 84326493f218fbf8a55a33d5e321c499a00ac03a528793c784a87d7e19f9ad3b Oct 30 13:03:16.351758 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:16.351725 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-ac0c58-7df9968d48-46vcj" Oct 30 13:03:16.510208 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:16.510134 5922 generic.go:356] "Generic (PLEG): container finished" podID="a68a48f7-d090-4eb7-af40-df304a84b319" containerID="933f6ee0846072e68eb36ab006c8cc87cea85081333069c8fb6816d7f3650b59" exitCode=0 Oct 30 13:03:16.510358 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:16.510216 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" event={"ID":"a68a48f7-d090-4eb7-af40-df304a84b319","Type":"ContainerDied","Data":"933f6ee0846072e68eb36ab006c8cc87cea85081333069c8fb6816d7f3650b59"} Oct 30 13:03:16.510669 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:16.510644 5922 scope.go:122] "RemoveContainer" containerID="933f6ee0846072e68eb36ab006c8cc87cea85081333069c8fb6816d7f3650b59" Oct 30 13:03:16.512266 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:16.512224 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:03:16.512363 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:16.512325 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:03:16.512448 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:16.512377 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:03:16.512448 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:16.512327 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" event={"ID":"c0ea7fde-2ac5-432d-bb9b-534024186ac9","Type":"ContainerStarted","Data":"a49b157703c046f271f6a406e5b85c923c3dc9a0a3f71a6fc6962a3f04cec30a"} Oct 30 13:03:16.512448 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:16.512434 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" event={"ID":"c0ea7fde-2ac5-432d-bb9b-534024186ac9","Type":"ContainerStarted","Data":"84326493f218fbf8a55a33d5e321c499a00ac03a528793c784a87d7e19f9ad3b"} Oct 30 13:03:16.603936 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:16.603571 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" podStartSLOduration=1.603550271 podStartE2EDuration="1.603550271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:16.60103131 +0000 UTC m=+653.070447066" watchObservedRunningTime="2026-10-30 13:03:16.603550271 +0000 UTC m=+653.072966017" Oct 30 13:03:16.612024 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:16.611991 5922 patch_prober.go:29] interesting pod/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 13:03:16.612191 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:16.612052 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" podUID="cecdab1d-a927-4d7a-b270-ba7908c88d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 13:03:16.852242 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:16.852154 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" Oct 30 13:03:17.051043 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:17.051006 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:17.814457 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:17.523742 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" event={"ID":"a68a48f7-d090-4eb7-af40-df304a84b319","Type":"ContainerStarted","Data":"170d1cca2c2bcc213a867eb484cd2cc7bbc0897f9156bff5b2620ebc42cbf4d7"} Oct 30 13:03:17.814457 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:17.613179 5922 patch_prober.go:29] interesting pod/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 13:03:17.814457 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:17.613240 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" podUID="cecdab1d-a927-4d7a-b270-ba7908c88d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 13:03:17.814457 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:17.782140 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" Oct 30 13:03:17.814457 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:17.784093 5922 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 13:03:17.814457 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:17.784143 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" podUID="a68a48f7-d090-4eb7-af40-df304a84b319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 13:03:18.052675 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:18.052644 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:18.529903 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:18.529873 5922 generic.go:356] "Generic (PLEG): container finished" podID="aced22f0-8827-4cad-b04b-85ddb56490af" containerID="622569e4d5eaebd803a4ee6adfb75551939cb04face002ebb9dd9a900633ff33" exitCode=0 Oct 30 13:03:18.530079 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:18.529941 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" event={"ID":"aced22f0-8827-4cad-b04b-85ddb56490af","Type":"ContainerDied","Data":"622569e4d5eaebd803a4ee6adfb75551939cb04face002ebb9dd9a900633ff33"} Oct 30 13:03:18.530348 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:18.530323 5922 scope.go:122] "RemoveContainer" containerID="622569e4d5eaebd803a4ee6adfb75551939cb04face002ebb9dd9a900633ff33" Oct 30 13:03:18.531589 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:18.531568 5922 generic.go:356] "Generic (PLEG): container finished" podID="d62b6ccd-e8a1-4320-87d0-0b12377ad8ee" containerID="5af4f5e7c7ee7cea6851d1ac6622767bda512d8d889d9f19d691bf996d53c70f" exitCode=0 Oct 30 13:03:18.531684 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:18.531647 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" event={"ID":"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee","Type":"ContainerDied","Data":"5af4f5e7c7ee7cea6851d1ac6622767bda512d8d889d9f19d691bf996d53c70f"} Oct 30 13:03:18.531724 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:18.531675 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:03:18.531765 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:18.531749 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:03:18.531803 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:18.531788 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:03:18.532132 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:18.532114 5922 scope.go:122] "RemoveContainer" containerID="5af4f5e7c7ee7cea6851d1ac6622767bda512d8d889d9f19d691bf996d53c70f" Oct 30 13:03:18.533211 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:18.533184 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:03:18.533303 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:18.533248 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:03:18.533365 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:18.533303 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:03:18.617870 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:18.616810 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" Oct 30 13:03:18.617870 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:18.617754 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" Oct 30 13:03:18.618736 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:18.618714 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" Oct 30 13:03:18.699633 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:18.697790 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:18.786117 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:18.785084 5922 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 13:03:18.786117 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:18.785143 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" podUID="a68a48f7-d090-4eb7-af40-df304a84b319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 13:03:18.820875 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:18.820774 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:18.934644 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:18.934279 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:19.210775 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:19.209230 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:19.545326 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:19.544275 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" event={"ID":"aced22f0-8827-4cad-b04b-85ddb56490af","Type":"ContainerStarted","Data":"9ccb1060f3bde50596ab6c0df7e12a58730d5a310da1f4299aa84a3a67358581"} Oct 30 13:03:19.549119 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:19.549083 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" event={"ID":"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee","Type":"ContainerStarted","Data":"f169a2618499b3b38b132015ed8e2245e35c526f25ddea2745e124a56074a928"} Oct 30 13:03:19.782636 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:19.782072 5922 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 13:03:19.782636 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:19.782161 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" podUID="a68a48f7-d090-4eb7-af40-df304a84b319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 13:03:19.880693 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:19.875372 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:03:19.935638 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:19.932412 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" Oct 30 13:03:19.943644 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:19.939911 5922 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 13:03:19.943644 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:19.939986 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" podUID="d62b6ccd-e8a1-4320-87d0-0b12377ad8ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 13:03:20.321911 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:20.321801 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" Oct 30 13:03:20.327094 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:20.326736 5922 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 13:03:20.327094 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:20.326809 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" podUID="aced22f0-8827-4cad-b04b-85ddb56490af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 13:03:20.783664 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:20.783074 5922 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 13:03:20.783664 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:20.783134 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" podUID="a68a48f7-d090-4eb7-af40-df304a84b319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 13:03:20.931637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:20.931109 5922 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 13:03:20.931637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:20.931184 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" podUID="d62b6ccd-e8a1-4320-87d0-0b12377ad8ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 13:03:21.267637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:21.258383 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:21.267637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:21.258631 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" Oct 30 13:03:21.267637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:21.258706 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" Oct 30 13:03:21.350928 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:21.350902 5922 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 13:03:21.351368 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:21.351347 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" podUID="aced22f0-8827-4cad-b04b-85ddb56490af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 13:03:21.784466 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:21.783997 5922 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 13:03:21.784466 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:21.784077 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" podUID="a68a48f7-d090-4eb7-af40-df304a84b319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 13:03:21.935008 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:21.934967 5922 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 13:03:21.935285 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:21.935255 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" podUID="d62b6ccd-e8a1-4320-87d0-0b12377ad8ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 13:03:22.241636 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:22.234037 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" Oct 30 13:03:22.321636 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:22.321001 5922 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 13:03:22.321636 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:22.321072 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" podUID="aced22f0-8827-4cad-b04b-85ddb56490af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 13:03:22.783632 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:22.783120 5922 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 13:03:22.783632 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:22.783197 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" podUID="a68a48f7-d090-4eb7-af40-df304a84b319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 13:03:22.929996 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:22.929958 5922 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 13:03:22.930205 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:22.930018 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" podUID="d62b6ccd-e8a1-4320-87d0-0b12377ad8ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 13:03:23.217237 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:23.217201 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:23.320242 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:23.320126 5922 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 13:03:23.320242 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:23.320185 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" podUID="aced22f0-8827-4cad-b04b-85ddb56490af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 13:03:23.334071 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:23.334041 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:23.483705 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:23.483445 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:23.558213 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:03:23.558178 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:03:23.805686 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:23.804894 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" Oct 30 13:03:23.811635 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:23.807823 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" Oct 30 13:03:23.820563 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:23.820401 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:23.929534 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:23.929501 5922 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 13:03:23.929743 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:23.929553 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" podUID="d62b6ccd-e8a1-4320-87d0-0b12377ad8ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 13:03:24.003180 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:24.003148 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:24.135767 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:24.135318 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:24.317528 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:24.316131 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:24.319879 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:24.319829 5922 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 13:03:24.320033 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:24.319899 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" podUID="aced22f0-8827-4cad-b04b-85ddb56490af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 13:03:24.929400 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:24.929368 5922 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 13:03:24.929885 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:24.929416 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" podUID="d62b6ccd-e8a1-4320-87d0-0b12377ad8ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 13:03:25.619504 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:25.619469 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" Oct 30 13:03:25.620752 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:25.620729 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" Oct 30 13:03:25.853087 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:25.853058 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" Oct 30 13:03:25.883042 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:25.882948 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" Oct 30 13:03:25.934161 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:25.934134 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" Oct 30 13:03:25.935279 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:25.935258 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" Oct 30 13:03:26.325839 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:26.325807 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:26.643074 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:26.642996 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" Oct 30 13:03:28.686202 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:28.686164 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:28.793902 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:28.793869 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:29.187608 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:29.187571 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:29.330379 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:29.330345 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:29.476271 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:29.476181 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:31.152042 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:31.152008 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5"] Oct 30 13:03:31.156939 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:31.156913 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" Oct 30 13:03:31.175576 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:31.175554 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5"] Oct 30 13:03:31.186126 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:31.186098 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-484607\"/\"openshift-service-ca.crt\"" Oct 30 13:03:31.186233 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:31.186160 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-484607\"/\"kube-root-ca.crt\"" Oct 30 13:03:31.189640 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:31.189604 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-484607\"/\"default-dockercfg-v4djb\"" Oct 30 13:03:31.189736 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:31.189641 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-484607\"/\"simulator-w6-484607-kserve-self-signed-certs\"" Oct 30 13:03:31.215176 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:31.215123 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:31.315961 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:31.315938 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b487a069-f519-45a3-b1d8-f32f2431797b-tmp-dir\") pod \"simulator-w6-484607-kserve-85f449c7dd-b5gv5\" (UID: \"b487a069-f519-45a3-b1d8-f32f2431797b\") " pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" Oct 30 13:03:31.316082 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:31.315983 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b487a069-f519-45a3-b1d8-f32f2431797b-dshm\") pod \"simulator-w6-484607-kserve-85f449c7dd-b5gv5\" (UID: \"b487a069-f519-45a3-b1d8-f32f2431797b\") " pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" Oct 30 13:03:31.316154 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:31.316115 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7twz\" (UniqueName: \"kubernetes.io/projected/b487a069-f519-45a3-b1d8-f32f2431797b-kube-api-access-v7twz\") pod \"simulator-w6-484607-kserve-85f449c7dd-b5gv5\" (UID: \"b487a069-f519-45a3-b1d8-f32f2431797b\") " pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" Oct 30 13:03:31.316265 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:31.316249 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b487a069-f519-45a3-b1d8-f32f2431797b-model-cache\") pod \"simulator-w6-484607-kserve-85f449c7dd-b5gv5\" (UID: \"b487a069-f519-45a3-b1d8-f32f2431797b\") " pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" Oct 30 13:03:31.316335 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:31.316278 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b487a069-f519-45a3-b1d8-f32f2431797b-tls-certs\") pod \"simulator-w6-484607-kserve-85f449c7dd-b5gv5\" (UID: \"b487a069-f519-45a3-b1d8-f32f2431797b\") " pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" Oct 30 13:03:31.316335 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:31.316309 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b487a069-f519-45a3-b1d8-f32f2431797b-home\") pod \"simulator-w6-484607-kserve-85f449c7dd-b5gv5\" (UID: \"b487a069-f519-45a3-b1d8-f32f2431797b\") " pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" Oct 30 13:03:31.349652 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:31.349598 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:31.417170 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:31.417085 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b487a069-f519-45a3-b1d8-f32f2431797b-model-cache\") pod \"simulator-w6-484607-kserve-85f449c7dd-b5gv5\" (UID: \"b487a069-f519-45a3-b1d8-f32f2431797b\") " pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" Oct 30 13:03:31.417170 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:31.417123 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b487a069-f519-45a3-b1d8-f32f2431797b-tls-certs\") pod \"simulator-w6-484607-kserve-85f449c7dd-b5gv5\" (UID: \"b487a069-f519-45a3-b1d8-f32f2431797b\") " pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" Oct 30 13:03:31.417170 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:31.417148 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b487a069-f519-45a3-b1d8-f32f2431797b-home\") pod \"simulator-w6-484607-kserve-85f449c7dd-b5gv5\" (UID: \"b487a069-f519-45a3-b1d8-f32f2431797b\") " pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" Oct 30 13:03:31.417422 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:31.417192 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b487a069-f519-45a3-b1d8-f32f2431797b-tmp-dir\") pod \"simulator-w6-484607-kserve-85f449c7dd-b5gv5\" (UID: \"b487a069-f519-45a3-b1d8-f32f2431797b\") " pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" Oct 30 13:03:31.417422 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:31.417233 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b487a069-f519-45a3-b1d8-f32f2431797b-dshm\") pod \"simulator-w6-484607-kserve-85f449c7dd-b5gv5\" (UID: \"b487a069-f519-45a3-b1d8-f32f2431797b\") " pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" Oct 30 13:03:31.417422 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:31.417285 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7twz\" (UniqueName: \"kubernetes.io/projected/b487a069-f519-45a3-b1d8-f32f2431797b-kube-api-access-v7twz\") pod \"simulator-w6-484607-kserve-85f449c7dd-b5gv5\" (UID: \"b487a069-f519-45a3-b1d8-f32f2431797b\") " pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" Oct 30 13:03:31.417693 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:31.417671 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b487a069-f519-45a3-b1d8-f32f2431797b-home\") pod \"simulator-w6-484607-kserve-85f449c7dd-b5gv5\" (UID: \"b487a069-f519-45a3-b1d8-f32f2431797b\") " pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" Oct 30 13:03:31.417751 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:31.417690 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b487a069-f519-45a3-b1d8-f32f2431797b-model-cache\") pod \"simulator-w6-484607-kserve-85f449c7dd-b5gv5\" (UID: \"b487a069-f519-45a3-b1d8-f32f2431797b\") " pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" Oct 30 13:03:31.417958 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:31.417929 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b487a069-f519-45a3-b1d8-f32f2431797b-tmp-dir\") pod \"simulator-w6-484607-kserve-85f449c7dd-b5gv5\" (UID: \"b487a069-f519-45a3-b1d8-f32f2431797b\") " pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" Oct 30 13:03:31.419875 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:31.419848 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b487a069-f519-45a3-b1d8-f32f2431797b-dshm\") pod \"simulator-w6-484607-kserve-85f449c7dd-b5gv5\" (UID: \"b487a069-f519-45a3-b1d8-f32f2431797b\") " pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" Oct 30 13:03:31.420413 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:31.420394 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b487a069-f519-45a3-b1d8-f32f2431797b-tls-certs\") pod \"simulator-w6-484607-kserve-85f449c7dd-b5gv5\" (UID: \"b487a069-f519-45a3-b1d8-f32f2431797b\") " pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" Oct 30 13:03:31.449632 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:31.449589 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7twz\" (UniqueName: \"kubernetes.io/projected/b487a069-f519-45a3-b1d8-f32f2431797b-kube-api-access-v7twz\") pod \"simulator-w6-484607-kserve-85f449c7dd-b5gv5\" (UID: \"b487a069-f519-45a3-b1d8-f32f2431797b\") " pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" Oct 30 13:03:31.467467 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:31.467450 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" Oct 30 13:03:31.779988 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:31.779957 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5"] Oct 30 13:03:31.783128 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:03:31.783091 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb487a069_f519_45a3_b1d8_f32f2431797b.slice/crio-9fc2085328042296bd03cde91b5ad97199a81d01bd400dd6d6a72e934d9c0eb1 WatchSource:0}: Error finding container 9fc2085328042296bd03cde91b5ad97199a81d01bd400dd6d6a72e934d9c0eb1: Status 404 returned error can't find the container with id 9fc2085328042296bd03cde91b5ad97199a81d01bd400dd6d6a72e934d9c0eb1 Oct 30 13:03:32.099162 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:32.099083 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:32.663588 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:32.663559 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" event={"ID":"b487a069-f519-45a3-b1d8-f32f2431797b","Type":"ContainerStarted","Data":"76966256847acca3935051c44da611a8c16596f36e378fb4dc8c1cc4a0791df2"} Oct 30 13:03:32.663588 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:32.663595 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" event={"ID":"b487a069-f519-45a3-b1d8-f32f2431797b","Type":"ContainerStarted","Data":"9fc2085328042296bd03cde91b5ad97199a81d01bd400dd6d6a72e934d9c0eb1"} Oct 30 13:03:32.711420 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:32.711369 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" podStartSLOduration=1.711354947 podStartE2EDuration="1.711354947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:32.710010472 +0000 UTC m=+669.179426218" watchObservedRunningTime="2026-10-30 13:03:32.711354947 +0000 UTC m=+669.180770691" Oct 30 13:03:32.801828 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:32.801781 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:33.175145 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:33.175112 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:33.468643 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:33.468608 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" Oct 30 13:03:33.755882 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:33.755800 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:34.038105 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:34.038021 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:03:34.361862 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:34.361765 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:34.499059 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:34.498986 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:34.638295 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:34.638194 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:35.137109 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:35.137060 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:36.935113 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:36.935086 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:38.902382 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:38.902334 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:39.743258 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:39.743223 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:40.643517 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:40.643484 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:41.467913 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:41.467881 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" Oct 30 13:03:41.473269 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:41.473226 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:41.484433 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:41.484407 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" Oct 30 13:03:41.596381 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:41.596346 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:41.724469 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:41.724397 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" Oct 30 13:03:41.886865 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:41.886836 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr"] Oct 30 13:03:41.890102 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:41.890081 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" Oct 30 13:03:41.907626 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:41.907581 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-54994c\"/\"openshift-service-ca.crt\"" Oct 30 13:03:41.907740 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:41.907677 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-54994c\"/\"simulator-w4-54994c-kserve-self-signed-certs\"" Oct 30 13:03:41.907740 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:41.907710 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-54994c\"/\"default-dockercfg-2msrl\"" Oct 30 13:03:41.907740 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:41.907735 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-54994c\"/\"kube-root-ca.crt\"" Oct 30 13:03:41.908850 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:41.908827 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr"] Oct 30 13:03:41.912155 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:41.912135 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35619ab9-cbeb-45da-8506-3af63a273177-dshm\") pod \"simulator-w4-54994c-kserve-f546dd889-h27lr\" (UID: \"35619ab9-cbeb-45da-8506-3af63a273177\") " pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" Oct 30 13:03:41.912241 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:41.912193 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35619ab9-cbeb-45da-8506-3af63a273177-home\") pod \"simulator-w4-54994c-kserve-f546dd889-h27lr\" (UID: \"35619ab9-cbeb-45da-8506-3af63a273177\") " pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" Oct 30 13:03:41.912300 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:41.912276 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35619ab9-cbeb-45da-8506-3af63a273177-tls-certs\") pod \"simulator-w4-54994c-kserve-f546dd889-h27lr\" (UID: \"35619ab9-cbeb-45da-8506-3af63a273177\") " pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" Oct 30 13:03:41.912352 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:41.912331 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35619ab9-cbeb-45da-8506-3af63a273177-model-cache\") pod \"simulator-w4-54994c-kserve-f546dd889-h27lr\" (UID: \"35619ab9-cbeb-45da-8506-3af63a273177\") " pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" Oct 30 13:03:41.912429 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:41.912406 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35619ab9-cbeb-45da-8506-3af63a273177-tmp-dir\") pod \"simulator-w4-54994c-kserve-f546dd889-h27lr\" (UID: \"35619ab9-cbeb-45da-8506-3af63a273177\") " pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" Oct 30 13:03:41.912486 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:41.912472 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlwzf\" (UniqueName: \"kubernetes.io/projected/35619ab9-cbeb-45da-8506-3af63a273177-kube-api-access-mlwzf\") pod \"simulator-w4-54994c-kserve-f546dd889-h27lr\" (UID: \"35619ab9-cbeb-45da-8506-3af63a273177\") " pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" Oct 30 13:03:41.948760 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:41.948734 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:42.013243 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:42.013161 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35619ab9-cbeb-45da-8506-3af63a273177-model-cache\") pod \"simulator-w4-54994c-kserve-f546dd889-h27lr\" (UID: \"35619ab9-cbeb-45da-8506-3af63a273177\") " pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" Oct 30 13:03:42.013368 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:42.013279 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35619ab9-cbeb-45da-8506-3af63a273177-tmp-dir\") pod \"simulator-w4-54994c-kserve-f546dd889-h27lr\" (UID: \"35619ab9-cbeb-45da-8506-3af63a273177\") " pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" Oct 30 13:03:42.013445 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:42.013373 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlwzf\" (UniqueName: \"kubernetes.io/projected/35619ab9-cbeb-45da-8506-3af63a273177-kube-api-access-mlwzf\") pod \"simulator-w4-54994c-kserve-f546dd889-h27lr\" (UID: \"35619ab9-cbeb-45da-8506-3af63a273177\") " pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" Oct 30 13:03:42.013529 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:42.013511 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35619ab9-cbeb-45da-8506-3af63a273177-dshm\") pod \"simulator-w4-54994c-kserve-f546dd889-h27lr\" (UID: \"35619ab9-cbeb-45da-8506-3af63a273177\") " pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" Oct 30 13:03:42.013589 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:42.013575 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35619ab9-cbeb-45da-8506-3af63a273177-home\") pod \"simulator-w4-54994c-kserve-f546dd889-h27lr\" (UID: \"35619ab9-cbeb-45da-8506-3af63a273177\") " pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" Oct 30 13:03:42.013657 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:42.013585 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35619ab9-cbeb-45da-8506-3af63a273177-model-cache\") pod \"simulator-w4-54994c-kserve-f546dd889-h27lr\" (UID: \"35619ab9-cbeb-45da-8506-3af63a273177\") " pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" Oct 30 13:03:42.013657 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:42.013599 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35619ab9-cbeb-45da-8506-3af63a273177-tls-certs\") pod \"simulator-w4-54994c-kserve-f546dd889-h27lr\" (UID: \"35619ab9-cbeb-45da-8506-3af63a273177\") " pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" Oct 30 13:03:42.013657 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:42.013642 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35619ab9-cbeb-45da-8506-3af63a273177-tmp-dir\") pod \"simulator-w4-54994c-kserve-f546dd889-h27lr\" (UID: \"35619ab9-cbeb-45da-8506-3af63a273177\") " pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" Oct 30 13:03:42.013953 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:42.013933 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35619ab9-cbeb-45da-8506-3af63a273177-home\") pod \"simulator-w4-54994c-kserve-f546dd889-h27lr\" (UID: \"35619ab9-cbeb-45da-8506-3af63a273177\") " pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" Oct 30 13:03:42.016842 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:42.016824 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35619ab9-cbeb-45da-8506-3af63a273177-dshm\") pod \"simulator-w4-54994c-kserve-f546dd889-h27lr\" (UID: \"35619ab9-cbeb-45da-8506-3af63a273177\") " pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" Oct 30 13:03:42.017205 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:42.017186 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35619ab9-cbeb-45da-8506-3af63a273177-tls-certs\") pod \"simulator-w4-54994c-kserve-f546dd889-h27lr\" (UID: \"35619ab9-cbeb-45da-8506-3af63a273177\") " pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" Oct 30 13:03:42.042176 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:42.041554 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlwzf\" (UniqueName: \"kubernetes.io/projected/35619ab9-cbeb-45da-8506-3af63a273177-kube-api-access-mlwzf\") pod \"simulator-w4-54994c-kserve-f546dd889-h27lr\" (UID: \"35619ab9-cbeb-45da-8506-3af63a273177\") " pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" Oct 30 13:03:42.203947 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:42.203913 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" Oct 30 13:03:42.347284 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:42.347198 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:42.506765 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:42.506740 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr"] Oct 30 13:03:42.509784 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:03:42.509758 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35619ab9_cbeb_45da_8506_3af63a273177.slice/crio-ba4385f4c37ed704a7a569f4799d7dc2443bdc4ebeb05a922b6e32beb5f67289 WatchSource:0}: Error finding container ba4385f4c37ed704a7a569f4799d7dc2443bdc4ebeb05a922b6e32beb5f67289: Status 404 returned error can't find the container with id ba4385f4c37ed704a7a569f4799d7dc2443bdc4ebeb05a922b6e32beb5f67289 Oct 30 13:03:42.721373 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:42.721334 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" event={"ID":"35619ab9-cbeb-45da-8506-3af63a273177","Type":"ContainerStarted","Data":"c9da677773175ee3fc9879e486f99b1fa783cce48bb86126759c48c48c28305c"} Oct 30 13:03:42.721373 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:42.721379 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" event={"ID":"35619ab9-cbeb-45da-8506-3af63a273177","Type":"ContainerStarted","Data":"ba4385f4c37ed704a7a569f4799d7dc2443bdc4ebeb05a922b6e32beb5f67289"} Oct 30 13:03:42.767379 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:42.767329 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" podStartSLOduration=1.7673150340000001 podStartE2EDuration="1.767315034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:42.766076226 +0000 UTC m=+679.235491995" watchObservedRunningTime="2026-10-30 13:03:42.767315034 +0000 UTC m=+679.236730776" Oct 30 13:03:43.204736 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:43.204704 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" Oct 30 13:03:44.130662 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:44.130627 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:45.962131 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:45.962094 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:47.627598 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:47.627569 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp"] Oct 30 13:03:47.630648 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:47.630599 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" Oct 30 13:03:47.660765 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:47.660740 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp"] Oct 30 13:03:47.671125 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:47.671099 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-3048f9\"/\"premium-w3-3048f9-kserve-self-signed-certs\"" Oct 30 13:03:47.671466 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:47.671444 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-665pc\" (UniqueName: \"kubernetes.io/projected/5f353928-8cb4-49a3-b999-1b90e6cb31bb-kube-api-access-665pc\") pod \"premium-w3-3048f9-kserve-5dbc4b8c7-wscwp\" (UID: \"5f353928-8cb4-49a3-b999-1b90e6cb31bb\") " pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" Oct 30 13:03:47.671632 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:47.671591 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f353928-8cb4-49a3-b999-1b90e6cb31bb-dshm\") pod \"premium-w3-3048f9-kserve-5dbc4b8c7-wscwp\" (UID: \"5f353928-8cb4-49a3-b999-1b90e6cb31bb\") " pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" Oct 30 13:03:47.671714 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:47.671695 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f353928-8cb4-49a3-b999-1b90e6cb31bb-tls-certs\") pod \"premium-w3-3048f9-kserve-5dbc4b8c7-wscwp\" (UID: \"5f353928-8cb4-49a3-b999-1b90e6cb31bb\") " pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" Oct 30 13:03:47.671804 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:47.671789 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f353928-8cb4-49a3-b999-1b90e6cb31bb-home\") pod \"premium-w3-3048f9-kserve-5dbc4b8c7-wscwp\" (UID: \"5f353928-8cb4-49a3-b999-1b90e6cb31bb\") " pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" Oct 30 13:03:47.671863 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:47.671818 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f353928-8cb4-49a3-b999-1b90e6cb31bb-model-cache\") pod \"premium-w3-3048f9-kserve-5dbc4b8c7-wscwp\" (UID: \"5f353928-8cb4-49a3-b999-1b90e6cb31bb\") " pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" Oct 30 13:03:47.671946 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:47.671929 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f353928-8cb4-49a3-b999-1b90e6cb31bb-tmp-dir\") pod \"premium-w3-3048f9-kserve-5dbc4b8c7-wscwp\" (UID: \"5f353928-8cb4-49a3-b999-1b90e6cb31bb\") " pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" Oct 30 13:03:47.709566 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:47.709542 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:47.773153 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:47.773119 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-665pc\" (UniqueName: \"kubernetes.io/projected/5f353928-8cb4-49a3-b999-1b90e6cb31bb-kube-api-access-665pc\") pod \"premium-w3-3048f9-kserve-5dbc4b8c7-wscwp\" (UID: \"5f353928-8cb4-49a3-b999-1b90e6cb31bb\") " pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" Oct 30 13:03:47.773316 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:47.773159 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f353928-8cb4-49a3-b999-1b90e6cb31bb-dshm\") pod \"premium-w3-3048f9-kserve-5dbc4b8c7-wscwp\" (UID: \"5f353928-8cb4-49a3-b999-1b90e6cb31bb\") " pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" Oct 30 13:03:47.773316 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:47.773184 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f353928-8cb4-49a3-b999-1b90e6cb31bb-tls-certs\") pod \"premium-w3-3048f9-kserve-5dbc4b8c7-wscwp\" (UID: \"5f353928-8cb4-49a3-b999-1b90e6cb31bb\") " pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" Oct 30 13:03:47.773316 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:47.773214 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f353928-8cb4-49a3-b999-1b90e6cb31bb-home\") pod \"premium-w3-3048f9-kserve-5dbc4b8c7-wscwp\" (UID: \"5f353928-8cb4-49a3-b999-1b90e6cb31bb\") " pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" Oct 30 13:03:47.773316 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:47.773232 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f353928-8cb4-49a3-b999-1b90e6cb31bb-model-cache\") pod \"premium-w3-3048f9-kserve-5dbc4b8c7-wscwp\" (UID: \"5f353928-8cb4-49a3-b999-1b90e6cb31bb\") " pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" Oct 30 13:03:47.773316 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:47.773278 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f353928-8cb4-49a3-b999-1b90e6cb31bb-tmp-dir\") pod \"premium-w3-3048f9-kserve-5dbc4b8c7-wscwp\" (UID: \"5f353928-8cb4-49a3-b999-1b90e6cb31bb\") " pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" Oct 30 13:03:47.773748 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:47.773720 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f353928-8cb4-49a3-b999-1b90e6cb31bb-model-cache\") pod \"premium-w3-3048f9-kserve-5dbc4b8c7-wscwp\" (UID: \"5f353928-8cb4-49a3-b999-1b90e6cb31bb\") " pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" Oct 30 13:03:47.773837 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:47.773732 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f353928-8cb4-49a3-b999-1b90e6cb31bb-home\") pod \"premium-w3-3048f9-kserve-5dbc4b8c7-wscwp\" (UID: \"5f353928-8cb4-49a3-b999-1b90e6cb31bb\") " pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" Oct 30 13:03:47.773837 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:47.773785 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f353928-8cb4-49a3-b999-1b90e6cb31bb-tmp-dir\") pod \"premium-w3-3048f9-kserve-5dbc4b8c7-wscwp\" (UID: \"5f353928-8cb4-49a3-b999-1b90e6cb31bb\") " pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" Oct 30 13:03:47.775436 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:47.775412 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f353928-8cb4-49a3-b999-1b90e6cb31bb-dshm\") pod \"premium-w3-3048f9-kserve-5dbc4b8c7-wscwp\" (UID: \"5f353928-8cb4-49a3-b999-1b90e6cb31bb\") " pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" Oct 30 13:03:47.775730 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:47.775714 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f353928-8cb4-49a3-b999-1b90e6cb31bb-tls-certs\") pod \"premium-w3-3048f9-kserve-5dbc4b8c7-wscwp\" (UID: \"5f353928-8cb4-49a3-b999-1b90e6cb31bb\") " pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" Oct 30 13:03:47.799721 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:47.799690 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-665pc\" (UniqueName: \"kubernetes.io/projected/5f353928-8cb4-49a3-b999-1b90e6cb31bb-kube-api-access-665pc\") pod \"premium-w3-3048f9-kserve-5dbc4b8c7-wscwp\" (UID: \"5f353928-8cb4-49a3-b999-1b90e6cb31bb\") " pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" Oct 30 13:03:47.943432 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:47.943409 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" Oct 30 13:03:48.086785 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:48.086746 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:48.244254 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:48.244176 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp"] Oct 30 13:03:48.247216 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:03:48.247192 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f353928_8cb4_49a3_b999_1b90e6cb31bb.slice/crio-5b16afcd292ec72f4f11bbfe127239fffeec210ccc27b7065750172afa11ee49 WatchSource:0}: Error finding container 5b16afcd292ec72f4f11bbfe127239fffeec210ccc27b7065750172afa11ee49: Status 404 returned error can't find the container with id 5b16afcd292ec72f4f11bbfe127239fffeec210ccc27b7065750172afa11ee49 Oct 30 13:03:48.756598 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:48.756560 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" event={"ID":"5f353928-8cb4-49a3-b999-1b90e6cb31bb","Type":"ContainerStarted","Data":"a028881b68f42e00fc7c15e02e711bd3904046f6c22ec739ab99fe6a6dc8dbb7"} Oct 30 13:03:48.756997 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:48.756622 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" event={"ID":"5f353928-8cb4-49a3-b999-1b90e6cb31bb","Type":"ContainerStarted","Data":"5b16afcd292ec72f4f11bbfe127239fffeec210ccc27b7065750172afa11ee49"} Oct 30 13:03:48.813023 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:48.812982 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" podStartSLOduration=1.81296816 podStartE2EDuration="1.81296816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:48.812725141 +0000 UTC m=+685.282140886" watchObservedRunningTime="2026-10-30 13:03:48.81296816 +0000 UTC m=+685.282383905" Oct 30 13:03:48.943551 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:48.943518 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" Oct 30 13:03:49.891277 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:49.891239 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:51.634859 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:51.634829 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:03:51.744720 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:51.744679 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:52.204679 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:52.204646 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" Oct 30 13:03:52.218347 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:52.218309 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" Oct 30 13:03:52.225491 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:52.225447 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:52.841635 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:52.840910 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" Oct 30 13:03:53.557969 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:03:53.557934 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:03:53.724030 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:53.723989 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:55.451241 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:55.451184 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:57.346862 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:57.346823 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:57.944232 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:57.944201 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" Oct 30 13:03:57.956565 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:57.956545 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" Oct 30 13:03:58.838187 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:58.838161 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" Oct 30 13:03:59.389584 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:03:59.389543 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:01.257531 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:01.257497 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:01.825436 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:01.825403 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg"] Oct 30 13:04:01.829684 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:01.829650 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" Oct 30 13:04:01.854241 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:01.854214 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg"] Oct 30 13:04:01.860124 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:01.860099 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-bb71e2\"/\"openshift-service-ca.crt\"" Oct 30 13:04:01.860124 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:01.860100 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-bb71e2\"/\"kube-root-ca.crt\"" Oct 30 13:04:01.860509 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:01.860490 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-bb71e2\"/\"default-dockercfg-8lldv\"" Oct 30 13:04:01.860587 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:01.860549 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-bb71e2\"/\"premium-w2-bb71e2-kserve-self-signed-certs\"" Oct 30 13:04:01.900194 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:01.900148 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:01.929871 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:01.929840 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a85de36b-f8c8-4ade-a468-c114080cb312-model-cache\") pod \"premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg\" (UID: \"a85de36b-f8c8-4ade-a468-c114080cb312\") " pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" Oct 30 13:04:01.929988 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:01.929875 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27rdg\" (UniqueName: \"kubernetes.io/projected/a85de36b-f8c8-4ade-a468-c114080cb312-kube-api-access-27rdg\") pod \"premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg\" (UID: \"a85de36b-f8c8-4ade-a468-c114080cb312\") " pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" Oct 30 13:04:01.930050 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:01.930031 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a85de36b-f8c8-4ade-a468-c114080cb312-tls-certs\") pod \"premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg\" (UID: \"a85de36b-f8c8-4ade-a468-c114080cb312\") " pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" Oct 30 13:04:01.930091 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:01.930068 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a85de36b-f8c8-4ade-a468-c114080cb312-home\") pod \"premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg\" (UID: \"a85de36b-f8c8-4ade-a468-c114080cb312\") " pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" Oct 30 13:04:01.930122 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:01.930111 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a85de36b-f8c8-4ade-a468-c114080cb312-dshm\") pod \"premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg\" (UID: \"a85de36b-f8c8-4ade-a468-c114080cb312\") " pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" Oct 30 13:04:01.930525 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:01.930134 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a85de36b-f8c8-4ade-a468-c114080cb312-tmp-dir\") pod \"premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg\" (UID: \"a85de36b-f8c8-4ade-a468-c114080cb312\") " pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" Oct 30 13:04:02.031059 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:02.031023 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a85de36b-f8c8-4ade-a468-c114080cb312-tls-certs\") pod \"premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg\" (UID: \"a85de36b-f8c8-4ade-a468-c114080cb312\") " pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" Oct 30 13:04:02.031059 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:02.031072 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a85de36b-f8c8-4ade-a468-c114080cb312-home\") pod \"premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg\" (UID: \"a85de36b-f8c8-4ade-a468-c114080cb312\") " pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" Oct 30 13:04:02.031336 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:02.031129 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a85de36b-f8c8-4ade-a468-c114080cb312-dshm\") pod \"premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg\" (UID: \"a85de36b-f8c8-4ade-a468-c114080cb312\") " pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" Oct 30 13:04:02.031336 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:02.031158 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a85de36b-f8c8-4ade-a468-c114080cb312-tmp-dir\") pod \"premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg\" (UID: \"a85de36b-f8c8-4ade-a468-c114080cb312\") " pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" Oct 30 13:04:02.031336 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:02.031249 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a85de36b-f8c8-4ade-a468-c114080cb312-model-cache\") pod \"premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg\" (UID: \"a85de36b-f8c8-4ade-a468-c114080cb312\") " pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" Oct 30 13:04:02.031336 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:02.031273 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27rdg\" (UniqueName: \"kubernetes.io/projected/a85de36b-f8c8-4ade-a468-c114080cb312-kube-api-access-27rdg\") pod \"premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg\" (UID: \"a85de36b-f8c8-4ade-a468-c114080cb312\") " pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" Oct 30 13:04:02.031923 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:02.031894 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a85de36b-f8c8-4ade-a468-c114080cb312-model-cache\") pod \"premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg\" (UID: \"a85de36b-f8c8-4ade-a468-c114080cb312\") " pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" Oct 30 13:04:02.032208 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:02.032178 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a85de36b-f8c8-4ade-a468-c114080cb312-tmp-dir\") pod \"premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg\" (UID: \"a85de36b-f8c8-4ade-a468-c114080cb312\") " pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" Oct 30 13:04:02.032640 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:02.032553 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a85de36b-f8c8-4ade-a468-c114080cb312-home\") pod \"premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg\" (UID: \"a85de36b-f8c8-4ade-a468-c114080cb312\") " pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" Oct 30 13:04:02.034006 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:02.033984 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a85de36b-f8c8-4ade-a468-c114080cb312-tls-certs\") pod \"premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg\" (UID: \"a85de36b-f8c8-4ade-a468-c114080cb312\") " pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" Oct 30 13:04:02.034341 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:02.034315 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a85de36b-f8c8-4ade-a468-c114080cb312-dshm\") pod \"premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg\" (UID: \"a85de36b-f8c8-4ade-a468-c114080cb312\") " pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" Oct 30 13:04:02.058602 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:02.058577 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27rdg\" (UniqueName: \"kubernetes.io/projected/a85de36b-f8c8-4ade-a468-c114080cb312-kube-api-access-27rdg\") pod \"premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg\" (UID: \"a85de36b-f8c8-4ade-a468-c114080cb312\") " pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" Oct 30 13:04:02.144145 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:02.144082 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" Oct 30 13:04:02.441374 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:02.441301 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg"] Oct 30 13:04:02.444323 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:04:02.444293 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda85de36b_f8c8_4ade_a468_c114080cb312.slice/crio-13000c8adfdc1ca90c644548d10c381c9080dd4aa972ea8dd0d70c6e92a0987c WatchSource:0}: Error finding container 13000c8adfdc1ca90c644548d10c381c9080dd4aa972ea8dd0d70c6e92a0987c: Status 404 returned error can't find the container with id 13000c8adfdc1ca90c644548d10c381c9080dd4aa972ea8dd0d70c6e92a0987c Oct 30 13:04:02.857981 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:02.857949 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" event={"ID":"a85de36b-f8c8-4ade-a468-c114080cb312","Type":"ContainerStarted","Data":"b814923153b3fa16975c2e0f656cf351391cfd1f88e77b0e2c6fe56e40626e67"} Oct 30 13:04:02.858152 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:02.857991 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" event={"ID":"a85de36b-f8c8-4ade-a468-c114080cb312","Type":"ContainerStarted","Data":"13000c8adfdc1ca90c644548d10c381c9080dd4aa972ea8dd0d70c6e92a0987c"} Oct 30 13:04:02.902314 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:02.902267 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" podStartSLOduration=1.902254268 podStartE2EDuration="1.902254268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:02.900819115 +0000 UTC m=+699.370234861" watchObservedRunningTime="2026-10-30 13:04:02.902254268 +0000 UTC m=+699.371670013" Oct 30 13:04:03.144320 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:03.144247 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" Oct 30 13:04:03.222528 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:03.222488 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:04.998521 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:04.998485 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:06.956896 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:06.956855 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:06.968006 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:06.967973 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:04:08.760127 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:08.760100 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:09.666679 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:09.666646 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:10.155796 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:10.155744 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:11.747551 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:11.747516 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:12.144426 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:12.144353 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" Oct 30 13:04:12.156363 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:12.156338 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" Oct 30 13:04:12.926108 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:12.926075 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" Oct 30 13:04:13.671862 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:13.671832 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:14.486301 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:14.486261 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:15.503923 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:15.503885 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:17.426121 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:17.426070 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:19.304657 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:19.304602 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:19.638911 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:19.638817 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm"] Oct 30 13:04:19.654185 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:19.654144 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" Oct 30 13:04:19.675812 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:19.675786 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-42ee9d\"/\"openshift-service-ca.crt\"" Oct 30 13:04:19.687549 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:19.687521 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-42ee9d\"/\"default-dockercfg-dn7vg\"" Oct 30 13:04:19.687809 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:19.687793 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-42ee9d\"/\"kube-root-ca.crt\"" Oct 30 13:04:19.688041 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:19.688024 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-42ee9d\"/\"premium-w0-42ee9d-kserve-self-signed-certs\"" Oct 30 13:04:19.741633 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:19.737193 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfd83cc8-6503-45a8-8260-c0edb8752471-model-cache\") pod \"premium-w0-42ee9d-kserve-b4df54c7c-4sskm\" (UID: \"dfd83cc8-6503-45a8-8260-c0edb8752471\") " pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" Oct 30 13:04:19.741633 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:19.737257 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfd83cc8-6503-45a8-8260-c0edb8752471-dshm\") pod \"premium-w0-42ee9d-kserve-b4df54c7c-4sskm\" (UID: \"dfd83cc8-6503-45a8-8260-c0edb8752471\") " pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" Oct 30 13:04:19.741633 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:19.737310 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfd83cc8-6503-45a8-8260-c0edb8752471-home\") pod \"premium-w0-42ee9d-kserve-b4df54c7c-4sskm\" (UID: \"dfd83cc8-6503-45a8-8260-c0edb8752471\") " pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" Oct 30 13:04:19.741633 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:19.737366 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfd83cc8-6503-45a8-8260-c0edb8752471-tmp-dir\") pod \"premium-w0-42ee9d-kserve-b4df54c7c-4sskm\" (UID: \"dfd83cc8-6503-45a8-8260-c0edb8752471\") " pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" Oct 30 13:04:19.741633 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:19.737461 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfd83cc8-6503-45a8-8260-c0edb8752471-tls-certs\") pod \"premium-w0-42ee9d-kserve-b4df54c7c-4sskm\" (UID: \"dfd83cc8-6503-45a8-8260-c0edb8752471\") " pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" Oct 30 13:04:19.741633 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:19.737530 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59crd\" (UniqueName: \"kubernetes.io/projected/dfd83cc8-6503-45a8-8260-c0edb8752471-kube-api-access-59crd\") pod \"premium-w0-42ee9d-kserve-b4df54c7c-4sskm\" (UID: \"dfd83cc8-6503-45a8-8260-c0edb8752471\") " pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" Oct 30 13:04:19.741633 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:19.737977 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:19.742780 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:19.742754 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm"] Oct 30 13:04:19.849261 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:19.849225 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfd83cc8-6503-45a8-8260-c0edb8752471-home\") pod \"premium-w0-42ee9d-kserve-b4df54c7c-4sskm\" (UID: \"dfd83cc8-6503-45a8-8260-c0edb8752471\") " pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" Oct 30 13:04:19.851077 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:19.851049 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfd83cc8-6503-45a8-8260-c0edb8752471-tmp-dir\") pod \"premium-w0-42ee9d-kserve-b4df54c7c-4sskm\" (UID: \"dfd83cc8-6503-45a8-8260-c0edb8752471\") " pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" Oct 30 13:04:19.851736 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:19.851717 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfd83cc8-6503-45a8-8260-c0edb8752471-tls-certs\") pod \"premium-w0-42ee9d-kserve-b4df54c7c-4sskm\" (UID: \"dfd83cc8-6503-45a8-8260-c0edb8752471\") " pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" Oct 30 13:04:19.852320 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:19.852301 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59crd\" (UniqueName: \"kubernetes.io/projected/dfd83cc8-6503-45a8-8260-c0edb8752471-kube-api-access-59crd\") pod \"premium-w0-42ee9d-kserve-b4df54c7c-4sskm\" (UID: \"dfd83cc8-6503-45a8-8260-c0edb8752471\") " pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" Oct 30 13:04:19.852781 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:19.852761 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfd83cc8-6503-45a8-8260-c0edb8752471-model-cache\") pod \"premium-w0-42ee9d-kserve-b4df54c7c-4sskm\" (UID: \"dfd83cc8-6503-45a8-8260-c0edb8752471\") " pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" Oct 30 13:04:19.852959 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:19.852931 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfd83cc8-6503-45a8-8260-c0edb8752471-dshm\") pod \"premium-w0-42ee9d-kserve-b4df54c7c-4sskm\" (UID: \"dfd83cc8-6503-45a8-8260-c0edb8752471\") " pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" Oct 30 13:04:19.853666 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:19.853645 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfd83cc8-6503-45a8-8260-c0edb8752471-model-cache\") pod \"premium-w0-42ee9d-kserve-b4df54c7c-4sskm\" (UID: \"dfd83cc8-6503-45a8-8260-c0edb8752471\") " pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" Oct 30 13:04:19.853794 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:19.850942 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfd83cc8-6503-45a8-8260-c0edb8752471-home\") pod \"premium-w0-42ee9d-kserve-b4df54c7c-4sskm\" (UID: \"dfd83cc8-6503-45a8-8260-c0edb8752471\") " pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" Oct 30 13:04:19.853877 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:19.851564 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfd83cc8-6503-45a8-8260-c0edb8752471-tmp-dir\") pod \"premium-w0-42ee9d-kserve-b4df54c7c-4sskm\" (UID: \"dfd83cc8-6503-45a8-8260-c0edb8752471\") " pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" Oct 30 13:04:19.864730 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:19.864710 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfd83cc8-6503-45a8-8260-c0edb8752471-dshm\") pod \"premium-w0-42ee9d-kserve-b4df54c7c-4sskm\" (UID: \"dfd83cc8-6503-45a8-8260-c0edb8752471\") " pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" Oct 30 13:04:19.867357 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:19.867321 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfd83cc8-6503-45a8-8260-c0edb8752471-tls-certs\") pod \"premium-w0-42ee9d-kserve-b4df54c7c-4sskm\" (UID: \"dfd83cc8-6503-45a8-8260-c0edb8752471\") " pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" Oct 30 13:04:19.904679 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:19.901296 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59crd\" (UniqueName: \"kubernetes.io/projected/dfd83cc8-6503-45a8-8260-c0edb8752471-kube-api-access-59crd\") pod \"premium-w0-42ee9d-kserve-b4df54c7c-4sskm\" (UID: \"dfd83cc8-6503-45a8-8260-c0edb8752471\") " pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" Oct 30 13:04:20.025891 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:20.025846 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" Oct 30 13:04:20.449244 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:20.449219 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm"] Oct 30 13:04:20.452673 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:04:20.452645 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddfd83cc8_6503_45a8_8260_c0edb8752471.slice/crio-ce1832b714c4052e537b824d135a0906451acdb891980a959fc750e2f05abb10 WatchSource:0}: Error finding container ce1832b714c4052e537b824d135a0906451acdb891980a959fc750e2f05abb10: Status 404 returned error can't find the container with id ce1832b714c4052e537b824d135a0906451acdb891980a959fc750e2f05abb10 Oct 30 13:04:20.969249 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:20.969212 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" event={"ID":"dfd83cc8-6503-45a8-8260-c0edb8752471","Type":"ContainerStarted","Data":"311c1aae849d63e5bfcd7772c98b907f696d432ea84d0c75175c8cc3d5836f54"} Oct 30 13:04:20.969452 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:20.969254 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" event={"ID":"dfd83cc8-6503-45a8-8260-c0edb8752471","Type":"ContainerStarted","Data":"ce1832b714c4052e537b824d135a0906451acdb891980a959fc750e2f05abb10"} Oct 30 13:04:21.014578 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:21.014532 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" podStartSLOduration=2.01451412 podStartE2EDuration="2.01451412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:21.012411794 +0000 UTC m=+717.481827539" watchObservedRunningTime="2026-10-30 13:04:21.01451412 +0000 UTC m=+717.483929866" Oct 30 13:04:21.027535 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:21.027511 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" Oct 30 13:04:21.207639 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:21.207573 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:22.986093 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:22.985934 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:23.553590 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:04:23.553546 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:04:24.847809 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:24.847778 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86"] Oct 30 13:04:24.851708 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:24.851683 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" Oct 30 13:04:24.870010 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:24.869990 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86"] Oct 30 13:04:24.880238 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:24.880211 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:24.880360 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:24.880338 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-54994c\"/\"premium-w4-54994c-kserve-self-signed-certs\"" Oct 30 13:04:24.916155 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:24.916132 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7751779d-b082-409a-a20f-ff0ff6263808-dshm\") pod \"premium-w4-54994c-kserve-5c69d8c5bd-hbs86\" (UID: \"7751779d-b082-409a-a20f-ff0ff6263808\") " pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" Oct 30 13:04:24.916330 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:24.916313 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7751779d-b082-409a-a20f-ff0ff6263808-home\") pod \"premium-w4-54994c-kserve-5c69d8c5bd-hbs86\" (UID: \"7751779d-b082-409a-a20f-ff0ff6263808\") " pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" Oct 30 13:04:24.916398 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:24.916344 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7751779d-b082-409a-a20f-ff0ff6263808-tmp-dir\") pod \"premium-w4-54994c-kserve-5c69d8c5bd-hbs86\" (UID: \"7751779d-b082-409a-a20f-ff0ff6263808\") " pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" Oct 30 13:04:24.916483 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:24.916461 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7751779d-b082-409a-a20f-ff0ff6263808-tls-certs\") pod \"premium-w4-54994c-kserve-5c69d8c5bd-hbs86\" (UID: \"7751779d-b082-409a-a20f-ff0ff6263808\") " pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" Oct 30 13:04:24.916552 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:24.916535 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2bn2\" (UniqueName: \"kubernetes.io/projected/7751779d-b082-409a-a20f-ff0ff6263808-kube-api-access-s2bn2\") pod \"premium-w4-54994c-kserve-5c69d8c5bd-hbs86\" (UID: \"7751779d-b082-409a-a20f-ff0ff6263808\") " pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" Oct 30 13:04:24.916591 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:24.916573 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7751779d-b082-409a-a20f-ff0ff6263808-model-cache\") pod \"premium-w4-54994c-kserve-5c69d8c5bd-hbs86\" (UID: \"7751779d-b082-409a-a20f-ff0ff6263808\") " pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" Oct 30 13:04:25.011094 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:25.011069 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:25.017266 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:25.017236 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7751779d-b082-409a-a20f-ff0ff6263808-home\") pod \"premium-w4-54994c-kserve-5c69d8c5bd-hbs86\" (UID: \"7751779d-b082-409a-a20f-ff0ff6263808\") " pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" Oct 30 13:04:25.017367 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:25.017268 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7751779d-b082-409a-a20f-ff0ff6263808-tmp-dir\") pod \"premium-w4-54994c-kserve-5c69d8c5bd-hbs86\" (UID: \"7751779d-b082-409a-a20f-ff0ff6263808\") " pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" Oct 30 13:04:25.017427 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:25.017409 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7751779d-b082-409a-a20f-ff0ff6263808-tls-certs\") pod \"premium-w4-54994c-kserve-5c69d8c5bd-hbs86\" (UID: \"7751779d-b082-409a-a20f-ff0ff6263808\") " pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" Oct 30 13:04:25.017476 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:25.017462 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2bn2\" (UniqueName: \"kubernetes.io/projected/7751779d-b082-409a-a20f-ff0ff6263808-kube-api-access-s2bn2\") pod \"premium-w4-54994c-kserve-5c69d8c5bd-hbs86\" (UID: \"7751779d-b082-409a-a20f-ff0ff6263808\") " pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" Oct 30 13:04:25.017512 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:25.017493 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7751779d-b082-409a-a20f-ff0ff6263808-model-cache\") pod \"premium-w4-54994c-kserve-5c69d8c5bd-hbs86\" (UID: \"7751779d-b082-409a-a20f-ff0ff6263808\") " pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" Oct 30 13:04:25.017545 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:25.017533 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7751779d-b082-409a-a20f-ff0ff6263808-dshm\") pod \"premium-w4-54994c-kserve-5c69d8c5bd-hbs86\" (UID: \"7751779d-b082-409a-a20f-ff0ff6263808\") " pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" Oct 30 13:04:25.017725 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:25.017701 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7751779d-b082-409a-a20f-ff0ff6263808-tmp-dir\") pod \"premium-w4-54994c-kserve-5c69d8c5bd-hbs86\" (UID: \"7751779d-b082-409a-a20f-ff0ff6263808\") " pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" Oct 30 13:04:25.017836 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:25.017743 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7751779d-b082-409a-a20f-ff0ff6263808-home\") pod \"premium-w4-54994c-kserve-5c69d8c5bd-hbs86\" (UID: \"7751779d-b082-409a-a20f-ff0ff6263808\") " pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" Oct 30 13:04:25.020007 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:25.019979 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7751779d-b082-409a-a20f-ff0ff6263808-dshm\") pod \"premium-w4-54994c-kserve-5c69d8c5bd-hbs86\" (UID: \"7751779d-b082-409a-a20f-ff0ff6263808\") " pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" Oct 30 13:04:25.020568 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:25.020525 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7751779d-b082-409a-a20f-ff0ff6263808-tls-certs\") pod \"premium-w4-54994c-kserve-5c69d8c5bd-hbs86\" (UID: \"7751779d-b082-409a-a20f-ff0ff6263808\") " pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" Oct 30 13:04:25.020900 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:25.020881 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7751779d-b082-409a-a20f-ff0ff6263808-model-cache\") pod \"premium-w4-54994c-kserve-5c69d8c5bd-hbs86\" (UID: \"7751779d-b082-409a-a20f-ff0ff6263808\") " pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" Oct 30 13:04:25.045640 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:25.045574 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2bn2\" (UniqueName: \"kubernetes.io/projected/7751779d-b082-409a-a20f-ff0ff6263808-kube-api-access-s2bn2\") pod \"premium-w4-54994c-kserve-5c69d8c5bd-hbs86\" (UID: \"7751779d-b082-409a-a20f-ff0ff6263808\") " pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" Oct 30 13:04:25.165032 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:25.164952 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" Oct 30 13:04:25.470261 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:25.470235 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86"] Oct 30 13:04:25.473989 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:04:25.473962 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7751779d_b082_409a_a20f_ff0ff6263808.slice/crio-916756c58cca3da6d680d85426798484ebcbeed1257e6ba1b6fc2eb1ed9e2701 WatchSource:0}: Error finding container 916756c58cca3da6d680d85426798484ebcbeed1257e6ba1b6fc2eb1ed9e2701: Status 404 returned error can't find the container with id 916756c58cca3da6d680d85426798484ebcbeed1257e6ba1b6fc2eb1ed9e2701 Oct 30 13:04:26.003921 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:26.003888 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" event={"ID":"7751779d-b082-409a-a20f-ff0ff6263808","Type":"ContainerStarted","Data":"f5908974c0c7d89bba23811e14edabc6cf167e63cc0957868f331697322b9c40"} Oct 30 13:04:26.004325 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:26.003930 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" event={"ID":"7751779d-b082-409a-a20f-ff0ff6263808","Type":"ContainerStarted","Data":"916756c58cca3da6d680d85426798484ebcbeed1257e6ba1b6fc2eb1ed9e2701"} Oct 30 13:04:26.048800 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:26.048750 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" podStartSLOduration=2.048737532 podStartE2EDuration="2.048737532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:26.048066942 +0000 UTC m=+722.517482687" watchObservedRunningTime="2026-10-30 13:04:26.048737532 +0000 UTC m=+722.518153274" Oct 30 13:04:26.166087 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:26.166041 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" Oct 30 13:04:26.709889 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:26.709850 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:28.545465 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:28.545427 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:29.434887 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:29.434848 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:04:30.026738 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:30.026702 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" Oct 30 13:04:30.040626 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:30.040587 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" Oct 30 13:04:30.099199 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:30.080187 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dad25b3a-8jt96"] Oct 30 13:04:30.099199 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:30.089084 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dad25b3a-8jt96" Oct 30 13:04:30.108737 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:30.108707 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:04:30.109838 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:30.109027 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:04:30.109838 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:30.109150 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:04:30.109838 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:30.109207 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:04:30.111066 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:30.111036 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dad25b3a-8jt96"] Oct 30 13:04:30.195467 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:30.195414 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50735aee-920f-4412-a82e-655ce2c43094-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dad25b3a-8jt96\" (UID: \"50735aee-920f-4412-a82e-655ce2c43094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dad25b3a-8jt96" Oct 30 13:04:30.195655 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:30.195546 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67kp8\" (UniqueName: \"kubernetes.io/projected/50735aee-920f-4412-a82e-655ce2c43094-kube-api-access-67kp8\") pod \"maas-api-revoke-sub-dad25b3a-8jt96\" (UID: \"50735aee-920f-4412-a82e-655ce2c43094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dad25b3a-8jt96" Oct 30 13:04:30.195884 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:30.195858 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50735aee-920f-4412-a82e-655ce2c43094-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dad25b3a-8jt96\" (UID: \"50735aee-920f-4412-a82e-655ce2c43094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dad25b3a-8jt96" Oct 30 13:04:30.297183 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:30.297098 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50735aee-920f-4412-a82e-655ce2c43094-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dad25b3a-8jt96\" (UID: \"50735aee-920f-4412-a82e-655ce2c43094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dad25b3a-8jt96" Oct 30 13:04:30.297183 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:30.297145 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50735aee-920f-4412-a82e-655ce2c43094-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dad25b3a-8jt96\" (UID: \"50735aee-920f-4412-a82e-655ce2c43094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dad25b3a-8jt96" Oct 30 13:04:30.297382 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:30.297197 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67kp8\" (UniqueName: \"kubernetes.io/projected/50735aee-920f-4412-a82e-655ce2c43094-kube-api-access-67kp8\") pod \"maas-api-revoke-sub-dad25b3a-8jt96\" (UID: \"50735aee-920f-4412-a82e-655ce2c43094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dad25b3a-8jt96" Oct 30 13:04:30.298201 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:30.298146 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50735aee-920f-4412-a82e-655ce2c43094-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dad25b3a-8jt96\" (UID: \"50735aee-920f-4412-a82e-655ce2c43094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dad25b3a-8jt96" Oct 30 13:04:30.305650 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:30.300880 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50735aee-920f-4412-a82e-655ce2c43094-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dad25b3a-8jt96\" (UID: \"50735aee-920f-4412-a82e-655ce2c43094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dad25b3a-8jt96" Oct 30 13:04:30.325212 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:30.325182 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67kp8\" (UniqueName: \"kubernetes.io/projected/50735aee-920f-4412-a82e-655ce2c43094-kube-api-access-67kp8\") pod \"maas-api-revoke-sub-dad25b3a-8jt96\" (UID: \"50735aee-920f-4412-a82e-655ce2c43094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dad25b3a-8jt96" Oct 30 13:04:30.411049 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:30.411018 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dad25b3a-8jt96" Oct 30 13:04:30.724578 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:30.724540 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:30.733459 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:30.733430 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dad25b3a-8jt96"] Oct 30 13:04:30.737011 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:04:30.736684 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50735aee_920f_4412_a82e_655ce2c43094.slice/crio-b88a83b0fa47758c1128c755363f13ce863ce5b4be12e3395215d623981cf470 WatchSource:0}: Error finding container b88a83b0fa47758c1128c755363f13ce863ce5b4be12e3395215d623981cf470: Status 404 returned error can't find the container with id b88a83b0fa47758c1128c755363f13ce863ce5b4be12e3395215d623981cf470 Oct 30 13:04:31.036037 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:31.035954 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dad25b3a-8jt96" event={"ID":"50735aee-920f-4412-a82e-655ce2c43094","Type":"ContainerStarted","Data":"b88a83b0fa47758c1128c755363f13ce863ce5b4be12e3395215d623981cf470"} Oct 30 13:04:31.039268 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:31.039240 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" Oct 30 13:04:31.927212 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:31.927177 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:32.218422 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:32.218387 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:32.587670 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:32.587571 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:34.064884 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:34.064847 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dad25b3a-8jt96" event={"ID":"50735aee-920f-4412-a82e-655ce2c43094","Type":"ContainerStarted","Data":"ae4700fca92ebefbf08d774e920cdeb6a2941608312e97e402f9e64bee83e892"} Oct 30 13:04:34.115928 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:34.115882 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dad25b3a-8jt96" podStartSLOduration=1.691280088 podStartE2EDuration="4.11587073s" totalImagesPullingTime="2.424590642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:34.113775883 +0000 UTC m=+730.583191628" watchObservedRunningTime="2026-10-30 13:04:34.11587073 +0000 UTC m=+730.585286475" Oct 30 13:04:34.601858 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:34.601823 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:35.071576 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:35.071539 5922 generic.go:356] "Generic (PLEG): container finished" podID="50735aee-920f-4412-a82e-655ce2c43094" containerID="ae4700fca92ebefbf08d774e920cdeb6a2941608312e97e402f9e64bee83e892" exitCode=0 Oct 30 13:04:35.071970 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:35.071637 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dad25b3a-8jt96" event={"ID":"50735aee-920f-4412-a82e-655ce2c43094","Type":"ContainerDied","Data":"ae4700fca92ebefbf08d774e920cdeb6a2941608312e97e402f9e64bee83e892"} Oct 30 13:04:35.165431 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:35.165406 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" Oct 30 13:04:35.178215 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:35.178187 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" Oct 30 13:04:35.254384 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:35.254352 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef480c1f-ffhjq"] Oct 30 13:04:35.257148 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:35.257128 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef480c1f-ffhjq" Oct 30 13:04:35.277092 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:35.277072 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef480c1f-ffhjq"] Oct 30 13:04:35.368389 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:35.368325 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpc4q\" (UniqueName: \"kubernetes.io/projected/c6e2aea3-e4de-4607-9d53-a5f30cba70e9-kube-api-access-fpc4q\") pod \"maas-api-revoke-sub-ef480c1f-ffhjq\" (UID: \"c6e2aea3-e4de-4607-9d53-a5f30cba70e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef480c1f-ffhjq" Oct 30 13:04:35.368389 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:35.368357 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6e2aea3-e4de-4607-9d53-a5f30cba70e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef480c1f-ffhjq\" (UID: \"c6e2aea3-e4de-4607-9d53-a5f30cba70e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef480c1f-ffhjq" Oct 30 13:04:35.368578 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:35.368395 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6e2aea3-e4de-4607-9d53-a5f30cba70e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef480c1f-ffhjq\" (UID: \"c6e2aea3-e4de-4607-9d53-a5f30cba70e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef480c1f-ffhjq" Oct 30 13:04:35.469414 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:35.469390 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpc4q\" (UniqueName: \"kubernetes.io/projected/c6e2aea3-e4de-4607-9d53-a5f30cba70e9-kube-api-access-fpc4q\") pod \"maas-api-revoke-sub-ef480c1f-ffhjq\" (UID: \"c6e2aea3-e4de-4607-9d53-a5f30cba70e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef480c1f-ffhjq" Oct 30 13:04:35.469414 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:35.469422 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6e2aea3-e4de-4607-9d53-a5f30cba70e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef480c1f-ffhjq\" (UID: \"c6e2aea3-e4de-4607-9d53-a5f30cba70e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef480c1f-ffhjq" Oct 30 13:04:35.469652 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:35.469457 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6e2aea3-e4de-4607-9d53-a5f30cba70e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef480c1f-ffhjq\" (UID: \"c6e2aea3-e4de-4607-9d53-a5f30cba70e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef480c1f-ffhjq" Oct 30 13:04:35.470042 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:35.470022 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6e2aea3-e4de-4607-9d53-a5f30cba70e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef480c1f-ffhjq\" (UID: \"c6e2aea3-e4de-4607-9d53-a5f30cba70e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef480c1f-ffhjq" Oct 30 13:04:35.471703 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:35.471684 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6e2aea3-e4de-4607-9d53-a5f30cba70e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef480c1f-ffhjq\" (UID: \"c6e2aea3-e4de-4607-9d53-a5f30cba70e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef480c1f-ffhjq" Oct 30 13:04:35.504369 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:35.504344 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpc4q\" (UniqueName: \"kubernetes.io/projected/c6e2aea3-e4de-4607-9d53-a5f30cba70e9-kube-api-access-fpc4q\") pod \"maas-api-revoke-sub-ef480c1f-ffhjq\" (UID: \"c6e2aea3-e4de-4607-9d53-a5f30cba70e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef480c1f-ffhjq" Oct 30 13:04:35.568527 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:35.568495 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef480c1f-ffhjq" Oct 30 13:04:35.869083 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:35.869045 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef480c1f-ffhjq"] Oct 30 13:04:35.877365 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:04:35.877334 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6e2aea3_e4de_4607_9d53_a5f30cba70e9.slice/crio-22da0da9e5f3dae5921ef0409952cf92eb10fb3163d222b92f5b614925fe0d22 WatchSource:0}: Error finding container 22da0da9e5f3dae5921ef0409952cf92eb10fb3163d222b92f5b614925fe0d22: Status 404 returned error can't find the container with id 22da0da9e5f3dae5921ef0409952cf92eb10fb3163d222b92f5b614925fe0d22 Oct 30 13:04:36.078062 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:36.078032 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef480c1f-ffhjq" event={"ID":"c6e2aea3-e4de-4607-9d53-a5f30cba70e9","Type":"ContainerStarted","Data":"22da0da9e5f3dae5921ef0409952cf92eb10fb3163d222b92f5b614925fe0d22"} Oct 30 13:04:36.090180 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:36.090152 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" Oct 30 13:04:36.217860 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:36.217835 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dad25b3a-8jt96" Oct 30 13:04:36.379738 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:36.379711 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50735aee-920f-4412-a82e-655ce2c43094-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50735aee-920f-4412-a82e-655ce2c43094-maas-api-xks-ca\") pod \"50735aee-920f-4412-a82e-655ce2c43094\" (UID: \"50735aee-920f-4412-a82e-655ce2c43094\") " Oct 30 13:04:36.379900 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:36.379746 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50735aee-920f-4412-a82e-655ce2c43094-kube-api-access-67kp8\" (UniqueName: \"kubernetes.io/projected/50735aee-920f-4412-a82e-655ce2c43094-kube-api-access-67kp8\") pod \"50735aee-920f-4412-a82e-655ce2c43094\" (UID: \"50735aee-920f-4412-a82e-655ce2c43094\") " Oct 30 13:04:36.379900 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:36.379778 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50735aee-920f-4412-a82e-655ce2c43094-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50735aee-920f-4412-a82e-655ce2c43094-maas-api-openshift-ca\") pod \"50735aee-920f-4412-a82e-655ce2c43094\" (UID: \"50735aee-920f-4412-a82e-655ce2c43094\") " Oct 30 13:04:36.380125 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:36.380104 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50735aee-920f-4412-a82e-655ce2c43094-maas-api-openshift-ca" pod "50735aee-920f-4412-a82e-655ce2c43094" (UID: "50735aee-920f-4412-a82e-655ce2c43094"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:36.380272 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:36.380257 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50735aee-920f-4412-a82e-655ce2c43094-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:04:36.381980 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:36.381918 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50735aee-920f-4412-a82e-655ce2c43094-maas-api-xks-ca" pod "50735aee-920f-4412-a82e-655ce2c43094" (UID: "50735aee-920f-4412-a82e-655ce2c43094"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:04:36.381980 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:36.381949 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50735aee-920f-4412-a82e-655ce2c43094-kube-api-access-67kp8" pod "50735aee-920f-4412-a82e-655ce2c43094" (UID: "50735aee-920f-4412-a82e-655ce2c43094"). InnerVolumeSpecName "kube-api-access-67kp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:36.448512 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:36.448471 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:36.482510 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:36.482482 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50735aee-920f-4412-a82e-655ce2c43094-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:04:36.482657 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:36.482515 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67kp8\" (UniqueName: \"kubernetes.io/projected/50735aee-920f-4412-a82e-655ce2c43094-kube-api-access-67kp8\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:04:37.086095 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:37.086072 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dad25b3a-8jt96" Oct 30 13:04:37.086508 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:37.086105 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dad25b3a-8jt96" event={"ID":"50735aee-920f-4412-a82e-655ce2c43094","Type":"ContainerDied","Data":"b88a83b0fa47758c1128c755363f13ce863ce5b4be12e3395215d623981cf470"} Oct 30 13:04:37.086508 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:37.086158 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b88a83b0fa47758c1128c755363f13ce863ce5b4be12e3395215d623981cf470" Oct 30 13:04:37.088041 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:37.088014 5922 generic.go:356] "Generic (PLEG): container finished" podID="c6e2aea3-e4de-4607-9d53-a5f30cba70e9" containerID="ebfbdcf2b28aad03e076b94d68e283ef2ac87cde6c6f50af28d2d3c0b50c9ed6" exitCode=0 Oct 30 13:04:37.088430 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:37.088404 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef480c1f-ffhjq" event={"ID":"c6e2aea3-e4de-4607-9d53-a5f30cba70e9","Type":"ContainerDied","Data":"ebfbdcf2b28aad03e076b94d68e283ef2ac87cde6c6f50af28d2d3c0b50c9ed6"} Oct 30 13:04:38.241411 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:38.241390 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef480c1f-ffhjq" Oct 30 13:04:38.401818 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:38.401752 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6e2aea3-e4de-4607-9d53-a5f30cba70e9-kube-api-access-fpc4q\" (UniqueName: \"kubernetes.io/projected/c6e2aea3-e4de-4607-9d53-a5f30cba70e9-kube-api-access-fpc4q\") pod \"c6e2aea3-e4de-4607-9d53-a5f30cba70e9\" (UID: \"c6e2aea3-e4de-4607-9d53-a5f30cba70e9\") " Oct 30 13:04:38.401941 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:38.401824 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6e2aea3-e4de-4607-9d53-a5f30cba70e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6e2aea3-e4de-4607-9d53-a5f30cba70e9-maas-api-openshift-ca\") pod \"c6e2aea3-e4de-4607-9d53-a5f30cba70e9\" (UID: \"c6e2aea3-e4de-4607-9d53-a5f30cba70e9\") " Oct 30 13:04:38.401941 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:38.401873 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6e2aea3-e4de-4607-9d53-a5f30cba70e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6e2aea3-e4de-4607-9d53-a5f30cba70e9-maas-api-xks-ca\") pod \"c6e2aea3-e4de-4607-9d53-a5f30cba70e9\" (UID: \"c6e2aea3-e4de-4607-9d53-a5f30cba70e9\") " Oct 30 13:04:38.402155 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:38.402132 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6e2aea3-e4de-4607-9d53-a5f30cba70e9-maas-api-openshift-ca" pod "c6e2aea3-e4de-4607-9d53-a5f30cba70e9" (UID: "c6e2aea3-e4de-4607-9d53-a5f30cba70e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:38.402233 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:38.402224 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6e2aea3-e4de-4607-9d53-a5f30cba70e9-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:04:38.403782 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:38.403758 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6e2aea3-e4de-4607-9d53-a5f30cba70e9-kube-api-access-fpc4q" pod "c6e2aea3-e4de-4607-9d53-a5f30cba70e9" (UID: "c6e2aea3-e4de-4607-9d53-a5f30cba70e9"). InnerVolumeSpecName "kube-api-access-fpc4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:38.403878 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:38.403809 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6e2aea3-e4de-4607-9d53-a5f30cba70e9-maas-api-xks-ca" pod "c6e2aea3-e4de-4607-9d53-a5f30cba70e9" (UID: "c6e2aea3-e4de-4607-9d53-a5f30cba70e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:04:38.498778 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:38.498747 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:38.503088 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:38.503068 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpc4q\" (UniqueName: \"kubernetes.io/projected/c6e2aea3-e4de-4607-9d53-a5f30cba70e9-kube-api-access-fpc4q\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:04:38.503163 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:38.503091 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6e2aea3-e4de-4607-9d53-a5f30cba70e9-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:04:39.099704 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:39.099664 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef480c1f-ffhjq" event={"ID":"c6e2aea3-e4de-4607-9d53-a5f30cba70e9","Type":"ContainerDied","Data":"22da0da9e5f3dae5921ef0409952cf92eb10fb3163d222b92f5b614925fe0d22"} Oct 30 13:04:39.099704 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:39.099699 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef480c1f-ffhjq" Oct 30 13:04:39.099962 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:39.099722 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22da0da9e5f3dae5921ef0409952cf92eb10fb3163d222b92f5b614925fe0d22" Oct 30 13:04:40.415337 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:40.415297 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:42.350243 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:42.350196 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:44.463845 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:44.463811 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:44.490519 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:44.490482 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b14df5e-jl9pc"] Oct 30 13:04:44.491096 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:44.491079 5922 cpu_manager.go:414] "containerMap: removing container" podUID="50735aee-920f-4412-a82e-655ce2c43094" containerName="revoke-keys" Oct 30 13:04:44.491158 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:44.491097 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50735aee-920f-4412-a82e-655ce2c43094" containerName="revoke-keys" Oct 30 13:04:44.491158 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:44.491107 5922 cpu_manager.go:414] "containerMap: removing container" podUID="c6e2aea3-e4de-4607-9d53-a5f30cba70e9" containerName="revoke-keys" Oct 30 13:04:44.491158 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:44.491112 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6e2aea3-e4de-4607-9d53-a5f30cba70e9" containerName="revoke-keys" Oct 30 13:04:44.491295 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:44.491196 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="50735aee-920f-4412-a82e-655ce2c43094" containerName="revoke-keys" Oct 30 13:04:44.491295 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:44.491204 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6e2aea3-e4de-4607-9d53-a5f30cba70e9" containerName="revoke-keys" Oct 30 13:04:44.493851 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:44.493831 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b14df5e-jl9pc" Oct 30 13:04:44.514941 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:44.514913 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b14df5e-jl9pc"] Oct 30 13:04:44.524217 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:44.524193 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:04:44.524217 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:44.524211 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:04:44.524428 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:44.524261 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:04:44.525089 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:44.525066 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:04:44.563293 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:44.563267 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f0a5c3b-02f4-433d-b710-aba5fb022ff8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b14df5e-jl9pc\" (UID: \"4f0a5c3b-02f4-433d-b710-aba5fb022ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b14df5e-jl9pc" Oct 30 13:04:44.563396 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:44.563341 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f0a5c3b-02f4-433d-b710-aba5fb022ff8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b14df5e-jl9pc\" (UID: \"4f0a5c3b-02f4-433d-b710-aba5fb022ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b14df5e-jl9pc" Oct 30 13:04:44.563442 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:44.563414 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2pqg\" (UniqueName: \"kubernetes.io/projected/4f0a5c3b-02f4-433d-b710-aba5fb022ff8-kube-api-access-h2pqg\") pod \"maas-api-revoke-sub-6b14df5e-jl9pc\" (UID: \"4f0a5c3b-02f4-433d-b710-aba5fb022ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b14df5e-jl9pc" Oct 30 13:04:44.664074 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:44.664040 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f0a5c3b-02f4-433d-b710-aba5fb022ff8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b14df5e-jl9pc\" (UID: \"4f0a5c3b-02f4-433d-b710-aba5fb022ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b14df5e-jl9pc" Oct 30 13:04:44.664204 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:44.664093 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2pqg\" (UniqueName: \"kubernetes.io/projected/4f0a5c3b-02f4-433d-b710-aba5fb022ff8-kube-api-access-h2pqg\") pod \"maas-api-revoke-sub-6b14df5e-jl9pc\" (UID: \"4f0a5c3b-02f4-433d-b710-aba5fb022ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b14df5e-jl9pc" Oct 30 13:04:44.664267 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:44.664201 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f0a5c3b-02f4-433d-b710-aba5fb022ff8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b14df5e-jl9pc\" (UID: \"4f0a5c3b-02f4-433d-b710-aba5fb022ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b14df5e-jl9pc" Oct 30 13:04:44.664666 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:44.664647 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f0a5c3b-02f4-433d-b710-aba5fb022ff8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b14df5e-jl9pc\" (UID: \"4f0a5c3b-02f4-433d-b710-aba5fb022ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b14df5e-jl9pc" Oct 30 13:04:44.666446 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:44.666425 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f0a5c3b-02f4-433d-b710-aba5fb022ff8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b14df5e-jl9pc\" (UID: \"4f0a5c3b-02f4-433d-b710-aba5fb022ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b14df5e-jl9pc" Oct 30 13:04:44.688450 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:44.688424 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2pqg\" (UniqueName: \"kubernetes.io/projected/4f0a5c3b-02f4-433d-b710-aba5fb022ff8-kube-api-access-h2pqg\") pod \"maas-api-revoke-sub-6b14df5e-jl9pc\" (UID: \"4f0a5c3b-02f4-433d-b710-aba5fb022ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b14df5e-jl9pc" Oct 30 13:04:44.807361 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:44.807304 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b14df5e-jl9pc" Oct 30 13:04:45.109507 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:45.109484 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b14df5e-jl9pc"] Oct 30 13:04:45.112193 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:04:45.112166 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f0a5c3b_02f4_433d_b710_aba5fb022ff8.slice/crio-caf49ddd03e4e3be8307e121ed49d341a0f061f620b44944a978e5484dc30023 WatchSource:0}: Error finding container caf49ddd03e4e3be8307e121ed49d341a0f061f620b44944a978e5484dc30023: Status 404 returned error can't find the container with id caf49ddd03e4e3be8307e121ed49d341a0f061f620b44944a978e5484dc30023 Oct 30 13:04:45.145582 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:45.145551 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b14df5e-jl9pc" event={"ID":"4f0a5c3b-02f4-433d-b710-aba5fb022ff8","Type":"ContainerStarted","Data":"caf49ddd03e4e3be8307e121ed49d341a0f061f620b44944a978e5484dc30023"} Oct 30 13:04:46.153510 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:46.153476 5922 generic.go:356] "Generic (PLEG): container finished" podID="4f0a5c3b-02f4-433d-b710-aba5fb022ff8" containerID="fa5b76c7113cd9b24bf23b452f2b0d950ed4680c3ec67ca9f1c568d7a817c9cc" exitCode=0 Oct 30 13:04:46.154028 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:46.153519 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b14df5e-jl9pc" event={"ID":"4f0a5c3b-02f4-433d-b710-aba5fb022ff8","Type":"ContainerDied","Data":"fa5b76c7113cd9b24bf23b452f2b0d950ed4680c3ec67ca9f1c568d7a817c9cc"} Oct 30 13:04:46.175256 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:46.175213 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:47.309408 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:47.309383 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b14df5e-jl9pc" Oct 30 13:04:47.393507 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:47.393480 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f0a5c3b-02f4-433d-b710-aba5fb022ff8-kube-api-access-h2pqg\" (UniqueName: \"kubernetes.io/projected/4f0a5c3b-02f4-433d-b710-aba5fb022ff8-kube-api-access-h2pqg\") pod \"4f0a5c3b-02f4-433d-b710-aba5fb022ff8\" (UID: \"4f0a5c3b-02f4-433d-b710-aba5fb022ff8\") " Oct 30 13:04:47.393662 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:47.393628 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f0a5c3b-02f4-433d-b710-aba5fb022ff8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f0a5c3b-02f4-433d-b710-aba5fb022ff8-maas-api-openshift-ca\") pod \"4f0a5c3b-02f4-433d-b710-aba5fb022ff8\" (UID: \"4f0a5c3b-02f4-433d-b710-aba5fb022ff8\") " Oct 30 13:04:47.393711 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:47.393697 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f0a5c3b-02f4-433d-b710-aba5fb022ff8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f0a5c3b-02f4-433d-b710-aba5fb022ff8-maas-api-xks-ca\") pod \"4f0a5c3b-02f4-433d-b710-aba5fb022ff8\" (UID: \"4f0a5c3b-02f4-433d-b710-aba5fb022ff8\") " Oct 30 13:04:47.393987 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:47.393961 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f0a5c3b-02f4-433d-b710-aba5fb022ff8-maas-api-openshift-ca" pod "4f0a5c3b-02f4-433d-b710-aba5fb022ff8" (UID: "4f0a5c3b-02f4-433d-b710-aba5fb022ff8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:47.394120 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:47.394084 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f0a5c3b-02f4-433d-b710-aba5fb022ff8-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:04:47.395725 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:47.395697 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f0a5c3b-02f4-433d-b710-aba5fb022ff8-maas-api-xks-ca" pod "4f0a5c3b-02f4-433d-b710-aba5fb022ff8" (UID: "4f0a5c3b-02f4-433d-b710-aba5fb022ff8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:04:47.395941 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:47.395924 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f0a5c3b-02f4-433d-b710-aba5fb022ff8-kube-api-access-h2pqg" pod "4f0a5c3b-02f4-433d-b710-aba5fb022ff8" (UID: "4f0a5c3b-02f4-433d-b710-aba5fb022ff8"). InnerVolumeSpecName "kube-api-access-h2pqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:47.495518 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:47.495483 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f0a5c3b-02f4-433d-b710-aba5fb022ff8-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:04:47.495518 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:47.495517 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2pqg\" (UniqueName: \"kubernetes.io/projected/4f0a5c3b-02f4-433d-b710-aba5fb022ff8-kube-api-access-h2pqg\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:04:48.166224 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:48.166191 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b14df5e-jl9pc" Oct 30 13:04:48.166385 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:48.166226 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b14df5e-jl9pc" event={"ID":"4f0a5c3b-02f4-433d-b710-aba5fb022ff8","Type":"ContainerDied","Data":"caf49ddd03e4e3be8307e121ed49d341a0f061f620b44944a978e5484dc30023"} Oct 30 13:04:48.166385 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:48.166285 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="caf49ddd03e4e3be8307e121ed49d341a0f061f620b44944a978e5484dc30023" Oct 30 13:04:48.202400 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:48.202372 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:49.600520 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:49.600488 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l"] Oct 30 13:04:49.601035 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:49.601019 5922 cpu_manager.go:414] "containerMap: removing container" podUID="4f0a5c3b-02f4-433d-b710-aba5fb022ff8" containerName="revoke-keys" Oct 30 13:04:49.601082 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:49.601036 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f0a5c3b-02f4-433d-b710-aba5fb022ff8" containerName="revoke-keys" Oct 30 13:04:49.601136 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:49.601126 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f0a5c3b-02f4-433d-b710-aba5fb022ff8" containerName="revoke-keys" Oct 30 13:04:49.603589 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:49.603576 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" Oct 30 13:04:49.621302 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:49.621278 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-42ee9d-openshift-default-dockercfg-sb4pn\"" Oct 30 13:04:49.634531 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:49.634509 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l"] Oct 30 13:04:49.718978 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:49.718946 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9126994e-0a2a-4558-a255-bc3f01359322-istio-envoy\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l\" (UID: \"9126994e-0a2a-4558-a255-bc3f01359322\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" Oct 30 13:04:49.719183 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:49.719120 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9126994e-0a2a-4558-a255-bc3f01359322-istio-data\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l\" (UID: \"9126994e-0a2a-4558-a255-bc3f01359322\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" Oct 30 13:04:49.719183 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:49.719156 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9126994e-0a2a-4558-a255-bc3f01359322-credential-socket\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l\" (UID: \"9126994e-0a2a-4558-a255-bc3f01359322\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" Oct 30 13:04:49.719316 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:49.719211 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9126994e-0a2a-4558-a255-bc3f01359322-istiod-ca-cert\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l\" (UID: \"9126994e-0a2a-4558-a255-bc3f01359322\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" Oct 30 13:04:49.719316 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:49.719279 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9126994e-0a2a-4558-a255-bc3f01359322-istio-podinfo\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l\" (UID: \"9126994e-0a2a-4558-a255-bc3f01359322\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" Oct 30 13:04:49.719438 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:49.719422 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9126994e-0a2a-4558-a255-bc3f01359322-workload-certs\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l\" (UID: \"9126994e-0a2a-4558-a255-bc3f01359322\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" Oct 30 13:04:49.719532 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:49.719511 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9126994e-0a2a-4558-a255-bc3f01359322-workload-socket\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l\" (UID: \"9126994e-0a2a-4558-a255-bc3f01359322\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" Oct 30 13:04:49.719668 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:49.719653 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9126994e-0a2a-4558-a255-bc3f01359322-istio-token\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l\" (UID: \"9126994e-0a2a-4558-a255-bc3f01359322\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" Oct 30 13:04:49.719719 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:49.719702 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcwkq\" (UniqueName: \"kubernetes.io/projected/9126994e-0a2a-4558-a255-bc3f01359322-kube-api-access-lcwkq\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l\" (UID: \"9126994e-0a2a-4558-a255-bc3f01359322\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" Oct 30 13:04:49.820607 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:49.820573 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9126994e-0a2a-4558-a255-bc3f01359322-workload-certs\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l\" (UID: \"9126994e-0a2a-4558-a255-bc3f01359322\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" Oct 30 13:04:49.820807 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:49.820650 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9126994e-0a2a-4558-a255-bc3f01359322-workload-socket\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l\" (UID: \"9126994e-0a2a-4558-a255-bc3f01359322\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" Oct 30 13:04:49.820807 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:49.820670 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9126994e-0a2a-4558-a255-bc3f01359322-istio-token\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l\" (UID: \"9126994e-0a2a-4558-a255-bc3f01359322\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" Oct 30 13:04:49.820807 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:49.820694 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcwkq\" (UniqueName: \"kubernetes.io/projected/9126994e-0a2a-4558-a255-bc3f01359322-kube-api-access-lcwkq\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l\" (UID: \"9126994e-0a2a-4558-a255-bc3f01359322\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" Oct 30 13:04:49.820807 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:49.820752 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9126994e-0a2a-4558-a255-bc3f01359322-istio-envoy\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l\" (UID: \"9126994e-0a2a-4558-a255-bc3f01359322\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" Oct 30 13:04:49.821039 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:49.820841 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9126994e-0a2a-4558-a255-bc3f01359322-istio-data\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l\" (UID: \"9126994e-0a2a-4558-a255-bc3f01359322\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" Oct 30 13:04:49.821039 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:49.820868 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9126994e-0a2a-4558-a255-bc3f01359322-credential-socket\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l\" (UID: \"9126994e-0a2a-4558-a255-bc3f01359322\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" Oct 30 13:04:49.821039 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:49.820901 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9126994e-0a2a-4558-a255-bc3f01359322-istiod-ca-cert\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l\" (UID: \"9126994e-0a2a-4558-a255-bc3f01359322\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" Oct 30 13:04:49.821039 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:49.820943 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9126994e-0a2a-4558-a255-bc3f01359322-istio-podinfo\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l\" (UID: \"9126994e-0a2a-4558-a255-bc3f01359322\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" Oct 30 13:04:49.821226 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:49.821072 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9126994e-0a2a-4558-a255-bc3f01359322-workload-socket\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l\" (UID: \"9126994e-0a2a-4558-a255-bc3f01359322\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" Oct 30 13:04:49.821226 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:49.821042 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9126994e-0a2a-4558-a255-bc3f01359322-workload-certs\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l\" (UID: \"9126994e-0a2a-4558-a255-bc3f01359322\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" Oct 30 13:04:49.821317 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:49.821260 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9126994e-0a2a-4558-a255-bc3f01359322-istio-data\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l\" (UID: \"9126994e-0a2a-4558-a255-bc3f01359322\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" Oct 30 13:04:49.821380 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:49.821337 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9126994e-0a2a-4558-a255-bc3f01359322-credential-socket\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l\" (UID: \"9126994e-0a2a-4558-a255-bc3f01359322\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" Oct 30 13:04:49.821583 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:49.821551 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9126994e-0a2a-4558-a255-bc3f01359322-istiod-ca-cert\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l\" (UID: \"9126994e-0a2a-4558-a255-bc3f01359322\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" Oct 30 13:04:49.823318 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:49.823294 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9126994e-0a2a-4558-a255-bc3f01359322-istio-envoy\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l\" (UID: \"9126994e-0a2a-4558-a255-bc3f01359322\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" Oct 30 13:04:49.823459 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:49.823440 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9126994e-0a2a-4558-a255-bc3f01359322-istio-podinfo\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l\" (UID: \"9126994e-0a2a-4558-a255-bc3f01359322\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" Oct 30 13:04:49.848863 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:49.848839 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9126994e-0a2a-4558-a255-bc3f01359322-istio-token\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l\" (UID: \"9126994e-0a2a-4558-a255-bc3f01359322\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" Oct 30 13:04:49.848989 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:49.848973 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcwkq\" (UniqueName: \"kubernetes.io/projected/9126994e-0a2a-4558-a255-bc3f01359322-kube-api-access-lcwkq\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l\" (UID: \"9126994e-0a2a-4558-a255-bc3f01359322\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" Oct 30 13:04:49.916097 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:49.916011 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" Oct 30 13:04:49.958194 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:49.958162 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd"] Oct 30 13:04:49.962135 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:49.962114 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" Oct 30 13:04:49.984005 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:49.982564 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-54994c-openshift-default-dockercfg-2qhnp\"" Oct 30 13:04:49.993708 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:49.984633 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd"] Oct 30 13:04:50.023470 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:50.023439 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5fb05c2-4a10-4c85-8464-75e247feae67-credential-socket\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd\" (UID: \"e5fb05c2-4a10-4c85-8464-75e247feae67\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" Oct 30 13:04:50.023674 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:50.023494 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5fb05c2-4a10-4c85-8464-75e247feae67-workload-socket\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd\" (UID: \"e5fb05c2-4a10-4c85-8464-75e247feae67\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" Oct 30 13:04:50.023674 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:50.023648 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgvxj\" (UniqueName: \"kubernetes.io/projected/e5fb05c2-4a10-4c85-8464-75e247feae67-kube-api-access-jgvxj\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd\" (UID: \"e5fb05c2-4a10-4c85-8464-75e247feae67\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" Oct 30 13:04:50.023811 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:50.023750 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5fb05c2-4a10-4c85-8464-75e247feae67-istiod-ca-cert\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd\" (UID: \"e5fb05c2-4a10-4c85-8464-75e247feae67\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" Oct 30 13:04:50.023811 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:50.023785 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5fb05c2-4a10-4c85-8464-75e247feae67-workload-certs\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd\" (UID: \"e5fb05c2-4a10-4c85-8464-75e247feae67\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" Oct 30 13:04:50.023922 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:50.023840 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5fb05c2-4a10-4c85-8464-75e247feae67-istio-envoy\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd\" (UID: \"e5fb05c2-4a10-4c85-8464-75e247feae67\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" Oct 30 13:04:50.023922 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:50.023856 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5fb05c2-4a10-4c85-8464-75e247feae67-istio-podinfo\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd\" (UID: \"e5fb05c2-4a10-4c85-8464-75e247feae67\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" Oct 30 13:04:50.024023 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:50.023990 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5fb05c2-4a10-4c85-8464-75e247feae67-istio-token\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd\" (UID: \"e5fb05c2-4a10-4c85-8464-75e247feae67\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" Oct 30 13:04:50.024023 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:50.024014 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5fb05c2-4a10-4c85-8464-75e247feae67-istio-data\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd\" (UID: \"e5fb05c2-4a10-4c85-8464-75e247feae67\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" Oct 30 13:04:50.125406 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:50.125352 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5fb05c2-4a10-4c85-8464-75e247feae67-istio-envoy\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd\" (UID: \"e5fb05c2-4a10-4c85-8464-75e247feae67\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" Oct 30 13:04:50.125406 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:50.125408 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5fb05c2-4a10-4c85-8464-75e247feae67-istio-podinfo\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd\" (UID: \"e5fb05c2-4a10-4c85-8464-75e247feae67\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" Oct 30 13:04:50.125817 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:50.125597 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5fb05c2-4a10-4c85-8464-75e247feae67-istio-token\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd\" (UID: \"e5fb05c2-4a10-4c85-8464-75e247feae67\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" Oct 30 13:04:50.125817 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:50.125670 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5fb05c2-4a10-4c85-8464-75e247feae67-istio-data\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd\" (UID: \"e5fb05c2-4a10-4c85-8464-75e247feae67\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" Oct 30 13:04:50.125817 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:50.125709 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5fb05c2-4a10-4c85-8464-75e247feae67-credential-socket\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd\" (UID: \"e5fb05c2-4a10-4c85-8464-75e247feae67\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" Oct 30 13:04:50.125817 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:50.125749 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5fb05c2-4a10-4c85-8464-75e247feae67-workload-socket\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd\" (UID: \"e5fb05c2-4a10-4c85-8464-75e247feae67\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" Oct 30 13:04:50.125817 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:50.125807 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgvxj\" (UniqueName: \"kubernetes.io/projected/e5fb05c2-4a10-4c85-8464-75e247feae67-kube-api-access-jgvxj\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd\" (UID: \"e5fb05c2-4a10-4c85-8464-75e247feae67\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" Oct 30 13:04:50.126090 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:50.125871 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5fb05c2-4a10-4c85-8464-75e247feae67-istiod-ca-cert\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd\" (UID: \"e5fb05c2-4a10-4c85-8464-75e247feae67\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" Oct 30 13:04:50.126090 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:50.125901 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5fb05c2-4a10-4c85-8464-75e247feae67-workload-certs\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd\" (UID: \"e5fb05c2-4a10-4c85-8464-75e247feae67\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" Oct 30 13:04:50.126208 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:50.126168 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5fb05c2-4a10-4c85-8464-75e247feae67-istio-data\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd\" (UID: \"e5fb05c2-4a10-4c85-8464-75e247feae67\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" Oct 30 13:04:50.126266 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:50.126252 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5fb05c2-4a10-4c85-8464-75e247feae67-workload-certs\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd\" (UID: \"e5fb05c2-4a10-4c85-8464-75e247feae67\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" Oct 30 13:04:50.126652 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:50.126503 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5fb05c2-4a10-4c85-8464-75e247feae67-credential-socket\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd\" (UID: \"e5fb05c2-4a10-4c85-8464-75e247feae67\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" Oct 30 13:04:50.126785 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:50.126764 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5fb05c2-4a10-4c85-8464-75e247feae67-workload-socket\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd\" (UID: \"e5fb05c2-4a10-4c85-8464-75e247feae67\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" Oct 30 13:04:50.127644 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:50.127582 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5fb05c2-4a10-4c85-8464-75e247feae67-istiod-ca-cert\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd\" (UID: \"e5fb05c2-4a10-4c85-8464-75e247feae67\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" Oct 30 13:04:50.129359 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:50.129330 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5fb05c2-4a10-4c85-8464-75e247feae67-istio-envoy\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd\" (UID: \"e5fb05c2-4a10-4c85-8464-75e247feae67\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" Oct 30 13:04:50.131264 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:50.131241 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5fb05c2-4a10-4c85-8464-75e247feae67-istio-podinfo\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd\" (UID: \"e5fb05c2-4a10-4c85-8464-75e247feae67\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" Oct 30 13:04:50.152022 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:50.151994 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgvxj\" (UniqueName: \"kubernetes.io/projected/e5fb05c2-4a10-4c85-8464-75e247feae67-kube-api-access-jgvxj\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd\" (UID: \"e5fb05c2-4a10-4c85-8464-75e247feae67\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" Oct 30 13:04:50.153725 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:50.153705 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5fb05c2-4a10-4c85-8464-75e247feae67-istio-token\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd\" (UID: \"e5fb05c2-4a10-4c85-8464-75e247feae67\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" Oct 30 13:04:50.195716 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:50.195681 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:50.253172 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:50.253144 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l"] Oct 30 13:04:50.255805 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:04:50.255767 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9126994e_0a2a_4558_a255_bc3f01359322.slice/crio-b4de3670c3ce9b89e890d9a9a928da79054f6549253735296e2b131aa2c37d2b WatchSource:0}: Error finding container b4de3670c3ce9b89e890d9a9a928da79054f6549253735296e2b131aa2c37d2b: Status 404 returned error can't find the container with id b4de3670c3ce9b89e890d9a9a928da79054f6549253735296e2b131aa2c37d2b Oct 30 13:04:50.258223 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:50.258189 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:04:50.258319 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:50.258267 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:04:50.258384 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:50.258316 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:04:50.283780 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:50.283758 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" Oct 30 13:04:50.614819 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:50.614796 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd"] Oct 30 13:04:50.616705 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:04:50.616675 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5fb05c2_4a10_4c85_8464_75e247feae67.slice/crio-6a9e02ff6f39c888bd7317c041b629355916ea80f202c0ead00d23c629c68f2e WatchSource:0}: Error finding container 6a9e02ff6f39c888bd7317c041b629355916ea80f202c0ead00d23c629c68f2e: Status 404 returned error can't find the container with id 6a9e02ff6f39c888bd7317c041b629355916ea80f202c0ead00d23c629c68f2e Oct 30 13:04:50.619262 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:50.619223 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:04:50.619372 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:50.619307 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:04:50.619372 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:50.619347 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:04:51.191867 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:51.191826 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" event={"ID":"e5fb05c2-4a10-4c85-8464-75e247feae67","Type":"ContainerStarted","Data":"b0ed688211bcc67268db4556a64ca2a9883c7b63106739c7381c6df918864c9c"} Oct 30 13:04:51.192087 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:51.191895 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" event={"ID":"e5fb05c2-4a10-4c85-8464-75e247feae67","Type":"ContainerStarted","Data":"6a9e02ff6f39c888bd7317c041b629355916ea80f202c0ead00d23c629c68f2e"} Oct 30 13:04:51.193669 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:51.193640 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" event={"ID":"9126994e-0a2a-4558-a255-bc3f01359322","Type":"ContainerStarted","Data":"f853776dc96b270a425160f55cd9ec498be7ed2caee6a8614d173fbda5a57667"} Oct 30 13:04:51.193779 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:51.193690 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" event={"ID":"9126994e-0a2a-4558-a255-bc3f01359322","Type":"ContainerStarted","Data":"b4de3670c3ce9b89e890d9a9a928da79054f6549253735296e2b131aa2c37d2b"} Oct 30 13:04:51.252424 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:51.252372 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" podStartSLOduration=2.25235734 podStartE2EDuration="2.25235734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:51.241538262 +0000 UTC m=+747.710954024" watchObservedRunningTime="2026-10-30 13:04:51.25235734 +0000 UTC m=+747.721773084" Oct 30 13:04:51.284136 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:51.284102 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" Oct 30 13:04:51.286793 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:51.286759 5922 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 13:04:51.286950 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:51.286813 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" podUID="e5fb05c2-4a10-4c85-8464-75e247feae67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 13:04:51.319825 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:51.319773 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" podStartSLOduration=2.3197585 podStartE2EDuration="2.3197585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:51.319078141 +0000 UTC m=+747.788493937" watchObservedRunningTime="2026-10-30 13:04:51.3197585 +0000 UTC m=+747.789174244" Oct 30 13:04:51.916752 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:51.916711 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" Oct 30 13:04:51.918208 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:51.918183 5922 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 13:04:51.918338 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:51.918236 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" podUID="9126994e-0a2a-4558-a255-bc3f01359322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 13:04:52.306636 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:52.288089 5922 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 13:04:52.306636 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:52.288150 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" podUID="e5fb05c2-4a10-4c85-8464-75e247feae67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 13:04:52.306636 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:52.289994 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:52.504636 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:52.496693 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:04:52.921648 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:52.921367 5922 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 13:04:52.921648 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:52.921438 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" podUID="9126994e-0a2a-4558-a255-bc3f01359322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 13:04:53.289099 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:53.289059 5922 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 13:04:53.289398 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:53.289365 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" podUID="e5fb05c2-4a10-4c85-8464-75e247feae67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 13:04:53.695091 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:04:53.695053 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:04:53.793678 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:53.793646 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:53.926765 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:53.926730 5922 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 13:04:53.927549 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:53.927513 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" podUID="9126994e-0a2a-4558-a255-bc3f01359322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 13:04:54.286043 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:54.286010 5922 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 13:04:54.286307 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:54.286270 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" podUID="e5fb05c2-4a10-4c85-8464-75e247feae67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 13:04:54.649888 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:54.649800 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" podUID="c0ea7fde-2ac5-432d-bb9b-534024186ac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:04:54.917176 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:54.916968 5922 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 13:04:54.917176 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:54.917024 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" podUID="9126994e-0a2a-4558-a255-bc3f01359322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 13:04:54.917416 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:54.917332 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5db1191-qqmdj"] Oct 30 13:04:54.920868 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:54.920850 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5db1191-qqmdj" Oct 30 13:04:54.939722 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:54.939694 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5db1191-qqmdj"] Oct 30 13:04:54.952959 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:54.952924 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:04:54.953070 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:54.953053 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:04:54.953154 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:54.953141 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:04:54.953251 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:54.953239 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:04:54.998635 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:54.989041 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qksbj\" (UniqueName: \"kubernetes.io/projected/bc9f17c6-e5e1-4a1e-9525-a511055902d3-kube-api-access-qksbj\") pod \"maas-api-revoke-sub-f5db1191-qqmdj\" (UID: \"bc9f17c6-e5e1-4a1e-9525-a511055902d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5db1191-qqmdj" Oct 30 13:04:54.998635 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:54.989152 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc9f17c6-e5e1-4a1e-9525-a511055902d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5db1191-qqmdj\" (UID: \"bc9f17c6-e5e1-4a1e-9525-a511055902d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5db1191-qqmdj" Oct 30 13:04:54.998635 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:54.989212 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc9f17c6-e5e1-4a1e-9525-a511055902d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5db1191-qqmdj\" (UID: \"bc9f17c6-e5e1-4a1e-9525-a511055902d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5db1191-qqmdj" Oct 30 13:04:55.101367 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:55.101323 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qksbj\" (UniqueName: \"kubernetes.io/projected/bc9f17c6-e5e1-4a1e-9525-a511055902d3-kube-api-access-qksbj\") pod \"maas-api-revoke-sub-f5db1191-qqmdj\" (UID: \"bc9f17c6-e5e1-4a1e-9525-a511055902d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5db1191-qqmdj" Oct 30 13:04:55.101731 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:55.101704 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc9f17c6-e5e1-4a1e-9525-a511055902d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5db1191-qqmdj\" (UID: \"bc9f17c6-e5e1-4a1e-9525-a511055902d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5db1191-qqmdj" Oct 30 13:04:55.101929 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:55.101911 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc9f17c6-e5e1-4a1e-9525-a511055902d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5db1191-qqmdj\" (UID: \"bc9f17c6-e5e1-4a1e-9525-a511055902d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5db1191-qqmdj" Oct 30 13:04:55.110636 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:55.103843 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc9f17c6-e5e1-4a1e-9525-a511055902d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5db1191-qqmdj\" (UID: \"bc9f17c6-e5e1-4a1e-9525-a511055902d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5db1191-qqmdj" Oct 30 13:04:55.116015 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:55.115972 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc9f17c6-e5e1-4a1e-9525-a511055902d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5db1191-qqmdj\" (UID: \"bc9f17c6-e5e1-4a1e-9525-a511055902d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5db1191-qqmdj" Oct 30 13:04:55.130375 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:55.130357 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qksbj\" (UniqueName: \"kubernetes.io/projected/bc9f17c6-e5e1-4a1e-9525-a511055902d3-kube-api-access-qksbj\") pod \"maas-api-revoke-sub-f5db1191-qqmdj\" (UID: \"bc9f17c6-e5e1-4a1e-9525-a511055902d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5db1191-qqmdj" Oct 30 13:04:55.236297 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:55.236266 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5db1191-qqmdj" Oct 30 13:04:55.299555 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:55.299519 5922 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 13:04:55.300181 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:55.299820 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" podUID="e5fb05c2-4a10-4c85-8464-75e247feae67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 13:04:55.521825 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:55.521730 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:55.612641 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:55.611837 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5db1191-qqmdj"] Oct 30 13:04:55.930635 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:55.925804 5922 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 13:04:55.930635 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:55.925909 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" podUID="9126994e-0a2a-4558-a255-bc3f01359322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 13:04:56.308053 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:56.308011 5922 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 13:04:56.308696 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:56.308662 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" podUID="e5fb05c2-4a10-4c85-8464-75e247feae67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 13:04:56.354754 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:56.353772 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5db1191-qqmdj" event={"ID":"bc9f17c6-e5e1-4a1e-9525-a511055902d3","Type":"ContainerStarted","Data":"20676957f431a8f0488edb5186e2fe83a70c667468543ce49f036a701fb03ab9"} Oct 30 13:04:56.354754 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:56.353872 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5db1191-qqmdj" event={"ID":"bc9f17c6-e5e1-4a1e-9525-a511055902d3","Type":"ContainerStarted","Data":"64a00f63289fe64863385a24502d956a9d4c1a893503f29c67ff69abc7da93b6"} Oct 30 13:04:56.921635 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:56.920712 5922 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 13:04:56.921635 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:56.920772 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" podUID="9126994e-0a2a-4558-a255-bc3f01359322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 13:04:57.285174 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:57.285137 5922 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 13:04:57.285346 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:57.285213 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" podUID="e5fb05c2-4a10-4c85-8464-75e247feae67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 13:04:57.362124 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:57.362087 5922 generic.go:356] "Generic (PLEG): container finished" podID="bc9f17c6-e5e1-4a1e-9525-a511055902d3" containerID="20676957f431a8f0488edb5186e2fe83a70c667468543ce49f036a701fb03ab9" exitCode=0 Oct 30 13:04:57.362561 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:57.362164 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5db1191-qqmdj" event={"ID":"bc9f17c6-e5e1-4a1e-9525-a511055902d3","Type":"ContainerDied","Data":"20676957f431a8f0488edb5186e2fe83a70c667468543ce49f036a701fb03ab9"} Oct 30 13:04:57.534624 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:57.534585 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:58.704528 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:58.704501 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5db1191-qqmdj" Oct 30 13:04:58.753690 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:58.752881 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" Oct 30 13:04:58.753968 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:58.753945 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" Oct 30 13:04:58.754903 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:58.754882 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" Oct 30 13:04:58.792063 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:58.792039 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc9f17c6-e5e1-4a1e-9525-a511055902d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc9f17c6-e5e1-4a1e-9525-a511055902d3-maas-api-openshift-ca\") pod \"bc9f17c6-e5e1-4a1e-9525-a511055902d3\" (UID: \"bc9f17c6-e5e1-4a1e-9525-a511055902d3\") " Oct 30 13:04:58.792299 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:58.792280 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc9f17c6-e5e1-4a1e-9525-a511055902d3-kube-api-access-qksbj\" (UniqueName: \"kubernetes.io/projected/bc9f17c6-e5e1-4a1e-9525-a511055902d3-kube-api-access-qksbj\") pod \"bc9f17c6-e5e1-4a1e-9525-a511055902d3\" (UID: \"bc9f17c6-e5e1-4a1e-9525-a511055902d3\") " Oct 30 13:04:58.792745 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:58.792722 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc9f17c6-e5e1-4a1e-9525-a511055902d3-maas-api-openshift-ca" pod "bc9f17c6-e5e1-4a1e-9525-a511055902d3" (UID: "bc9f17c6-e5e1-4a1e-9525-a511055902d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:58.795405 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:58.795379 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc9f17c6-e5e1-4a1e-9525-a511055902d3-kube-api-access-qksbj" pod "bc9f17c6-e5e1-4a1e-9525-a511055902d3" (UID: "bc9f17c6-e5e1-4a1e-9525-a511055902d3"). InnerVolumeSpecName "kube-api-access-qksbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:58.898865 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:58.895708 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc9f17c6-e5e1-4a1e-9525-a511055902d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc9f17c6-e5e1-4a1e-9525-a511055902d3-maas-api-xks-ca\") pod \"bc9f17c6-e5e1-4a1e-9525-a511055902d3\" (UID: \"bc9f17c6-e5e1-4a1e-9525-a511055902d3\") " Oct 30 13:04:58.898865 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:58.896171 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc9f17c6-e5e1-4a1e-9525-a511055902d3-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.898865 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:58.896193 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qksbj\" (UniqueName: \"kubernetes.io/projected/bc9f17c6-e5e1-4a1e-9525-a511055902d3-kube-api-access-qksbj\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.899075 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:58.898885 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc9f17c6-e5e1-4a1e-9525-a511055902d3-maas-api-xks-ca" pod "bc9f17c6-e5e1-4a1e-9525-a511055902d3" (UID: "bc9f17c6-e5e1-4a1e-9525-a511055902d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:04:58.978068 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:58.978030 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" Oct 30 13:04:58.978414 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:58.978377 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" Oct 30 13:04:58.979286 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:58.979263 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" Oct 30 13:04:58.997071 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:58.997046 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc9f17c6-e5e1-4a1e-9525-a511055902d3-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:04:59.377188 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:59.377153 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5db1191-qqmdj" Oct 30 13:04:59.377188 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:59.377181 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5db1191-qqmdj" event={"ID":"bc9f17c6-e5e1-4a1e-9525-a511055902d3","Type":"ContainerDied","Data":"64a00f63289fe64863385a24502d956a9d4c1a893503f29c67ff69abc7da93b6"} Oct 30 13:04:59.377390 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:59.377239 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64a00f63289fe64863385a24502d956a9d4c1a893503f29c67ff69abc7da93b6" Oct 30 13:04:59.694993 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:04:59.694958 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:01.518320 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:01.518285 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:01.701253 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:01.701214 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c97ae1fc-mqwzp"] Oct 30 13:05:01.701870 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:01.701848 5922 cpu_manager.go:414] "containerMap: removing container" podUID="bc9f17c6-e5e1-4a1e-9525-a511055902d3" containerName="revoke-keys" Oct 30 13:05:01.701870 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:01.701870 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc9f17c6-e5e1-4a1e-9525-a511055902d3" containerName="revoke-keys" Oct 30 13:05:01.702053 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:01.701987 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc9f17c6-e5e1-4a1e-9525-a511055902d3" containerName="revoke-keys" Oct 30 13:05:01.704375 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:01.704353 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c97ae1fc-mqwzp" Oct 30 13:05:01.725084 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:01.725058 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c97ae1fc-mqwzp"] Oct 30 13:05:01.732787 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:01.732757 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1cc7211-6b43-4fed-874d-749577c986dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c97ae1fc-mqwzp\" (UID: \"c1cc7211-6b43-4fed-874d-749577c986dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c97ae1fc-mqwzp" Oct 30 13:05:01.732948 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:01.732914 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qrnt\" (UniqueName: \"kubernetes.io/projected/c1cc7211-6b43-4fed-874d-749577c986dc-kube-api-access-9qrnt\") pod \"maas-api-revoke-sub-c97ae1fc-mqwzp\" (UID: \"c1cc7211-6b43-4fed-874d-749577c986dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c97ae1fc-mqwzp" Oct 30 13:05:01.733058 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:01.733043 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1cc7211-6b43-4fed-874d-749577c986dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c97ae1fc-mqwzp\" (UID: \"c1cc7211-6b43-4fed-874d-749577c986dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c97ae1fc-mqwzp" Oct 30 13:05:01.733456 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:01.733425 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:05:01.733568 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:01.733497 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:05:01.733654 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:01.733595 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:05:01.733725 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:01.733711 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:05:01.758978 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:01.758951 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d278e5f4-vk9z7"] Oct 30 13:05:01.762733 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:01.762705 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d278e5f4-vk9z7" Oct 30 13:05:01.781515 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:01.781494 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d278e5f4-vk9z7"] Oct 30 13:05:01.834499 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:01.834472 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1cc7211-6b43-4fed-874d-749577c986dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c97ae1fc-mqwzp\" (UID: \"c1cc7211-6b43-4fed-874d-749577c986dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c97ae1fc-mqwzp" Oct 30 13:05:01.834674 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:01.834549 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qrnt\" (UniqueName: \"kubernetes.io/projected/c1cc7211-6b43-4fed-874d-749577c986dc-kube-api-access-9qrnt\") pod \"maas-api-revoke-sub-c97ae1fc-mqwzp\" (UID: \"c1cc7211-6b43-4fed-874d-749577c986dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c97ae1fc-mqwzp" Oct 30 13:05:01.834674 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:01.834634 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cj9b\" (UniqueName: \"kubernetes.io/projected/a94f1261-e840-4407-87c9-395005a540b1-kube-api-access-9cj9b\") pod \"maas-api-revoke-sub-d278e5f4-vk9z7\" (UID: \"a94f1261-e840-4407-87c9-395005a540b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d278e5f4-vk9z7" Oct 30 13:05:01.834797 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:01.834671 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1cc7211-6b43-4fed-874d-749577c986dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c97ae1fc-mqwzp\" (UID: \"c1cc7211-6b43-4fed-874d-749577c986dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c97ae1fc-mqwzp" Oct 30 13:05:01.834797 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:01.834772 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a94f1261-e840-4407-87c9-395005a540b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d278e5f4-vk9z7\" (UID: \"a94f1261-e840-4407-87c9-395005a540b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d278e5f4-vk9z7" Oct 30 13:05:01.834899 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:01.834875 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a94f1261-e840-4407-87c9-395005a540b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d278e5f4-vk9z7\" (UID: \"a94f1261-e840-4407-87c9-395005a540b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d278e5f4-vk9z7" Oct 30 13:05:01.835268 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:01.835245 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1cc7211-6b43-4fed-874d-749577c986dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c97ae1fc-mqwzp\" (UID: \"c1cc7211-6b43-4fed-874d-749577c986dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c97ae1fc-mqwzp" Oct 30 13:05:01.837289 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:01.837267 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1cc7211-6b43-4fed-874d-749577c986dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c97ae1fc-mqwzp\" (UID: \"c1cc7211-6b43-4fed-874d-749577c986dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c97ae1fc-mqwzp" Oct 30 13:05:01.859795 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:01.859772 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qrnt\" (UniqueName: \"kubernetes.io/projected/c1cc7211-6b43-4fed-874d-749577c986dc-kube-api-access-9qrnt\") pod \"maas-api-revoke-sub-c97ae1fc-mqwzp\" (UID: \"c1cc7211-6b43-4fed-874d-749577c986dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c97ae1fc-mqwzp" Oct 30 13:05:01.936338 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:01.936308 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a94f1261-e840-4407-87c9-395005a540b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d278e5f4-vk9z7\" (UID: \"a94f1261-e840-4407-87c9-395005a540b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d278e5f4-vk9z7" Oct 30 13:05:01.936480 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:01.936351 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a94f1261-e840-4407-87c9-395005a540b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d278e5f4-vk9z7\" (UID: \"a94f1261-e840-4407-87c9-395005a540b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d278e5f4-vk9z7" Oct 30 13:05:01.936480 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:01.936438 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cj9b\" (UniqueName: \"kubernetes.io/projected/a94f1261-e840-4407-87c9-395005a540b1-kube-api-access-9cj9b\") pod \"maas-api-revoke-sub-d278e5f4-vk9z7\" (UID: \"a94f1261-e840-4407-87c9-395005a540b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d278e5f4-vk9z7" Oct 30 13:05:01.937088 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:01.937062 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a94f1261-e840-4407-87c9-395005a540b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d278e5f4-vk9z7\" (UID: \"a94f1261-e840-4407-87c9-395005a540b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d278e5f4-vk9z7" Oct 30 13:05:01.938599 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:01.938578 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a94f1261-e840-4407-87c9-395005a540b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d278e5f4-vk9z7\" (UID: \"a94f1261-e840-4407-87c9-395005a540b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d278e5f4-vk9z7" Oct 30 13:05:01.960797 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:01.960775 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cj9b\" (UniqueName: \"kubernetes.io/projected/a94f1261-e840-4407-87c9-395005a540b1-kube-api-access-9cj9b\") pod \"maas-api-revoke-sub-d278e5f4-vk9z7\" (UID: \"a94f1261-e840-4407-87c9-395005a540b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d278e5f4-vk9z7" Oct 30 13:05:02.019634 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:02.017385 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c97ae1fc-mqwzp" Oct 30 13:05:02.077633 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:02.075495 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d278e5f4-vk9z7" Oct 30 13:05:02.326719 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:02.326186 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c97ae1fc-mqwzp"] Oct 30 13:05:02.329319 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:05:02.329291 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1cc7211_6b43_4fed_874d_749577c986dc.slice/crio-555d7f5aaa5d4b854bc5458467157389770e623df80f94b1dbbbc9d35c2d12b9 WatchSource:0}: Error finding container 555d7f5aaa5d4b854bc5458467157389770e623df80f94b1dbbbc9d35c2d12b9: Status 404 returned error can't find the container with id 555d7f5aaa5d4b854bc5458467157389770e623df80f94b1dbbbc9d35c2d12b9 Oct 30 13:05:02.395767 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:02.395732 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c97ae1fc-mqwzp" event={"ID":"c1cc7211-6b43-4fed-874d-749577c986dc","Type":"ContainerStarted","Data":"555d7f5aaa5d4b854bc5458467157389770e623df80f94b1dbbbc9d35c2d12b9"} Oct 30 13:05:02.415212 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:02.415188 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d278e5f4-vk9z7"] Oct 30 13:05:02.417989 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:05:02.417967 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda94f1261_e840_4407_87c9_395005a540b1.slice/crio-c83b699c46bde5c273bc27edbfeddf6e13a7730610e88522f5ce7aa45d975963 WatchSource:0}: Error finding container c83b699c46bde5c273bc27edbfeddf6e13a7730610e88522f5ce7aa45d975963: Status 404 returned error can't find the container with id c83b699c46bde5c273bc27edbfeddf6e13a7730610e88522f5ce7aa45d975963 Oct 30 13:05:02.948682 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:02.948651 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edbca662-tc4ch"] Oct 30 13:05:02.951775 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:02.951753 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edbca662-tc4ch" Oct 30 13:05:02.972272 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:02.972243 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edbca662-tc4ch"] Oct 30 13:05:03.051817 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:03.051792 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edbca662-tc4ch\" (UID: \"0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edbca662-tc4ch" Oct 30 13:05:03.051978 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:03.051826 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5zzc\" (UniqueName: \"kubernetes.io/projected/0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa-kube-api-access-z5zzc\") pod \"maas-api-revoke-sub-edbca662-tc4ch\" (UID: \"0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edbca662-tc4ch" Oct 30 13:05:03.051978 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:03.051850 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edbca662-tc4ch\" (UID: \"0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edbca662-tc4ch" Oct 30 13:05:03.152544 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:03.152517 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edbca662-tc4ch\" (UID: \"0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edbca662-tc4ch" Oct 30 13:05:03.152719 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:03.152551 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5zzc\" (UniqueName: \"kubernetes.io/projected/0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa-kube-api-access-z5zzc\") pod \"maas-api-revoke-sub-edbca662-tc4ch\" (UID: \"0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edbca662-tc4ch" Oct 30 13:05:03.152719 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:03.152571 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edbca662-tc4ch\" (UID: \"0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edbca662-tc4ch" Oct 30 13:05:03.153209 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:03.153187 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edbca662-tc4ch\" (UID: \"0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edbca662-tc4ch" Oct 30 13:05:03.154788 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:03.154771 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edbca662-tc4ch\" (UID: \"0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edbca662-tc4ch" Oct 30 13:05:03.176909 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:03.176886 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5zzc\" (UniqueName: \"kubernetes.io/projected/0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa-kube-api-access-z5zzc\") pod \"maas-api-revoke-sub-edbca662-tc4ch\" (UID: \"0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edbca662-tc4ch" Oct 30 13:05:03.264957 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:03.264924 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edbca662-tc4ch" Oct 30 13:05:03.291759 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:03.291728 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:03.403653 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:03.403604 5922 generic.go:356] "Generic (PLEG): container finished" podID="a94f1261-e840-4407-87c9-395005a540b1" containerID="44e6283939464347d11b827c761ff6fd91027a884eea611decf26f2a816d0a92" exitCode=0 Oct 30 13:05:03.403830 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:03.403658 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d278e5f4-vk9z7" event={"ID":"a94f1261-e840-4407-87c9-395005a540b1","Type":"ContainerDied","Data":"44e6283939464347d11b827c761ff6fd91027a884eea611decf26f2a816d0a92"} Oct 30 13:05:03.403830 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:03.403713 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d278e5f4-vk9z7" event={"ID":"a94f1261-e840-4407-87c9-395005a540b1","Type":"ContainerStarted","Data":"c83b699c46bde5c273bc27edbfeddf6e13a7730610e88522f5ce7aa45d975963"} Oct 30 13:05:03.405167 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:03.405146 5922 generic.go:356] "Generic (PLEG): container finished" podID="c1cc7211-6b43-4fed-874d-749577c986dc" containerID="71fcc72846b179a730312469ee4971897c17d850a99d0ff8d47e8c9e851d0b46" exitCode=0 Oct 30 13:05:03.405274 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:03.405223 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c97ae1fc-mqwzp" event={"ID":"c1cc7211-6b43-4fed-874d-749577c986dc","Type":"ContainerDied","Data":"71fcc72846b179a730312469ee4971897c17d850a99d0ff8d47e8c9e851d0b46"} Oct 30 13:05:03.565797 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:03.565771 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edbca662-tc4ch"] Oct 30 13:05:03.568171 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:05:03.568144 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b77e948_6004_45d0_8cfa_a1cbd3fdb0fa.slice/crio-915fd1a08d4a2b0fb8f043dfe0999561318a8ec81cd472c9d2227430c3a8c00b WatchSource:0}: Error finding container 915fd1a08d4a2b0fb8f043dfe0999561318a8ec81cd472c9d2227430c3a8c00b: Status 404 returned error can't find the container with id 915fd1a08d4a2b0fb8f043dfe0999561318a8ec81cd472c9d2227430c3a8c00b Oct 30 13:05:04.415575 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:04.415542 5922 generic.go:356] "Generic (PLEG): container finished" podID="0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa" containerID="5534485af417ce5160e60dc319437d88d6b2784413c9beb2fa1f889e32a4f1fd" exitCode=0 Oct 30 13:05:04.416034 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:04.415686 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edbca662-tc4ch" event={"ID":"0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa","Type":"ContainerDied","Data":"5534485af417ce5160e60dc319437d88d6b2784413c9beb2fa1f889e32a4f1fd"} Oct 30 13:05:04.416034 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:04.415739 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edbca662-tc4ch" event={"ID":"0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa","Type":"ContainerStarted","Data":"915fd1a08d4a2b0fb8f043dfe0999561318a8ec81cd472c9d2227430c3a8c00b"} Oct 30 13:05:04.570494 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:04.570467 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d278e5f4-vk9z7" Oct 30 13:05:04.595488 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:04.595466 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c97ae1fc-mqwzp" Oct 30 13:05:04.669085 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:04.669016 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1cc7211-6b43-4fed-874d-749577c986dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1cc7211-6b43-4fed-874d-749577c986dc-maas-api-openshift-ca\") pod \"c1cc7211-6b43-4fed-874d-749577c986dc\" (UID: \"c1cc7211-6b43-4fed-874d-749577c986dc\") " Oct 30 13:05:04.669085 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:04.669072 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1cc7211-6b43-4fed-874d-749577c986dc-kube-api-access-9qrnt\" (UniqueName: \"kubernetes.io/projected/c1cc7211-6b43-4fed-874d-749577c986dc-kube-api-access-9qrnt\") pod \"c1cc7211-6b43-4fed-874d-749577c986dc\" (UID: \"c1cc7211-6b43-4fed-874d-749577c986dc\") " Oct 30 13:05:04.669307 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:04.669109 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a94f1261-e840-4407-87c9-395005a540b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a94f1261-e840-4407-87c9-395005a540b1-maas-api-openshift-ca\") pod \"a94f1261-e840-4407-87c9-395005a540b1\" (UID: \"a94f1261-e840-4407-87c9-395005a540b1\") " Oct 30 13:05:04.669307 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:04.669154 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a94f1261-e840-4407-87c9-395005a540b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a94f1261-e840-4407-87c9-395005a540b1-maas-api-xks-ca\") pod \"a94f1261-e840-4407-87c9-395005a540b1\" (UID: \"a94f1261-e840-4407-87c9-395005a540b1\") " Oct 30 13:05:04.669307 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:04.669186 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1cc7211-6b43-4fed-874d-749577c986dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1cc7211-6b43-4fed-874d-749577c986dc-maas-api-xks-ca\") pod \"c1cc7211-6b43-4fed-874d-749577c986dc\" (UID: \"c1cc7211-6b43-4fed-874d-749577c986dc\") " Oct 30 13:05:04.669307 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:04.669219 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a94f1261-e840-4407-87c9-395005a540b1-kube-api-access-9cj9b\" (UniqueName: \"kubernetes.io/projected/a94f1261-e840-4407-87c9-395005a540b1-kube-api-access-9cj9b\") pod \"a94f1261-e840-4407-87c9-395005a540b1\" (UID: \"a94f1261-e840-4407-87c9-395005a540b1\") " Oct 30 13:05:04.669754 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:04.669724 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a94f1261-e840-4407-87c9-395005a540b1-maas-api-openshift-ca" pod "a94f1261-e840-4407-87c9-395005a540b1" (UID: "a94f1261-e840-4407-87c9-395005a540b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:04.669856 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:04.669728 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1cc7211-6b43-4fed-874d-749577c986dc-maas-api-openshift-ca" pod "c1cc7211-6b43-4fed-874d-749577c986dc" (UID: "c1cc7211-6b43-4fed-874d-749577c986dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:04.670400 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:04.670377 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1cc7211-6b43-4fed-874d-749577c986dc-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:05:04.670512 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:04.670405 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a94f1261-e840-4407-87c9-395005a540b1-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:05:04.671742 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:04.671715 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a94f1261-e840-4407-87c9-395005a540b1-maas-api-xks-ca" pod "a94f1261-e840-4407-87c9-395005a540b1" (UID: "a94f1261-e840-4407-87c9-395005a540b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:05:04.671823 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:04.671794 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1cc7211-6b43-4fed-874d-749577c986dc-maas-api-xks-ca" pod "c1cc7211-6b43-4fed-874d-749577c986dc" (UID: "c1cc7211-6b43-4fed-874d-749577c986dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:05:04.671940 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:04.671923 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1cc7211-6b43-4fed-874d-749577c986dc-kube-api-access-9qrnt" pod "c1cc7211-6b43-4fed-874d-749577c986dc" (UID: "c1cc7211-6b43-4fed-874d-749577c986dc"). InnerVolumeSpecName "kube-api-access-9qrnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:04.672305 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:04.672286 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a94f1261-e840-4407-87c9-395005a540b1-kube-api-access-9cj9b" pod "a94f1261-e840-4407-87c9-395005a540b1" (UID: "a94f1261-e840-4407-87c9-395005a540b1"). InnerVolumeSpecName "kube-api-access-9cj9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:04.771086 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:04.771055 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qrnt\" (UniqueName: \"kubernetes.io/projected/c1cc7211-6b43-4fed-874d-749577c986dc-kube-api-access-9qrnt\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:05:04.771086 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:04.771085 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a94f1261-e840-4407-87c9-395005a540b1-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:05:04.771086 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:04.771097 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1cc7211-6b43-4fed-874d-749577c986dc-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:05:04.771312 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:04.771110 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cj9b\" (UniqueName: \"kubernetes.io/projected/a94f1261-e840-4407-87c9-395005a540b1-kube-api-access-9cj9b\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:05:05.214184 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:05.214151 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:05.264970 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:05.264912 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0aed1791-cwtk2"] Oct 30 13:05:05.265558 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:05.265539 5922 cpu_manager.go:414] "containerMap: removing container" podUID="a94f1261-e840-4407-87c9-395005a540b1" containerName="revoke-keys" Oct 30 13:05:05.265558 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:05.265557 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a94f1261-e840-4407-87c9-395005a540b1" containerName="revoke-keys" Oct 30 13:05:05.265706 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:05.265580 5922 cpu_manager.go:414] "containerMap: removing container" podUID="c1cc7211-6b43-4fed-874d-749577c986dc" containerName="revoke-keys" Oct 30 13:05:05.265706 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:05.265585 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1cc7211-6b43-4fed-874d-749577c986dc" containerName="revoke-keys" Oct 30 13:05:05.265778 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:05.265743 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="a94f1261-e840-4407-87c9-395005a540b1" containerName="revoke-keys" Oct 30 13:05:05.265778 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:05.265753 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1cc7211-6b43-4fed-874d-749577c986dc" containerName="revoke-keys" Oct 30 13:05:05.275531 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:05.275502 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aed1791-cwtk2" Oct 30 13:05:05.288016 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:05.287994 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0aed1791-cwtk2"] Oct 30 13:05:05.377586 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:05.377555 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7lz2\" (UniqueName: \"kubernetes.io/projected/215248bc-d335-49c0-96a0-6835913f5b35-kube-api-access-r7lz2\") pod \"maas-api-revoke-sub-0aed1791-cwtk2\" (UID: \"215248bc-d335-49c0-96a0-6835913f5b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aed1791-cwtk2" Oct 30 13:05:05.377586 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:05.377590 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/215248bc-d335-49c0-96a0-6835913f5b35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0aed1791-cwtk2\" (UID: \"215248bc-d335-49c0-96a0-6835913f5b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aed1791-cwtk2" Oct 30 13:05:05.377783 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:05.377723 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/215248bc-d335-49c0-96a0-6835913f5b35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0aed1791-cwtk2\" (UID: \"215248bc-d335-49c0-96a0-6835913f5b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aed1791-cwtk2" Oct 30 13:05:05.423285 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:05.423259 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d278e5f4-vk9z7" Oct 30 13:05:05.423701 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:05.423285 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d278e5f4-vk9z7" event={"ID":"a94f1261-e840-4407-87c9-395005a540b1","Type":"ContainerDied","Data":"c83b699c46bde5c273bc27edbfeddf6e13a7730610e88522f5ce7aa45d975963"} Oct 30 13:05:05.423701 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:05.423356 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c83b699c46bde5c273bc27edbfeddf6e13a7730610e88522f5ce7aa45d975963" Oct 30 13:05:05.424699 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:05.424673 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c97ae1fc-mqwzp" event={"ID":"c1cc7211-6b43-4fed-874d-749577c986dc","Type":"ContainerDied","Data":"555d7f5aaa5d4b854bc5458467157389770e623df80f94b1dbbbc9d35c2d12b9"} Oct 30 13:05:05.424792 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:05.424714 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="555d7f5aaa5d4b854bc5458467157389770e623df80f94b1dbbbc9d35c2d12b9" Oct 30 13:05:05.424836 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:05.424789 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c97ae1fc-mqwzp" Oct 30 13:05:05.478858 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:05.478783 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7lz2\" (UniqueName: \"kubernetes.io/projected/215248bc-d335-49c0-96a0-6835913f5b35-kube-api-access-r7lz2\") pod \"maas-api-revoke-sub-0aed1791-cwtk2\" (UID: \"215248bc-d335-49c0-96a0-6835913f5b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aed1791-cwtk2" Oct 30 13:05:05.478858 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:05.478831 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/215248bc-d335-49c0-96a0-6835913f5b35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0aed1791-cwtk2\" (UID: \"215248bc-d335-49c0-96a0-6835913f5b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aed1791-cwtk2" Oct 30 13:05:05.479059 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:05.478913 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/215248bc-d335-49c0-96a0-6835913f5b35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0aed1791-cwtk2\" (UID: \"215248bc-d335-49c0-96a0-6835913f5b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aed1791-cwtk2" Oct 30 13:05:05.479628 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:05.479592 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/215248bc-d335-49c0-96a0-6835913f5b35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0aed1791-cwtk2\" (UID: \"215248bc-d335-49c0-96a0-6835913f5b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aed1791-cwtk2" Oct 30 13:05:05.481853 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:05.481695 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/215248bc-d335-49c0-96a0-6835913f5b35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0aed1791-cwtk2\" (UID: \"215248bc-d335-49c0-96a0-6835913f5b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aed1791-cwtk2" Oct 30 13:05:05.505803 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:05.505779 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7lz2\" (UniqueName: \"kubernetes.io/projected/215248bc-d335-49c0-96a0-6835913f5b35-kube-api-access-r7lz2\") pod \"maas-api-revoke-sub-0aed1791-cwtk2\" (UID: \"215248bc-d335-49c0-96a0-6835913f5b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aed1791-cwtk2" Oct 30 13:05:05.576753 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:05.576733 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edbca662-tc4ch" Oct 30 13:05:05.602378 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:05.602356 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aed1791-cwtk2" Oct 30 13:05:05.681982 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:05.681945 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa-kube-api-access-z5zzc\" (UniqueName: \"kubernetes.io/projected/0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa-kube-api-access-z5zzc\") pod \"0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa\" (UID: \"0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa\") " Oct 30 13:05:05.682177 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:05.682154 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa-maas-api-openshift-ca\") pod \"0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa\" (UID: \"0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa\") " Oct 30 13:05:05.682293 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:05.682208 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa-maas-api-xks-ca\") pod \"0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa\" (UID: \"0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa\") " Oct 30 13:05:05.682773 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:05.682709 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa-maas-api-openshift-ca" pod "0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa" (UID: "0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:05.685013 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:05.684987 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa-kube-api-access-z5zzc" pod "0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa" (UID: "0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa"). InnerVolumeSpecName "kube-api-access-z5zzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:05.685107 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:05.685061 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa-maas-api-xks-ca" pod "0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa" (UID: "0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:05:05.784261 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:05.784191 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:05:05.784261 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:05.784218 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:05:05.784261 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:05.784228 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5zzc\" (UniqueName: \"kubernetes.io/projected/0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa-kube-api-access-z5zzc\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:05:05.906212 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:05.906185 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0aed1791-cwtk2"] Oct 30 13:05:05.908746 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:05:05.908713 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod215248bc_d335_49c0_96a0_6835913f5b35.slice/crio-bf166537c9fc46053ec54f2b80a886e0efaee0d1f0f990bfdd60477055ca0354 WatchSource:0}: Error finding container bf166537c9fc46053ec54f2b80a886e0efaee0d1f0f990bfdd60477055ca0354: Status 404 returned error can't find the container with id bf166537c9fc46053ec54f2b80a886e0efaee0d1f0f990bfdd60477055ca0354 Oct 30 13:05:06.431931 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:06.431853 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edbca662-tc4ch" Oct 30 13:05:06.431931 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:06.431891 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edbca662-tc4ch" event={"ID":"0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa","Type":"ContainerDied","Data":"915fd1a08d4a2b0fb8f043dfe0999561318a8ec81cd472c9d2227430c3a8c00b"} Oct 30 13:05:06.432408 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:06.431935 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="915fd1a08d4a2b0fb8f043dfe0999561318a8ec81cd472c9d2227430c3a8c00b" Oct 30 13:05:06.433474 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:06.433452 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aed1791-cwtk2" event={"ID":"215248bc-d335-49c0-96a0-6835913f5b35","Type":"ContainerStarted","Data":"68207c3d77ef68e743e90e957aeaf91efb4226aee63410b423af20c431974e74"} Oct 30 13:05:06.433576 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:06.433488 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aed1791-cwtk2" event={"ID":"215248bc-d335-49c0-96a0-6835913f5b35","Type":"ContainerStarted","Data":"bf166537c9fc46053ec54f2b80a886e0efaee0d1f0f990bfdd60477055ca0354"} Oct 30 13:05:06.476353 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:06.476312 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aed1791-cwtk2" podStartSLOduration=1.476299358 podStartE2EDuration="1.476299358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:06.474770119 +0000 UTC m=+762.944185884" watchObservedRunningTime="2026-10-30 13:05:06.476299358 +0000 UTC m=+762.945715152" Oct 30 13:05:06.995552 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:06.995520 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:07.448844 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:07.448804 5922 generic.go:356] "Generic (PLEG): container finished" podID="215248bc-d335-49c0-96a0-6835913f5b35" containerID="68207c3d77ef68e743e90e957aeaf91efb4226aee63410b423af20c431974e74" exitCode=0 Oct 30 13:05:07.449304 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:07.448863 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aed1791-cwtk2" event={"ID":"215248bc-d335-49c0-96a0-6835913f5b35","Type":"ContainerDied","Data":"68207c3d77ef68e743e90e957aeaf91efb4226aee63410b423af20c431974e74"} Oct 30 13:05:09.219635 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:09.215119 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aed1791-cwtk2" Oct 30 13:05:09.297935 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:09.297895 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/215248bc-d335-49c0-96a0-6835913f5b35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/215248bc-d335-49c0-96a0-6835913f5b35-maas-api-openshift-ca\") pod \"215248bc-d335-49c0-96a0-6835913f5b35\" (UID: \"215248bc-d335-49c0-96a0-6835913f5b35\") " Oct 30 13:05:09.298104 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:09.298013 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/215248bc-d335-49c0-96a0-6835913f5b35-kube-api-access-r7lz2\" (UniqueName: \"kubernetes.io/projected/215248bc-d335-49c0-96a0-6835913f5b35-kube-api-access-r7lz2\") pod \"215248bc-d335-49c0-96a0-6835913f5b35\" (UID: \"215248bc-d335-49c0-96a0-6835913f5b35\") " Oct 30 13:05:09.298104 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:09.298096 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/215248bc-d335-49c0-96a0-6835913f5b35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/215248bc-d335-49c0-96a0-6835913f5b35-maas-api-xks-ca\") pod \"215248bc-d335-49c0-96a0-6835913f5b35\" (UID: \"215248bc-d335-49c0-96a0-6835913f5b35\") " Oct 30 13:05:09.298265 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:09.298234 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/215248bc-d335-49c0-96a0-6835913f5b35-maas-api-openshift-ca" pod "215248bc-d335-49c0-96a0-6835913f5b35" (UID: "215248bc-d335-49c0-96a0-6835913f5b35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:09.298762 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:09.298736 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/215248bc-d335-49c0-96a0-6835913f5b35-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:05:09.300153 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:09.300119 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/215248bc-d335-49c0-96a0-6835913f5b35-kube-api-access-r7lz2" pod "215248bc-d335-49c0-96a0-6835913f5b35" (UID: "215248bc-d335-49c0-96a0-6835913f5b35"). InnerVolumeSpecName "kube-api-access-r7lz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:09.300243 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:09.300224 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/215248bc-d335-49c0-96a0-6835913f5b35-maas-api-xks-ca" pod "215248bc-d335-49c0-96a0-6835913f5b35" (UID: "215248bc-d335-49c0-96a0-6835913f5b35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:05:09.399895 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:09.399822 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7lz2\" (UniqueName: \"kubernetes.io/projected/215248bc-d335-49c0-96a0-6835913f5b35-kube-api-access-r7lz2\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:05:09.399895 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:09.399856 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/215248bc-d335-49c0-96a0-6835913f5b35-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:05:09.472244 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:09.472218 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aed1791-cwtk2" Oct 30 13:05:09.472359 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:09.472256 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aed1791-cwtk2" event={"ID":"215248bc-d335-49c0-96a0-6835913f5b35","Type":"ContainerDied","Data":"bf166537c9fc46053ec54f2b80a886e0efaee0d1f0f990bfdd60477055ca0354"} Oct 30 13:05:09.472359 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:09.472311 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf166537c9fc46053ec54f2b80a886e0efaee0d1f0f990bfdd60477055ca0354" Oct 30 13:05:09.763865 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:09.763826 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:10.986901 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:10.986871 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a85a078-798zq"] Oct 30 13:05:10.987907 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:10.987884 5922 cpu_manager.go:414] "containerMap: removing container" podUID="215248bc-d335-49c0-96a0-6835913f5b35" containerName="revoke-keys" Oct 30 13:05:10.987907 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:10.987907 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="215248bc-d335-49c0-96a0-6835913f5b35" containerName="revoke-keys" Oct 30 13:05:10.988085 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:10.987950 5922 cpu_manager.go:414] "containerMap: removing container" podUID="0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa" containerName="revoke-keys" Oct 30 13:05:10.988085 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:10.987958 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa" containerName="revoke-keys" Oct 30 13:05:10.988189 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:10.988112 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="215248bc-d335-49c0-96a0-6835913f5b35" containerName="revoke-keys" Oct 30 13:05:10.988189 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:10.988134 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa" containerName="revoke-keys" Oct 30 13:05:10.991310 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:10.991271 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a85a078-798zq" Oct 30 13:05:11.024956 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:11.024924 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:05:11.025531 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:11.025510 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:05:11.029596 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:11.029573 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a85a078-798zq"] Oct 30 13:05:11.032870 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:11.032813 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:05:11.032962 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:11.032891 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:05:11.034437 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:11.034408 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aac684d3-17d4-4d51-a781-81a824f44956-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a85a078-798zq\" (UID: \"aac684d3-17d4-4d51-a781-81a824f44956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a85a078-798zq" Oct 30 13:05:11.034572 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:11.034554 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rclzv\" (UniqueName: \"kubernetes.io/projected/aac684d3-17d4-4d51-a781-81a824f44956-kube-api-access-rclzv\") pod \"maas-api-revoke-sub-7a85a078-798zq\" (UID: \"aac684d3-17d4-4d51-a781-81a824f44956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a85a078-798zq" Oct 30 13:05:11.034703 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:11.034584 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aac684d3-17d4-4d51-a781-81a824f44956-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a85a078-798zq\" (UID: \"aac684d3-17d4-4d51-a781-81a824f44956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a85a078-798zq" Oct 30 13:05:11.135906 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:11.135873 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aac684d3-17d4-4d51-a781-81a824f44956-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a85a078-798zq\" (UID: \"aac684d3-17d4-4d51-a781-81a824f44956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a85a078-798zq" Oct 30 13:05:11.136037 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:11.135963 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rclzv\" (UniqueName: \"kubernetes.io/projected/aac684d3-17d4-4d51-a781-81a824f44956-kube-api-access-rclzv\") pod \"maas-api-revoke-sub-7a85a078-798zq\" (UID: \"aac684d3-17d4-4d51-a781-81a824f44956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a85a078-798zq" Oct 30 13:05:11.136037 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:11.135986 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aac684d3-17d4-4d51-a781-81a824f44956-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a85a078-798zq\" (UID: \"aac684d3-17d4-4d51-a781-81a824f44956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a85a078-798zq" Oct 30 13:05:11.136557 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:11.136533 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aac684d3-17d4-4d51-a781-81a824f44956-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a85a078-798zq\" (UID: \"aac684d3-17d4-4d51-a781-81a824f44956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a85a078-798zq" Oct 30 13:05:11.138175 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:11.138154 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aac684d3-17d4-4d51-a781-81a824f44956-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a85a078-798zq\" (UID: \"aac684d3-17d4-4d51-a781-81a824f44956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a85a078-798zq" Oct 30 13:05:11.162536 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:11.162511 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rclzv\" (UniqueName: \"kubernetes.io/projected/aac684d3-17d4-4d51-a781-81a824f44956-kube-api-access-rclzv\") pod \"maas-api-revoke-sub-7a85a078-798zq\" (UID: \"aac684d3-17d4-4d51-a781-81a824f44956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a85a078-798zq" Oct 30 13:05:11.328181 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:11.328098 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a85a078-798zq" Oct 30 13:05:11.655665 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:11.655643 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a85a078-798zq"] Oct 30 13:05:11.657945 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:05:11.657916 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaac684d3_17d4_4d51_a781_81a824f44956.slice/crio-2a374600e3fb988880e3b9e8dec247caaa74535b63fa6e22199771e5958bd97d WatchSource:0}: Error finding container 2a374600e3fb988880e3b9e8dec247caaa74535b63fa6e22199771e5958bd97d: Status 404 returned error can't find the container with id 2a374600e3fb988880e3b9e8dec247caaa74535b63fa6e22199771e5958bd97d Oct 30 13:05:12.109745 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:12.109713 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:12.499018 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:12.498981 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a85a078-798zq" event={"ID":"aac684d3-17d4-4d51-a781-81a824f44956","Type":"ContainerStarted","Data":"8047d4d35ea048be67b7f96e4a0cd5a4f728e49882f631288f3c7a0abc6f81ce"} Oct 30 13:05:12.499182 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:12.499036 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a85a078-798zq" event={"ID":"aac684d3-17d4-4d51-a781-81a824f44956","Type":"ContainerStarted","Data":"2a374600e3fb988880e3b9e8dec247caaa74535b63fa6e22199771e5958bd97d"} Oct 30 13:05:12.544379 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:12.544330 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a85a078-798zq" podStartSLOduration=2.544317544 podStartE2EDuration="2.544317544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:12.541573322 +0000 UTC m=+769.010989067" watchObservedRunningTime="2026-10-30 13:05:12.544317544 +0000 UTC m=+769.013733289" Oct 30 13:05:13.505850 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:13.505815 5922 generic.go:356] "Generic (PLEG): container finished" podID="aac684d3-17d4-4d51-a781-81a824f44956" containerID="8047d4d35ea048be67b7f96e4a0cd5a4f728e49882f631288f3c7a0abc6f81ce" exitCode=0 Oct 30 13:05:13.506248 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:13.505893 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a85a078-798zq" event={"ID":"aac684d3-17d4-4d51-a781-81a824f44956","Type":"ContainerDied","Data":"8047d4d35ea048be67b7f96e4a0cd5a4f728e49882f631288f3c7a0abc6f81ce"} Oct 30 13:05:14.082161 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:14.082118 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:14.678966 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:14.678943 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a85a078-798zq" Oct 30 13:05:14.772388 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:14.772317 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aac684d3-17d4-4d51-a781-81a824f44956-kube-api-access-rclzv\" (UniqueName: \"kubernetes.io/projected/aac684d3-17d4-4d51-a781-81a824f44956-kube-api-access-rclzv\") pod \"aac684d3-17d4-4d51-a781-81a824f44956\" (UID: \"aac684d3-17d4-4d51-a781-81a824f44956\") " Oct 30 13:05:14.772388 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:14.772365 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aac684d3-17d4-4d51-a781-81a824f44956-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aac684d3-17d4-4d51-a781-81a824f44956-maas-api-openshift-ca\") pod \"aac684d3-17d4-4d51-a781-81a824f44956\" (UID: \"aac684d3-17d4-4d51-a781-81a824f44956\") " Oct 30 13:05:14.772578 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:14.772475 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aac684d3-17d4-4d51-a781-81a824f44956-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aac684d3-17d4-4d51-a781-81a824f44956-maas-api-xks-ca\") pod \"aac684d3-17d4-4d51-a781-81a824f44956\" (UID: \"aac684d3-17d4-4d51-a781-81a824f44956\") " Oct 30 13:05:14.773047 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:14.772825 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aac684d3-17d4-4d51-a781-81a824f44956-maas-api-openshift-ca" pod "aac684d3-17d4-4d51-a781-81a824f44956" (UID: "aac684d3-17d4-4d51-a781-81a824f44956"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:14.773277 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:14.773261 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aac684d3-17d4-4d51-a781-81a824f44956-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:05:14.775092 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:14.775065 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aac684d3-17d4-4d51-a781-81a824f44956-maas-api-xks-ca" pod "aac684d3-17d4-4d51-a781-81a824f44956" (UID: "aac684d3-17d4-4d51-a781-81a824f44956"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:05:14.775191 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:14.775147 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aac684d3-17d4-4d51-a781-81a824f44956-kube-api-access-rclzv" pod "aac684d3-17d4-4d51-a781-81a824f44956" (UID: "aac684d3-17d4-4d51-a781-81a824f44956"). InnerVolumeSpecName "kube-api-access-rclzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:14.874420 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:14.874384 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aac684d3-17d4-4d51-a781-81a824f44956-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:05:14.874420 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:14.874414 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rclzv\" (UniqueName: \"kubernetes.io/projected/aac684d3-17d4-4d51-a781-81a824f44956-kube-api-access-rclzv\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:05:15.527869 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:15.527838 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a85a078-798zq" Oct 30 13:05:15.527869 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:15.527862 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a85a078-798zq" event={"ID":"aac684d3-17d4-4d51-a781-81a824f44956","Type":"ContainerDied","Data":"2a374600e3fb988880e3b9e8dec247caaa74535b63fa6e22199771e5958bd97d"} Oct 30 13:05:15.528061 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:15.527907 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a374600e3fb988880e3b9e8dec247caaa74535b63fa6e22199771e5958bd97d" Oct 30 13:05:15.611035 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:15.611003 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc8032e6-xsrnf"] Oct 30 13:05:15.611512 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:15.611500 5922 cpu_manager.go:414] "containerMap: removing container" podUID="aac684d3-17d4-4d51-a781-81a824f44956" containerName="revoke-keys" Oct 30 13:05:15.611558 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:15.611513 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aac684d3-17d4-4d51-a781-81a824f44956" containerName="revoke-keys" Oct 30 13:05:15.611605 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:15.611596 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="aac684d3-17d4-4d51-a781-81a824f44956" containerName="revoke-keys" Oct 30 13:05:15.613876 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:15.613861 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8032e6-xsrnf" Oct 30 13:05:15.631478 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:15.631455 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:05:15.631605 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:15.631580 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:05:15.631720 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:15.631708 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:05:15.631771 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:15.631710 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:05:15.647644 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:15.647602 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc8032e6-xsrnf"] Oct 30 13:05:15.683885 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:15.683856 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0484e59d-627c-4eab-a3b5-2436e0a6c7a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc8032e6-xsrnf\" (UID: \"0484e59d-627c-4eab-a3b5-2436e0a6c7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8032e6-xsrnf" Oct 30 13:05:15.684346 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:15.683949 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0484e59d-627c-4eab-a3b5-2436e0a6c7a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc8032e6-xsrnf\" (UID: \"0484e59d-627c-4eab-a3b5-2436e0a6c7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8032e6-xsrnf" Oct 30 13:05:15.684346 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:15.684079 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b88tf\" (UniqueName: \"kubernetes.io/projected/0484e59d-627c-4eab-a3b5-2436e0a6c7a0-kube-api-access-b88tf\") pod \"maas-api-revoke-sub-dc8032e6-xsrnf\" (UID: \"0484e59d-627c-4eab-a3b5-2436e0a6c7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8032e6-xsrnf" Oct 30 13:05:15.785410 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:15.785344 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0484e59d-627c-4eab-a3b5-2436e0a6c7a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc8032e6-xsrnf\" (UID: \"0484e59d-627c-4eab-a3b5-2436e0a6c7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8032e6-xsrnf" Oct 30 13:05:15.785410 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:15.785388 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b88tf\" (UniqueName: \"kubernetes.io/projected/0484e59d-627c-4eab-a3b5-2436e0a6c7a0-kube-api-access-b88tf\") pod \"maas-api-revoke-sub-dc8032e6-xsrnf\" (UID: \"0484e59d-627c-4eab-a3b5-2436e0a6c7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8032e6-xsrnf" Oct 30 13:05:15.785559 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:15.785467 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0484e59d-627c-4eab-a3b5-2436e0a6c7a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc8032e6-xsrnf\" (UID: \"0484e59d-627c-4eab-a3b5-2436e0a6c7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8032e6-xsrnf" Oct 30 13:05:15.785999 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:15.785978 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0484e59d-627c-4eab-a3b5-2436e0a6c7a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc8032e6-xsrnf\" (UID: \"0484e59d-627c-4eab-a3b5-2436e0a6c7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8032e6-xsrnf" Oct 30 13:05:15.787688 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:15.787668 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0484e59d-627c-4eab-a3b5-2436e0a6c7a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc8032e6-xsrnf\" (UID: \"0484e59d-627c-4eab-a3b5-2436e0a6c7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8032e6-xsrnf" Oct 30 13:05:15.810963 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:15.810934 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b88tf\" (UniqueName: \"kubernetes.io/projected/0484e59d-627c-4eab-a3b5-2436e0a6c7a0-kube-api-access-b88tf\") pod \"maas-api-revoke-sub-dc8032e6-xsrnf\" (UID: \"0484e59d-627c-4eab-a3b5-2436e0a6c7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8032e6-xsrnf" Oct 30 13:05:15.926488 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:15.926461 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8032e6-xsrnf" Oct 30 13:05:16.052286 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:16.052200 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:16.228085 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:16.228061 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc8032e6-xsrnf"] Oct 30 13:05:16.230441 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:05:16.230410 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0484e59d_627c_4eab_a3b5_2436e0a6c7a0.slice/crio-0322ad6f25d6d60baa90ae824f43cc5c5f3058a012ab6c969b985e76517e2c59 WatchSource:0}: Error finding container 0322ad6f25d6d60baa90ae824f43cc5c5f3058a012ab6c969b985e76517e2c59: Status 404 returned error can't find the container with id 0322ad6f25d6d60baa90ae824f43cc5c5f3058a012ab6c969b985e76517e2c59 Oct 30 13:05:16.535674 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:16.535643 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8032e6-xsrnf" event={"ID":"0484e59d-627c-4eab-a3b5-2436e0a6c7a0","Type":"ContainerStarted","Data":"0322ad6f25d6d60baa90ae824f43cc5c5f3058a012ab6c969b985e76517e2c59"} Oct 30 13:05:17.543361 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:17.543278 5922 generic.go:356] "Generic (PLEG): container finished" podID="0484e59d-627c-4eab-a3b5-2436e0a6c7a0" containerID="eada85d9138e5d81244704ecad9ca1629c3afa876b31882c0d736fad205b6d9c" exitCode=0 Oct 30 13:05:17.543757 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:17.543358 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8032e6-xsrnf" event={"ID":"0484e59d-627c-4eab-a3b5-2436e0a6c7a0","Type":"ContainerDied","Data":"eada85d9138e5d81244704ecad9ca1629c3afa876b31882c0d736fad205b6d9c"} Oct 30 13:05:17.879066 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:17.878994 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:18.536102 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:18.536071 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:05:18.700454 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:18.700430 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8032e6-xsrnf" Oct 30 13:05:18.817367 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:18.817292 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0484e59d-627c-4eab-a3b5-2436e0a6c7a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0484e59d-627c-4eab-a3b5-2436e0a6c7a0-maas-api-xks-ca\") pod \"0484e59d-627c-4eab-a3b5-2436e0a6c7a0\" (UID: \"0484e59d-627c-4eab-a3b5-2436e0a6c7a0\") " Oct 30 13:05:18.817514 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:18.817368 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0484e59d-627c-4eab-a3b5-2436e0a6c7a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0484e59d-627c-4eab-a3b5-2436e0a6c7a0-maas-api-openshift-ca\") pod \"0484e59d-627c-4eab-a3b5-2436e0a6c7a0\" (UID: \"0484e59d-627c-4eab-a3b5-2436e0a6c7a0\") " Oct 30 13:05:18.817514 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:18.817416 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0484e59d-627c-4eab-a3b5-2436e0a6c7a0-kube-api-access-b88tf\" (UniqueName: \"kubernetes.io/projected/0484e59d-627c-4eab-a3b5-2436e0a6c7a0-kube-api-access-b88tf\") pod \"0484e59d-627c-4eab-a3b5-2436e0a6c7a0\" (UID: \"0484e59d-627c-4eab-a3b5-2436e0a6c7a0\") " Oct 30 13:05:18.817812 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:18.817781 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0484e59d-627c-4eab-a3b5-2436e0a6c7a0-maas-api-openshift-ca" pod "0484e59d-627c-4eab-a3b5-2436e0a6c7a0" (UID: "0484e59d-627c-4eab-a3b5-2436e0a6c7a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:18.817936 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:18.817920 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0484e59d-627c-4eab-a3b5-2436e0a6c7a0-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:05:18.819443 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:18.819419 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0484e59d-627c-4eab-a3b5-2436e0a6c7a0-kube-api-access-b88tf" pod "0484e59d-627c-4eab-a3b5-2436e0a6c7a0" (UID: "0484e59d-627c-4eab-a3b5-2436e0a6c7a0"). InnerVolumeSpecName "kube-api-access-b88tf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:18.819522 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:18.819437 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0484e59d-627c-4eab-a3b5-2436e0a6c7a0-maas-api-xks-ca" pod "0484e59d-627c-4eab-a3b5-2436e0a6c7a0" (UID: "0484e59d-627c-4eab-a3b5-2436e0a6c7a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:05:18.919518 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:18.919493 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0484e59d-627c-4eab-a3b5-2436e0a6c7a0-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:05:18.919518 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:18.919523 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b88tf\" (UniqueName: \"kubernetes.io/projected/0484e59d-627c-4eab-a3b5-2436e0a6c7a0-kube-api-access-b88tf\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:05:19.379598 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:19.379566 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:19.557912 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:19.557878 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8032e6-xsrnf" event={"ID":"0484e59d-627c-4eab-a3b5-2436e0a6c7a0","Type":"ContainerDied","Data":"0322ad6f25d6d60baa90ae824f43cc5c5f3058a012ab6c969b985e76517e2c59"} Oct 30 13:05:19.557912 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:19.557900 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8032e6-xsrnf" Oct 30 13:05:19.558075 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:19.557961 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0322ad6f25d6d60baa90ae824f43cc5c5f3058a012ab6c969b985e76517e2c59" Oct 30 13:05:19.891646 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:19.891337 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:22.703916 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:22.703879 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:23.553325 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:05:23.553293 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:05:25.117712 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:25.117677 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:27.043426 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:27.043389 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:29.098308 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:29.098267 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:31.111016 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:31.110982 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:32.341262 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:32.341231 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:32.948860 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:32.948827 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:34.404146 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:34.404112 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd"] Oct 30 13:05:34.404722 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:34.404702 5922 cpu_manager.go:414] "containerMap: removing container" podUID="0484e59d-627c-4eab-a3b5-2436e0a6c7a0" containerName="revoke-keys" Oct 30 13:05:34.404845 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:34.404724 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0484e59d-627c-4eab-a3b5-2436e0a6c7a0" containerName="revoke-keys" Oct 30 13:05:34.404845 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:34.404840 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="0484e59d-627c-4eab-a3b5-2436e0a6c7a0" containerName="revoke-keys" Oct 30 13:05:34.407745 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:34.407722 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" Oct 30 13:05:34.424329 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:34.424305 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-ac0c58-openshift-default-dockercfg-5mbd5\"" Oct 30 13:05:34.439596 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:34.439523 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd"] Oct 30 13:05:34.508255 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:34.508216 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef454a96-712b-4ece-b4d8-b116964867a5-credential-socket\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd\" (UID: \"ef454a96-712b-4ece-b4d8-b116964867a5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" Oct 30 13:05:34.508425 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:34.508276 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef454a96-712b-4ece-b4d8-b116964867a5-istio-token\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd\" (UID: \"ef454a96-712b-4ece-b4d8-b116964867a5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" Oct 30 13:05:34.508425 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:34.508352 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef454a96-712b-4ece-b4d8-b116964867a5-istio-envoy\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd\" (UID: \"ef454a96-712b-4ece-b4d8-b116964867a5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" Oct 30 13:05:34.508425 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:34.508406 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef454a96-712b-4ece-b4d8-b116964867a5-istiod-ca-cert\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd\" (UID: \"ef454a96-712b-4ece-b4d8-b116964867a5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" Oct 30 13:05:34.508538 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:34.508505 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef454a96-712b-4ece-b4d8-b116964867a5-istio-data\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd\" (UID: \"ef454a96-712b-4ece-b4d8-b116964867a5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" Oct 30 13:05:34.508595 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:34.508580 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef454a96-712b-4ece-b4d8-b116964867a5-workload-certs\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd\" (UID: \"ef454a96-712b-4ece-b4d8-b116964867a5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" Oct 30 13:05:34.508644 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:34.508604 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef454a96-712b-4ece-b4d8-b116964867a5-istio-podinfo\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd\" (UID: \"ef454a96-712b-4ece-b4d8-b116964867a5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" Oct 30 13:05:34.508684 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:34.508647 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wm6z\" (UniqueName: \"kubernetes.io/projected/ef454a96-712b-4ece-b4d8-b116964867a5-kube-api-access-2wm6z\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd\" (UID: \"ef454a96-712b-4ece-b4d8-b116964867a5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" Oct 30 13:05:34.508764 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:34.508743 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef454a96-712b-4ece-b4d8-b116964867a5-workload-socket\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd\" (UID: \"ef454a96-712b-4ece-b4d8-b116964867a5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" Oct 30 13:05:34.609537 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:34.609504 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef454a96-712b-4ece-b4d8-b116964867a5-credential-socket\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd\" (UID: \"ef454a96-712b-4ece-b4d8-b116964867a5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" Oct 30 13:05:34.609744 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:34.609552 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef454a96-712b-4ece-b4d8-b116964867a5-istio-token\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd\" (UID: \"ef454a96-712b-4ece-b4d8-b116964867a5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" Oct 30 13:05:34.609744 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:34.609586 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef454a96-712b-4ece-b4d8-b116964867a5-istio-envoy\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd\" (UID: \"ef454a96-712b-4ece-b4d8-b116964867a5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" Oct 30 13:05:34.609744 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:34.609623 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef454a96-712b-4ece-b4d8-b116964867a5-istiod-ca-cert\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd\" (UID: \"ef454a96-712b-4ece-b4d8-b116964867a5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" Oct 30 13:05:34.609744 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:34.609689 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef454a96-712b-4ece-b4d8-b116964867a5-istio-data\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd\" (UID: \"ef454a96-712b-4ece-b4d8-b116964867a5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" Oct 30 13:05:34.609965 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:34.609750 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef454a96-712b-4ece-b4d8-b116964867a5-workload-certs\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd\" (UID: \"ef454a96-712b-4ece-b4d8-b116964867a5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" Oct 30 13:05:34.609965 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:34.609776 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef454a96-712b-4ece-b4d8-b116964867a5-istio-podinfo\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd\" (UID: \"ef454a96-712b-4ece-b4d8-b116964867a5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" Oct 30 13:05:34.609965 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:34.609850 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wm6z\" (UniqueName: \"kubernetes.io/projected/ef454a96-712b-4ece-b4d8-b116964867a5-kube-api-access-2wm6z\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd\" (UID: \"ef454a96-712b-4ece-b4d8-b116964867a5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" Oct 30 13:05:34.609965 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:34.609914 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef454a96-712b-4ece-b4d8-b116964867a5-workload-socket\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd\" (UID: \"ef454a96-712b-4ece-b4d8-b116964867a5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" Oct 30 13:05:34.610188 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:34.609978 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef454a96-712b-4ece-b4d8-b116964867a5-credential-socket\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd\" (UID: \"ef454a96-712b-4ece-b4d8-b116964867a5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" Oct 30 13:05:34.610188 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:34.610092 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef454a96-712b-4ece-b4d8-b116964867a5-istio-data\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd\" (UID: \"ef454a96-712b-4ece-b4d8-b116964867a5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" Oct 30 13:05:34.610188 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:34.610114 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef454a96-712b-4ece-b4d8-b116964867a5-workload-certs\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd\" (UID: \"ef454a96-712b-4ece-b4d8-b116964867a5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" Oct 30 13:05:34.610349 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:34.610306 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef454a96-712b-4ece-b4d8-b116964867a5-workload-socket\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd\" (UID: \"ef454a96-712b-4ece-b4d8-b116964867a5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" Oct 30 13:05:34.610437 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:34.610411 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef454a96-712b-4ece-b4d8-b116964867a5-istiod-ca-cert\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd\" (UID: \"ef454a96-712b-4ece-b4d8-b116964867a5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" Oct 30 13:05:34.611881 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:34.611857 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef454a96-712b-4ece-b4d8-b116964867a5-istio-envoy\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd\" (UID: \"ef454a96-712b-4ece-b4d8-b116964867a5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" Oct 30 13:05:34.612402 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:34.612380 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef454a96-712b-4ece-b4d8-b116964867a5-istio-podinfo\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd\" (UID: \"ef454a96-712b-4ece-b4d8-b116964867a5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" Oct 30 13:05:34.636725 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:34.636701 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef454a96-712b-4ece-b4d8-b116964867a5-istio-token\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd\" (UID: \"ef454a96-712b-4ece-b4d8-b116964867a5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" Oct 30 13:05:34.636968 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:34.636946 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wm6z\" (UniqueName: \"kubernetes.io/projected/ef454a96-712b-4ece-b4d8-b116964867a5-kube-api-access-2wm6z\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd\" (UID: \"ef454a96-712b-4ece-b4d8-b116964867a5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" Oct 30 13:05:34.722916 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:34.722877 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:34.727092 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:34.727049 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" Oct 30 13:05:35.069486 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:35.069405 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd"] Oct 30 13:05:35.075808 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:05:35.075773 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef454a96_712b_4ece_b4d8_b116964867a5.slice/crio-e06aa1d1fc90d682469277a6deede54cd10406feb35cbb1870643a20c52ea5d5 WatchSource:0}: Error finding container e06aa1d1fc90d682469277a6deede54cd10406feb35cbb1870643a20c52ea5d5: Status 404 returned error can't find the container with id e06aa1d1fc90d682469277a6deede54cd10406feb35cbb1870643a20c52ea5d5 Oct 30 13:05:35.078504 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:35.078351 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:05:35.078504 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:35.078445 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:05:35.078504 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:35.078488 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:05:35.299495 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:35.299459 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2"] Oct 30 13:05:35.303370 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:35.303348 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" Oct 30 13:05:35.339744 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:35.339676 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2"] Oct 30 13:05:35.424730 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:35.424703 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5813c167-a1c1-49ca-b93c-50fb1ece0b64-istiod-ca-cert\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2\" (UID: \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" Oct 30 13:05:35.425147 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:35.424737 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8v6c\" (UniqueName: \"kubernetes.io/projected/5813c167-a1c1-49ca-b93c-50fb1ece0b64-kube-api-access-d8v6c\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2\" (UID: \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" Oct 30 13:05:35.425147 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:35.424772 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5813c167-a1c1-49ca-b93c-50fb1ece0b64-workload-socket\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2\" (UID: \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" Oct 30 13:05:35.425147 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:35.424844 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5813c167-a1c1-49ca-b93c-50fb1ece0b64-workload-certs\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2\" (UID: \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" Oct 30 13:05:35.425147 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:35.424865 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5813c167-a1c1-49ca-b93c-50fb1ece0b64-istio-data\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2\" (UID: \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" Oct 30 13:05:35.425147 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:35.425024 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5813c167-a1c1-49ca-b93c-50fb1ece0b64-credential-socket\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2\" (UID: \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" Oct 30 13:05:35.425147 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:35.425054 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5813c167-a1c1-49ca-b93c-50fb1ece0b64-istio-token\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2\" (UID: \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" Oct 30 13:05:35.425147 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:35.425090 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5813c167-a1c1-49ca-b93c-50fb1ece0b64-istio-envoy\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2\" (UID: \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" Oct 30 13:05:35.425495 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:35.425201 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5813c167-a1c1-49ca-b93c-50fb1ece0b64-istio-podinfo\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2\" (UID: \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" Oct 30 13:05:35.525943 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:35.525911 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5813c167-a1c1-49ca-b93c-50fb1ece0b64-istio-token\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2\" (UID: \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" Oct 30 13:05:35.526304 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:35.525961 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5813c167-a1c1-49ca-b93c-50fb1ece0b64-istio-envoy\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2\" (UID: \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" Oct 30 13:05:35.526304 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:35.526013 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5813c167-a1c1-49ca-b93c-50fb1ece0b64-istio-podinfo\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2\" (UID: \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" Oct 30 13:05:35.526304 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:35.526088 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5813c167-a1c1-49ca-b93c-50fb1ece0b64-istiod-ca-cert\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2\" (UID: \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" Oct 30 13:05:35.526304 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:35.526117 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8v6c\" (UniqueName: \"kubernetes.io/projected/5813c167-a1c1-49ca-b93c-50fb1ece0b64-kube-api-access-d8v6c\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2\" (UID: \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" Oct 30 13:05:35.526304 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:35.526156 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5813c167-a1c1-49ca-b93c-50fb1ece0b64-workload-socket\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2\" (UID: \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" Oct 30 13:05:35.526304 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:35.526186 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5813c167-a1c1-49ca-b93c-50fb1ece0b64-workload-certs\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2\" (UID: \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" Oct 30 13:05:35.526304 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:35.526241 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5813c167-a1c1-49ca-b93c-50fb1ece0b64-istio-data\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2\" (UID: \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" Oct 30 13:05:35.526909 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:35.526760 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5813c167-a1c1-49ca-b93c-50fb1ece0b64-workload-certs\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2\" (UID: \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" Oct 30 13:05:35.526968 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:35.526901 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5813c167-a1c1-49ca-b93c-50fb1ece0b64-credential-socket\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2\" (UID: \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" Oct 30 13:05:35.527084 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:35.527064 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5813c167-a1c1-49ca-b93c-50fb1ece0b64-istiod-ca-cert\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2\" (UID: \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" Oct 30 13:05:35.527269 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:35.527251 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5813c167-a1c1-49ca-b93c-50fb1ece0b64-credential-socket\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2\" (UID: \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" Oct 30 13:05:35.528029 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:35.528002 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5813c167-a1c1-49ca-b93c-50fb1ece0b64-istio-data\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2\" (UID: \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" Oct 30 13:05:35.528319 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:35.528293 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5813c167-a1c1-49ca-b93c-50fb1ece0b64-workload-socket\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2\" (UID: \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" Oct 30 13:05:35.529583 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:35.529555 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5813c167-a1c1-49ca-b93c-50fb1ece0b64-istio-podinfo\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2\" (UID: \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" Oct 30 13:05:35.529583 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:35.529574 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5813c167-a1c1-49ca-b93c-50fb1ece0b64-istio-envoy\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2\" (UID: \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" Oct 30 13:05:35.551099 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:35.551071 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5813c167-a1c1-49ca-b93c-50fb1ece0b64-istio-token\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2\" (UID: \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" Oct 30 13:05:35.551749 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:35.551725 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8v6c\" (UniqueName: \"kubernetes.io/projected/5813c167-a1c1-49ca-b93c-50fb1ece0b64-kube-api-access-d8v6c\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2\" (UID: \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" Oct 30 13:05:35.620632 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:35.620528 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" Oct 30 13:05:35.708693 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:35.708652 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" event={"ID":"ef454a96-712b-4ece-b4d8-b116964867a5","Type":"ContainerStarted","Data":"eb6bae3ed9ff47988290782ac4816999a680baf832d88b2a74e4563e5c9ac15c"} Oct 30 13:05:35.708994 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:35.708967 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" event={"ID":"ef454a96-712b-4ece-b4d8-b116964867a5","Type":"ContainerStarted","Data":"e06aa1d1fc90d682469277a6deede54cd10406feb35cbb1870643a20c52ea5d5"} Oct 30 13:05:35.732536 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:35.731389 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" Oct 30 13:05:35.998679 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:35.997471 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" podStartSLOduration=1.9974506490000001 podStartE2EDuration="1.997450649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:35.801339595 +0000 UTC m=+792.270755360" watchObservedRunningTime="2026-10-30 13:05:35.997450649 +0000 UTC m=+792.466866395" Oct 30 13:05:35.998679 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:35.998137 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2"] Oct 30 13:05:36.007724 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:05:36.007691 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5813c167_a1c1_49ca_b93c_50fb1ece0b64.slice/crio-48b81290c1a3a8b9d8acb4db69cc6388b980e2229519d4eb265759f9644be7f0 WatchSource:0}: Error finding container 48b81290c1a3a8b9d8acb4db69cc6388b980e2229519d4eb265759f9644be7f0: Status 404 returned error can't find the container with id 48b81290c1a3a8b9d8acb4db69cc6388b980e2229519d4eb265759f9644be7f0 Oct 30 13:05:36.011417 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:36.011376 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:05:36.011526 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:36.011468 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:05:36.011526 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:36.011512 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:05:36.721245 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:36.721206 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" event={"ID":"5813c167-a1c1-49ca-b93c-50fb1ece0b64","Type":"ContainerStarted","Data":"d5cb6b7d0aca9cface57705be1dfc1fa8aeb2daa1aab7c6c6eded7b2f25e3080"} Oct 30 13:05:36.721665 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:36.721295 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" event={"ID":"5813c167-a1c1-49ca-b93c-50fb1ece0b64","Type":"ContainerStarted","Data":"48b81290c1a3a8b9d8acb4db69cc6388b980e2229519d4eb265759f9644be7f0"} Oct 30 13:05:36.729345 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:36.729315 5922 patch_prober.go:29] interesting pod/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 13:05:36.729606 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:36.729369 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" podUID="ef454a96-712b-4ece-b4d8-b116964867a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 13:05:36.804633 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:36.801539 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" podStartSLOduration=1.8015198 podStartE2EDuration="1.8015198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:36.774638969 +0000 UTC m=+793.244054716" watchObservedRunningTime="2026-10-30 13:05:36.8015198 +0000 UTC m=+793.270935546" Oct 30 13:05:37.176718 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:37.171136 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:37.620895 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:37.620867 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" Oct 30 13:05:37.622208 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:37.622185 5922 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 13:05:37.622285 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:37.622234 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" podUID="5813c167-a1c1-49ca-b93c-50fb1ece0b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 13:05:37.727485 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:37.727443 5922 patch_prober.go:29] interesting pod/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 13:05:37.727966 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:37.727503 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" podUID="ef454a96-712b-4ece-b4d8-b116964867a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 13:05:38.621789 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:38.621749 5922 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 13:05:38.621966 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:38.621813 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" podUID="5813c167-a1c1-49ca-b93c-50fb1ece0b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 13:05:38.727563 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:38.727527 5922 patch_prober.go:29] interesting pod/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 13:05:38.728056 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:38.727584 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" podUID="ef454a96-712b-4ece-b4d8-b116964867a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 13:05:39.235446 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:39.235410 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:39.354865 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:39.354832 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs"] Oct 30 13:05:39.358392 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:39.358370 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs" Oct 30 13:05:39.449065 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:39.447899 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:39.450091 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:39.450065 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs"] Oct 30 13:05:39.531435 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:39.531345 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3577d48e-eeaf-4608-ac46-99d2667ea4c0-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bmhqs\" (UID: \"3577d48e-eeaf-4608-ac46-99d2667ea4c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs" Oct 30 13:05:39.531583 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:39.531461 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3577d48e-eeaf-4608-ac46-99d2667ea4c0-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bmhqs\" (UID: \"3577d48e-eeaf-4608-ac46-99d2667ea4c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs" Oct 30 13:05:39.531583 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:39.531491 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3577d48e-eeaf-4608-ac46-99d2667ea4c0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bmhqs\" (UID: \"3577d48e-eeaf-4608-ac46-99d2667ea4c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs" Oct 30 13:05:39.531695 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:39.531606 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzb7p\" (UniqueName: \"kubernetes.io/projected/3577d48e-eeaf-4608-ac46-99d2667ea4c0-kube-api-access-zzb7p\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bmhqs\" (UID: \"3577d48e-eeaf-4608-ac46-99d2667ea4c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs" Oct 30 13:05:39.531728 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:39.531718 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3577d48e-eeaf-4608-ac46-99d2667ea4c0-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bmhqs\" (UID: \"3577d48e-eeaf-4608-ac46-99d2667ea4c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs" Oct 30 13:05:39.531772 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:39.531752 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3577d48e-eeaf-4608-ac46-99d2667ea4c0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bmhqs\" (UID: \"3577d48e-eeaf-4608-ac46-99d2667ea4c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs" Oct 30 13:05:39.531810 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:39.531801 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3577d48e-eeaf-4608-ac46-99d2667ea4c0-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bmhqs\" (UID: \"3577d48e-eeaf-4608-ac46-99d2667ea4c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs" Oct 30 13:05:39.531851 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:39.531838 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3577d48e-eeaf-4608-ac46-99d2667ea4c0-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bmhqs\" (UID: \"3577d48e-eeaf-4608-ac46-99d2667ea4c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs" Oct 30 13:05:39.531886 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:39.531864 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3577d48e-eeaf-4608-ac46-99d2667ea4c0-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bmhqs\" (UID: \"3577d48e-eeaf-4608-ac46-99d2667ea4c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs" Oct 30 13:05:39.621166 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:39.621131 5922 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 13:05:39.621337 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:39.621181 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" podUID="5813c167-a1c1-49ca-b93c-50fb1ece0b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 13:05:39.633184 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:39.633156 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3577d48e-eeaf-4608-ac46-99d2667ea4c0-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bmhqs\" (UID: \"3577d48e-eeaf-4608-ac46-99d2667ea4c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs" Oct 30 13:05:39.633273 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:39.633189 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3577d48e-eeaf-4608-ac46-99d2667ea4c0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bmhqs\" (UID: \"3577d48e-eeaf-4608-ac46-99d2667ea4c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs" Oct 30 13:05:39.633273 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:39.633235 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzb7p\" (UniqueName: \"kubernetes.io/projected/3577d48e-eeaf-4608-ac46-99d2667ea4c0-kube-api-access-zzb7p\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bmhqs\" (UID: \"3577d48e-eeaf-4608-ac46-99d2667ea4c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs" Oct 30 13:05:39.633334 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:39.633275 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3577d48e-eeaf-4608-ac46-99d2667ea4c0-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bmhqs\" (UID: \"3577d48e-eeaf-4608-ac46-99d2667ea4c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs" Oct 30 13:05:39.633334 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:39.633296 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3577d48e-eeaf-4608-ac46-99d2667ea4c0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bmhqs\" (UID: \"3577d48e-eeaf-4608-ac46-99d2667ea4c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs" Oct 30 13:05:39.633396 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:39.633332 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3577d48e-eeaf-4608-ac46-99d2667ea4c0-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bmhqs\" (UID: \"3577d48e-eeaf-4608-ac46-99d2667ea4c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs" Oct 30 13:05:39.633396 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:39.633357 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3577d48e-eeaf-4608-ac46-99d2667ea4c0-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bmhqs\" (UID: \"3577d48e-eeaf-4608-ac46-99d2667ea4c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs" Oct 30 13:05:39.633396 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:39.633372 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3577d48e-eeaf-4608-ac46-99d2667ea4c0-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bmhqs\" (UID: \"3577d48e-eeaf-4608-ac46-99d2667ea4c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs" Oct 30 13:05:39.633491 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:39.633419 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3577d48e-eeaf-4608-ac46-99d2667ea4c0-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bmhqs\" (UID: \"3577d48e-eeaf-4608-ac46-99d2667ea4c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs" Oct 30 13:05:39.633918 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:39.633900 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3577d48e-eeaf-4608-ac46-99d2667ea4c0-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bmhqs\" (UID: \"3577d48e-eeaf-4608-ac46-99d2667ea4c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs" Oct 30 13:05:39.634160 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:39.634140 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3577d48e-eeaf-4608-ac46-99d2667ea4c0-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bmhqs\" (UID: \"3577d48e-eeaf-4608-ac46-99d2667ea4c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs" Oct 30 13:05:39.634319 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:39.634293 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3577d48e-eeaf-4608-ac46-99d2667ea4c0-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bmhqs\" (UID: \"3577d48e-eeaf-4608-ac46-99d2667ea4c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs" Oct 30 13:05:39.634576 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:39.634557 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3577d48e-eeaf-4608-ac46-99d2667ea4c0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bmhqs\" (UID: \"3577d48e-eeaf-4608-ac46-99d2667ea4c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs" Oct 30 13:05:39.634777 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:39.634754 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3577d48e-eeaf-4608-ac46-99d2667ea4c0-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bmhqs\" (UID: \"3577d48e-eeaf-4608-ac46-99d2667ea4c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs" Oct 30 13:05:39.636377 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:39.636355 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3577d48e-eeaf-4608-ac46-99d2667ea4c0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bmhqs\" (UID: \"3577d48e-eeaf-4608-ac46-99d2667ea4c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs" Oct 30 13:05:39.636557 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:39.636539 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3577d48e-eeaf-4608-ac46-99d2667ea4c0-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bmhqs\" (UID: \"3577d48e-eeaf-4608-ac46-99d2667ea4c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs" Oct 30 13:05:39.658513 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:39.658489 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3577d48e-eeaf-4608-ac46-99d2667ea4c0-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bmhqs\" (UID: \"3577d48e-eeaf-4608-ac46-99d2667ea4c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs" Oct 30 13:05:39.658626 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:39.658560 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzb7p\" (UniqueName: \"kubernetes.io/projected/3577d48e-eeaf-4608-ac46-99d2667ea4c0-kube-api-access-zzb7p\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bmhqs\" (UID: \"3577d48e-eeaf-4608-ac46-99d2667ea4c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs" Oct 30 13:05:39.727791 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:39.727754 5922 patch_prober.go:29] interesting pod/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 13:05:39.728180 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:39.727808 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" podUID="ef454a96-712b-4ece-b4d8-b116964867a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 13:05:39.760270 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:39.760238 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs" Oct 30 13:05:40.128954 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:40.128868 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs"] Oct 30 13:05:40.134416 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:05:40.134355 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3577d48e_eeaf_4608_ac46_99d2667ea4c0.slice/crio-c0ac12994ec8ced9e755f7747df4e6434067e2b27c7962b5d6bcf3cf7936be00 WatchSource:0}: Error finding container c0ac12994ec8ced9e755f7747df4e6434067e2b27c7962b5d6bcf3cf7936be00: Status 404 returned error can't find the container with id c0ac12994ec8ced9e755f7747df4e6434067e2b27c7962b5d6bcf3cf7936be00 Oct 30 13:05:40.137873 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:40.137438 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:05:40.137873 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:40.137504 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:05:40.137873 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:40.137536 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:05:40.621017 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:40.620979 5922 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 13:05:40.621310 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:40.621274 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" podUID="5813c167-a1c1-49ca-b93c-50fb1ece0b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 13:05:40.728033 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:40.727984 5922 patch_prober.go:29] interesting pod/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 13:05:40.728459 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:40.728052 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" podUID="ef454a96-712b-4ece-b4d8-b116964867a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 13:05:40.747803 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:40.747764 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs" event={"ID":"3577d48e-eeaf-4608-ac46-99d2667ea4c0","Type":"ContainerStarted","Data":"b3d638a96e5dd68d8cc9546394eb8fbe755cf70bff0e3206f8f07c2334cd3534"} Oct 30 13:05:40.747948 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:40.747854 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs" event={"ID":"3577d48e-eeaf-4608-ac46-99d2667ea4c0","Type":"ContainerStarted","Data":"c0ac12994ec8ced9e755f7747df4e6434067e2b27c7962b5d6bcf3cf7936be00"} Oct 30 13:05:40.760487 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:40.760464 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs" Oct 30 13:05:40.814643 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:40.812968 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs" podStartSLOduration=1.812946502 podStartE2EDuration="1.812946502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:40.797399467 +0000 UTC m=+797.266815238" watchObservedRunningTime="2026-10-30 13:05:40.812946502 +0000 UTC m=+797.282362246" Oct 30 13:05:41.621309 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:41.621258 5922 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 13:05:41.621494 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:41.621333 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" podUID="5813c167-a1c1-49ca-b93c-50fb1ece0b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 13:05:41.728731 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:41.728691 5922 patch_prober.go:29] interesting pod/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 13:05:41.729182 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:41.728758 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" podUID="ef454a96-712b-4ece-b4d8-b116964867a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 13:05:41.763366 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:41.763335 5922 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bmhqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 13:05:41.763657 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:41.763629 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs" podUID="3577d48e-eeaf-4608-ac46-99d2667ea4c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 13:05:41.821859 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:41.821822 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:42.621767 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:42.621735 5922 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 13:05:42.621948 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:42.621786 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" podUID="5813c167-a1c1-49ca-b93c-50fb1ece0b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 13:05:42.734203 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:42.734170 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" Oct 30 13:05:42.734599 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:42.734513 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" Oct 30 13:05:42.735246 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:42.735222 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" Oct 30 13:05:42.760520 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:42.760488 5922 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bmhqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 13:05:42.760654 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:42.760533 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs" podUID="3577d48e-eeaf-4608-ac46-99d2667ea4c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 13:05:43.625300 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:43.625265 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" Oct 30 13:05:43.625580 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:43.625554 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" Oct 30 13:05:43.626342 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:43.626324 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" Oct 30 13:05:43.761004 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:43.760961 5922 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bmhqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 13:05:43.761454 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:43.761036 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs" podUID="3577d48e-eeaf-4608-ac46-99d2667ea4c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 13:05:43.908031 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:43.907941 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:44.394341 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:44.394303 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:44.761321 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:44.761290 5922 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bmhqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 13:05:44.761755 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:44.761348 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs" podUID="3577d48e-eeaf-4608-ac46-99d2667ea4c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 13:05:45.761031 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:45.760992 5922 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bmhqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 13:05:45.761206 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:45.761061 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs" podUID="3577d48e-eeaf-4608-ac46-99d2667ea4c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 13:05:46.088246 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:46.088157 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:46.760488 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:46.760459 5922 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bmhqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 13:05:46.760686 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:46.760513 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs" podUID="3577d48e-eeaf-4608-ac46-99d2667ea4c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 13:05:47.648383 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:47.648349 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:05:47.761626 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:47.761581 5922 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bmhqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 13:05:47.761817 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:47.761657 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs" podUID="3577d48e-eeaf-4608-ac46-99d2667ea4c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 13:05:47.905890 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:47.905805 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:48.959884 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:48.959857 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs" Oct 30 13:05:48.960338 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:48.960317 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs" Oct 30 13:05:48.961076 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:48.961017 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bmhqs" Oct 30 13:05:50.207239 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:50.207206 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:50.784585 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:50.784551 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:52.115591 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:52.115553 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:53.552412 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:05:53.552378 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:05:53.986217 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:53.986184 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:54.968465 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:54.968426 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2"] Oct 30 13:05:54.972765 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:54.972746 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" Oct 30 13:05:54.992481 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:54.992457 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2"] Oct 30 13:05:55.001982 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:55.001955 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-175831\"/\"distinct-2-w5-175831-kserve-self-signed-certs\"" Oct 30 13:05:55.002090 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:55.002037 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-175831\"/\"default-dockercfg-png5h\"" Oct 30 13:05:55.002413 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:55.002391 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-175831\"/\"kube-root-ca.crt\"" Oct 30 13:05:55.002530 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:55.002420 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-175831\"/\"openshift-service-ca.crt\"" Oct 30 13:05:55.033050 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:55.033024 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:55.054902 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:55.054881 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47517373-759d-4eeb-ab5e-6150b7e82bb7-dshm\") pod \"distinct-2-w5-175831-kserve-64db498f4b-d76n2\" (UID: \"47517373-759d-4eeb-ab5e-6150b7e82bb7\") " pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" Oct 30 13:05:55.055005 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:55.054927 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47517373-759d-4eeb-ab5e-6150b7e82bb7-home\") pod \"distinct-2-w5-175831-kserve-64db498f4b-d76n2\" (UID: \"47517373-759d-4eeb-ab5e-6150b7e82bb7\") " pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" Oct 30 13:05:55.055005 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:55.054947 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47517373-759d-4eeb-ab5e-6150b7e82bb7-tmp-dir\") pod \"distinct-2-w5-175831-kserve-64db498f4b-d76n2\" (UID: \"47517373-759d-4eeb-ab5e-6150b7e82bb7\") " pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" Oct 30 13:05:55.055005 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:55.054993 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjkbr\" (UniqueName: \"kubernetes.io/projected/47517373-759d-4eeb-ab5e-6150b7e82bb7-kube-api-access-vjkbr\") pod \"distinct-2-w5-175831-kserve-64db498f4b-d76n2\" (UID: \"47517373-759d-4eeb-ab5e-6150b7e82bb7\") " pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" Oct 30 13:05:55.055155 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:55.055139 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47517373-759d-4eeb-ab5e-6150b7e82bb7-model-cache\") pod \"distinct-2-w5-175831-kserve-64db498f4b-d76n2\" (UID: \"47517373-759d-4eeb-ab5e-6150b7e82bb7\") " pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" Oct 30 13:05:55.055217 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:55.055206 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47517373-759d-4eeb-ab5e-6150b7e82bb7-tls-certs\") pod \"distinct-2-w5-175831-kserve-64db498f4b-d76n2\" (UID: \"47517373-759d-4eeb-ab5e-6150b7e82bb7\") " pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" Oct 30 13:05:55.156352 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:55.156330 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47517373-759d-4eeb-ab5e-6150b7e82bb7-home\") pod \"distinct-2-w5-175831-kserve-64db498f4b-d76n2\" (UID: \"47517373-759d-4eeb-ab5e-6150b7e82bb7\") " pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" Oct 30 13:05:55.156481 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:55.156358 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47517373-759d-4eeb-ab5e-6150b7e82bb7-tmp-dir\") pod \"distinct-2-w5-175831-kserve-64db498f4b-d76n2\" (UID: \"47517373-759d-4eeb-ab5e-6150b7e82bb7\") " pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" Oct 30 13:05:55.156481 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:55.156375 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjkbr\" (UniqueName: \"kubernetes.io/projected/47517373-759d-4eeb-ab5e-6150b7e82bb7-kube-api-access-vjkbr\") pod \"distinct-2-w5-175831-kserve-64db498f4b-d76n2\" (UID: \"47517373-759d-4eeb-ab5e-6150b7e82bb7\") " pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" Oct 30 13:05:55.156481 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:55.156463 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47517373-759d-4eeb-ab5e-6150b7e82bb7-model-cache\") pod \"distinct-2-w5-175831-kserve-64db498f4b-d76n2\" (UID: \"47517373-759d-4eeb-ab5e-6150b7e82bb7\") " pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" Oct 30 13:05:55.156694 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:55.156532 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47517373-759d-4eeb-ab5e-6150b7e82bb7-tls-certs\") pod \"distinct-2-w5-175831-kserve-64db498f4b-d76n2\" (UID: \"47517373-759d-4eeb-ab5e-6150b7e82bb7\") " pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" Oct 30 13:05:55.156694 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:55.156577 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47517373-759d-4eeb-ab5e-6150b7e82bb7-dshm\") pod \"distinct-2-w5-175831-kserve-64db498f4b-d76n2\" (UID: \"47517373-759d-4eeb-ab5e-6150b7e82bb7\") " pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" Oct 30 13:05:55.156807 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:55.156760 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47517373-759d-4eeb-ab5e-6150b7e82bb7-home\") pod \"distinct-2-w5-175831-kserve-64db498f4b-d76n2\" (UID: \"47517373-759d-4eeb-ab5e-6150b7e82bb7\") " pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" Oct 30 13:05:55.156867 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:55.156818 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47517373-759d-4eeb-ab5e-6150b7e82bb7-tmp-dir\") pod \"distinct-2-w5-175831-kserve-64db498f4b-d76n2\" (UID: \"47517373-759d-4eeb-ab5e-6150b7e82bb7\") " pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" Oct 30 13:05:55.156867 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:55.156853 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47517373-759d-4eeb-ab5e-6150b7e82bb7-model-cache\") pod \"distinct-2-w5-175831-kserve-64db498f4b-d76n2\" (UID: \"47517373-759d-4eeb-ab5e-6150b7e82bb7\") " pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" Oct 30 13:05:55.158871 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:55.158849 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47517373-759d-4eeb-ab5e-6150b7e82bb7-dshm\") pod \"distinct-2-w5-175831-kserve-64db498f4b-d76n2\" (UID: \"47517373-759d-4eeb-ab5e-6150b7e82bb7\") " pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" Oct 30 13:05:55.159101 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:55.159085 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47517373-759d-4eeb-ab5e-6150b7e82bb7-tls-certs\") pod \"distinct-2-w5-175831-kserve-64db498f4b-d76n2\" (UID: \"47517373-759d-4eeb-ab5e-6150b7e82bb7\") " pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" Oct 30 13:05:55.182601 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:55.182577 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjkbr\" (UniqueName: \"kubernetes.io/projected/47517373-759d-4eeb-ab5e-6150b7e82bb7-kube-api-access-vjkbr\") pod \"distinct-2-w5-175831-kserve-64db498f4b-d76n2\" (UID: \"47517373-759d-4eeb-ab5e-6150b7e82bb7\") " pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" Oct 30 13:05:55.293819 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:55.293736 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" Oct 30 13:05:55.600134 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:55.600109 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2"] Oct 30 13:05:55.602885 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:05:55.602852 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47517373_759d_4eeb_ab5e_6150b7e82bb7.slice/crio-bd3b756465a6d07daa10c1d4ca5f346bf13a93240ddc93ac7c0b3cc38fd6d250 WatchSource:0}: Error finding container bd3b756465a6d07daa10c1d4ca5f346bf13a93240ddc93ac7c0b3cc38fd6d250: Status 404 returned error can't find the container with id bd3b756465a6d07daa10c1d4ca5f346bf13a93240ddc93ac7c0b3cc38fd6d250 Oct 30 13:05:55.878198 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:55.878117 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" event={"ID":"47517373-759d-4eeb-ab5e-6150b7e82bb7","Type":"ContainerStarted","Data":"2150b5f29f2bbc8413a20cba99ce41eeef399d503c0c3629db635a9cbb6edc9c"} Oct 30 13:05:55.878198 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:55.878165 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" event={"ID":"47517373-759d-4eeb-ab5e-6150b7e82bb7","Type":"ContainerStarted","Data":"bd3b756465a6d07daa10c1d4ca5f346bf13a93240ddc93ac7c0b3cc38fd6d250"} Oct 30 13:05:55.925794 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:55.925739 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" podStartSLOduration=1.925724607 podStartE2EDuration="1.925724607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:55.923196678 +0000 UTC m=+812.392612423" watchObservedRunningTime="2026-10-30 13:05:55.925724607 +0000 UTC m=+812.395140352" Oct 30 13:05:55.995073 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:55.995044 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:56.294709 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:56.294681 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" Oct 30 13:05:57.884871 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:57.884825 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:59.790984 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:05:59.790951 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:01.509958 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:01.509924 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:03.207374 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:03.207337 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:05.297754 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:05.297661 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" Oct 30 13:06:05.336372 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:05.336342 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" Oct 30 13:06:05.400305 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:05.400268 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:05.959132 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:05.959104 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" Oct 30 13:06:07.181389 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:07.181357 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:09.066735 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:09.066700 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:10.784205 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:10.784170 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:12.643429 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:12.643396 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:12.967333 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:12.967300 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:06:14.509137 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:14.509100 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:15.712965 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:15.712922 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:16.376298 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:16.376255 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:16.462579 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:16.462544 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7"] Oct 30 13:06:16.465830 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:16.465807 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" Oct 30 13:06:16.497662 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:16.497634 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-484607\"/\"distinct-2-w6-484607-kserve-self-signed-certs\"" Oct 30 13:06:16.500658 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:16.500630 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7"] Oct 30 13:06:16.519331 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:16.519305 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad5bdad-cd6f-41f1-a124-540c3524ead7-home\") pod \"distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7\" (UID: \"dad5bdad-cd6f-41f1-a124-540c3524ead7\") " pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" Oct 30 13:06:16.519436 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:16.519346 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad5bdad-cd6f-41f1-a124-540c3524ead7-tls-certs\") pod \"distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7\" (UID: \"dad5bdad-cd6f-41f1-a124-540c3524ead7\") " pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" Oct 30 13:06:16.519436 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:16.519419 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad5bdad-cd6f-41f1-a124-540c3524ead7-model-cache\") pod \"distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7\" (UID: \"dad5bdad-cd6f-41f1-a124-540c3524ead7\") " pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" Oct 30 13:06:16.519512 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:16.519464 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad5bdad-cd6f-41f1-a124-540c3524ead7-tmp-dir\") pod \"distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7\" (UID: \"dad5bdad-cd6f-41f1-a124-540c3524ead7\") " pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" Oct 30 13:06:16.519593 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:16.519574 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvf97\" (UniqueName: \"kubernetes.io/projected/dad5bdad-cd6f-41f1-a124-540c3524ead7-kube-api-access-qvf97\") pod \"distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7\" (UID: \"dad5bdad-cd6f-41f1-a124-540c3524ead7\") " pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" Oct 30 13:06:16.519789 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:16.519766 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad5bdad-cd6f-41f1-a124-540c3524ead7-dshm\") pod \"distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7\" (UID: \"dad5bdad-cd6f-41f1-a124-540c3524ead7\") " pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" Oct 30 13:06:16.522896 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:16.522867 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:16.620646 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:16.620606 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvf97\" (UniqueName: \"kubernetes.io/projected/dad5bdad-cd6f-41f1-a124-540c3524ead7-kube-api-access-qvf97\") pod \"distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7\" (UID: \"dad5bdad-cd6f-41f1-a124-540c3524ead7\") " pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" Oct 30 13:06:16.620790 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:16.620743 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad5bdad-cd6f-41f1-a124-540c3524ead7-dshm\") pod \"distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7\" (UID: \"dad5bdad-cd6f-41f1-a124-540c3524ead7\") " pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" Oct 30 13:06:16.620879 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:16.620860 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad5bdad-cd6f-41f1-a124-540c3524ead7-home\") pod \"distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7\" (UID: \"dad5bdad-cd6f-41f1-a124-540c3524ead7\") " pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" Oct 30 13:06:16.620946 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:16.620893 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad5bdad-cd6f-41f1-a124-540c3524ead7-tls-certs\") pod \"distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7\" (UID: \"dad5bdad-cd6f-41f1-a124-540c3524ead7\") " pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" Oct 30 13:06:16.620946 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:16.620928 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad5bdad-cd6f-41f1-a124-540c3524ead7-model-cache\") pod \"distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7\" (UID: \"dad5bdad-cd6f-41f1-a124-540c3524ead7\") " pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" Oct 30 13:06:16.621056 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:16.620981 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad5bdad-cd6f-41f1-a124-540c3524ead7-tmp-dir\") pod \"distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7\" (UID: \"dad5bdad-cd6f-41f1-a124-540c3524ead7\") " pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" Oct 30 13:06:16.621244 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:16.621214 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad5bdad-cd6f-41f1-a124-540c3524ead7-home\") pod \"distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7\" (UID: \"dad5bdad-cd6f-41f1-a124-540c3524ead7\") " pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" Oct 30 13:06:16.621354 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:16.621260 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad5bdad-cd6f-41f1-a124-540c3524ead7-model-cache\") pod \"distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7\" (UID: \"dad5bdad-cd6f-41f1-a124-540c3524ead7\") " pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" Oct 30 13:06:16.621354 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:16.621335 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad5bdad-cd6f-41f1-a124-540c3524ead7-tmp-dir\") pod \"distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7\" (UID: \"dad5bdad-cd6f-41f1-a124-540c3524ead7\") " pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" Oct 30 13:06:16.623061 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:16.623041 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad5bdad-cd6f-41f1-a124-540c3524ead7-dshm\") pod \"distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7\" (UID: \"dad5bdad-cd6f-41f1-a124-540c3524ead7\") " pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" Oct 30 13:06:16.623324 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:16.623304 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad5bdad-cd6f-41f1-a124-540c3524ead7-tls-certs\") pod \"distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7\" (UID: \"dad5bdad-cd6f-41f1-a124-540c3524ead7\") " pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" Oct 30 13:06:16.645766 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:16.645702 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvf97\" (UniqueName: \"kubernetes.io/projected/dad5bdad-cd6f-41f1-a124-540c3524ead7-kube-api-access-qvf97\") pod \"distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7\" (UID: \"dad5bdad-cd6f-41f1-a124-540c3524ead7\") " pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" Oct 30 13:06:16.778269 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:16.778233 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" Oct 30 13:06:17.101426 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:17.101398 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7"] Oct 30 13:06:17.103507 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:06:17.103480 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddad5bdad_cd6f_41f1_a124_540c3524ead7.slice/crio-132b93fb8371cf88a6f56f12a8802392cc36659824f911aa504b8889d6062727 WatchSource:0}: Error finding container 132b93fb8371cf88a6f56f12a8802392cc36659824f911aa504b8889d6062727: Status 404 returned error can't find the container with id 132b93fb8371cf88a6f56f12a8802392cc36659824f911aa504b8889d6062727 Oct 30 13:06:17.190181 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:17.190150 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fea9f444-sxqcc"] Oct 30 13:06:17.193236 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:17.193215 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea9f444-sxqcc" Oct 30 13:06:17.219027 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:17.218999 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fea9f444-sxqcc"] Oct 30 13:06:17.220103 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:17.220070 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:06:17.226828 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:17.226803 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:06:17.226976 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:17.226806 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:06:17.227095 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:17.226830 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:06:17.229011 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:17.228990 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgzrs\" (UniqueName: \"kubernetes.io/projected/f0df24ea-4b3c-42b3-9283-cecc6b2858ed-kube-api-access-pgzrs\") pod \"maas-api-revoke-sub-fea9f444-sxqcc\" (UID: \"f0df24ea-4b3c-42b3-9283-cecc6b2858ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea9f444-sxqcc" Oct 30 13:06:17.229127 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:17.229023 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0df24ea-4b3c-42b3-9283-cecc6b2858ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fea9f444-sxqcc\" (UID: \"f0df24ea-4b3c-42b3-9283-cecc6b2858ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea9f444-sxqcc" Oct 30 13:06:17.229127 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:17.229042 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0df24ea-4b3c-42b3-9283-cecc6b2858ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fea9f444-sxqcc\" (UID: \"f0df24ea-4b3c-42b3-9283-cecc6b2858ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea9f444-sxqcc" Oct 30 13:06:17.330340 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:17.330308 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgzrs\" (UniqueName: \"kubernetes.io/projected/f0df24ea-4b3c-42b3-9283-cecc6b2858ed-kube-api-access-pgzrs\") pod \"maas-api-revoke-sub-fea9f444-sxqcc\" (UID: \"f0df24ea-4b3c-42b3-9283-cecc6b2858ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea9f444-sxqcc" Oct 30 13:06:17.330501 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:17.330360 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0df24ea-4b3c-42b3-9283-cecc6b2858ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fea9f444-sxqcc\" (UID: \"f0df24ea-4b3c-42b3-9283-cecc6b2858ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea9f444-sxqcc" Oct 30 13:06:17.330501 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:17.330465 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0df24ea-4b3c-42b3-9283-cecc6b2858ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fea9f444-sxqcc\" (UID: \"f0df24ea-4b3c-42b3-9283-cecc6b2858ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea9f444-sxqcc" Oct 30 13:06:17.331169 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:17.331147 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0df24ea-4b3c-42b3-9283-cecc6b2858ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fea9f444-sxqcc\" (UID: \"f0df24ea-4b3c-42b3-9283-cecc6b2858ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea9f444-sxqcc" Oct 30 13:06:17.332678 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:17.332658 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0df24ea-4b3c-42b3-9283-cecc6b2858ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fea9f444-sxqcc\" (UID: \"f0df24ea-4b3c-42b3-9283-cecc6b2858ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea9f444-sxqcc" Oct 30 13:06:17.355102 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:17.355051 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgzrs\" (UniqueName: \"kubernetes.io/projected/f0df24ea-4b3c-42b3-9283-cecc6b2858ed-kube-api-access-pgzrs\") pod \"maas-api-revoke-sub-fea9f444-sxqcc\" (UID: \"f0df24ea-4b3c-42b3-9283-cecc6b2858ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea9f444-sxqcc" Oct 30 13:06:17.506212 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:17.506176 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea9f444-sxqcc" Oct 30 13:06:17.806990 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:17.806738 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fea9f444-sxqcc"] Oct 30 13:06:17.810061 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:06:17.810016 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0df24ea_4b3c_42b3_9283_cecc6b2858ed.slice/crio-127957e71eb4c586f603321f56d7cca934bb1fc29d2d6ca74c4ae4ab815624fb WatchSource:0}: Error finding container 127957e71eb4c586f603321f56d7cca934bb1fc29d2d6ca74c4ae4ab815624fb: Status 404 returned error can't find the container with id 127957e71eb4c586f603321f56d7cca934bb1fc29d2d6ca74c4ae4ab815624fb Oct 30 13:06:18.028524 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:18.028495 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea9f444-sxqcc" event={"ID":"f0df24ea-4b3c-42b3-9283-cecc6b2858ed","Type":"ContainerStarted","Data":"127957e71eb4c586f603321f56d7cca934bb1fc29d2d6ca74c4ae4ab815624fb"} Oct 30 13:06:18.030142 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:18.030117 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" event={"ID":"dad5bdad-cd6f-41f1-a124-540c3524ead7","Type":"ContainerStarted","Data":"330dec08aa5847cd63b4903ab3adf5835b08240b106b2e489044666aaa1c27c8"} Oct 30 13:06:18.030245 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:18.030159 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" event={"ID":"dad5bdad-cd6f-41f1-a124-540c3524ead7","Type":"ContainerStarted","Data":"132b93fb8371cf88a6f56f12a8802392cc36659824f911aa504b8889d6062727"} Oct 30 13:06:18.083514 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:18.083465 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" podStartSLOduration=2.083448103 podStartE2EDuration="2.083448103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:18.078923971 +0000 UTC m=+834.548339718" watchObservedRunningTime="2026-10-30 13:06:18.083448103 +0000 UTC m=+834.552863852" Oct 30 13:06:18.227643 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:18.227598 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:18.778703 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:18.778679 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" Oct 30 13:06:19.037672 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:19.037642 5922 generic.go:356] "Generic (PLEG): container finished" podID="f0df24ea-4b3c-42b3-9283-cecc6b2858ed" containerID="a9e66db09439796f2e60293ac8baf2f0b8fe97ccc3eff288c7cfa2fd6dad9caf" exitCode=0 Oct 30 13:06:19.038017 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:19.037729 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea9f444-sxqcc" event={"ID":"f0df24ea-4b3c-42b3-9283-cecc6b2858ed","Type":"ContainerDied","Data":"a9e66db09439796f2e60293ac8baf2f0b8fe97ccc3eff288c7cfa2fd6dad9caf"} Oct 30 13:06:19.993638 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:19.993580 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:20.147708 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.147679 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb"] Oct 30 13:06:20.150837 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.150812 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" Oct 30 13:06:20.183783 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.183754 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb"] Oct 30 13:06:20.219495 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.219469 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea9f444-sxqcc" Oct 30 13:06:20.262970 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.262913 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0df24ea-4b3c-42b3-9283-cecc6b2858ed-kube-api-access-pgzrs\" (UniqueName: \"kubernetes.io/projected/f0df24ea-4b3c-42b3-9283-cecc6b2858ed-kube-api-access-pgzrs\") pod \"f0df24ea-4b3c-42b3-9283-cecc6b2858ed\" (UID: \"f0df24ea-4b3c-42b3-9283-cecc6b2858ed\") " Oct 30 13:06:20.263101 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.263088 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0df24ea-4b3c-42b3-9283-cecc6b2858ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0df24ea-4b3c-42b3-9283-cecc6b2858ed-maas-api-xks-ca\") pod \"f0df24ea-4b3c-42b3-9283-cecc6b2858ed\" (UID: \"f0df24ea-4b3c-42b3-9283-cecc6b2858ed\") " Oct 30 13:06:20.263184 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.263170 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0df24ea-4b3c-42b3-9283-cecc6b2858ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0df24ea-4b3c-42b3-9283-cecc6b2858ed-maas-api-openshift-ca\") pod \"f0df24ea-4b3c-42b3-9283-cecc6b2858ed\" (UID: \"f0df24ea-4b3c-42b3-9283-cecc6b2858ed\") " Oct 30 13:06:20.263459 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.263432 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-workload-socket\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb\" (UID: \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" Oct 30 13:06:20.263588 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.263491 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmrmn\" (UniqueName: \"kubernetes.io/projected/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-kube-api-access-jmrmn\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb\" (UID: \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" Oct 30 13:06:20.263662 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.263592 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0df24ea-4b3c-42b3-9283-cecc6b2858ed-maas-api-openshift-ca" pod "f0df24ea-4b3c-42b3-9283-cecc6b2858ed" (UID: "f0df24ea-4b3c-42b3-9283-cecc6b2858ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:20.263733 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.263595 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-workload-certs\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb\" (UID: \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" Oct 30 13:06:20.263733 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.263697 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-istio-podinfo\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb\" (UID: \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" Oct 30 13:06:20.263847 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.263804 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-istiod-ca-cert\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb\" (UID: \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" Oct 30 13:06:20.263887 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.263866 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-istio-envoy\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb\" (UID: \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" Oct 30 13:06:20.263925 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.263886 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-istio-token\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb\" (UID: \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" Oct 30 13:06:20.264032 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.264002 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-credential-socket\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb\" (UID: \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" Oct 30 13:06:20.264159 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.264141 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-istio-data\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb\" (UID: \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" Oct 30 13:06:20.264261 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.264248 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0df24ea-4b3c-42b3-9283-cecc6b2858ed-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:06:20.265502 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.265478 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0df24ea-4b3c-42b3-9283-cecc6b2858ed-kube-api-access-pgzrs" pod "f0df24ea-4b3c-42b3-9283-cecc6b2858ed" (UID: "f0df24ea-4b3c-42b3-9283-cecc6b2858ed"). InnerVolumeSpecName "kube-api-access-pgzrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:20.265590 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.265563 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0df24ea-4b3c-42b3-9283-cecc6b2858ed-maas-api-xks-ca" pod "f0df24ea-4b3c-42b3-9283-cecc6b2858ed" (UID: "f0df24ea-4b3c-42b3-9283-cecc6b2858ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:06:20.366348 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.366308 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-istio-data\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb\" (UID: \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" Oct 30 13:06:20.366485 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.366404 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-workload-socket\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb\" (UID: \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" Oct 30 13:06:20.366485 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.366449 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmrmn\" (UniqueName: \"kubernetes.io/projected/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-kube-api-access-jmrmn\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb\" (UID: \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" Oct 30 13:06:20.366593 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.366505 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-workload-certs\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb\" (UID: \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" Oct 30 13:06:20.366593 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.366539 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-istio-podinfo\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb\" (UID: \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" Oct 30 13:06:20.366850 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.366827 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-istiod-ca-cert\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb\" (UID: \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" Oct 30 13:06:20.366925 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.366908 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-istio-envoy\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb\" (UID: \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" Oct 30 13:06:20.366980 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.366946 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-istio-token\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb\" (UID: \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" Oct 30 13:06:20.367038 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.367023 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-credential-socket\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb\" (UID: \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" Oct 30 13:06:20.367184 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.367170 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0df24ea-4b3c-42b3-9283-cecc6b2858ed-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:06:20.367234 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.367193 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgzrs\" (UniqueName: \"kubernetes.io/projected/f0df24ea-4b3c-42b3-9283-cecc6b2858ed-kube-api-access-pgzrs\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:06:20.367599 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.367574 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-credential-socket\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb\" (UID: \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" Oct 30 13:06:20.367915 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.367861 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-istio-data\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb\" (UID: \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" Oct 30 13:06:20.368164 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.368144 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-workload-socket\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb\" (UID: \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" Oct 30 13:06:20.368761 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.368735 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-workload-certs\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb\" (UID: \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" Oct 30 13:06:20.370063 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.370037 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-istiod-ca-cert\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb\" (UID: \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" Oct 30 13:06:20.375729 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.375703 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-istio-podinfo\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb\" (UID: \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" Oct 30 13:06:20.375817 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.375710 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-istio-envoy\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb\" (UID: \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" Oct 30 13:06:20.394464 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.394440 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmrmn\" (UniqueName: \"kubernetes.io/projected/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-kube-api-access-jmrmn\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb\" (UID: \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" Oct 30 13:06:20.396944 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.396915 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-istio-token\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb\" (UID: \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" Oct 30 13:06:20.468287 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.468259 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" Oct 30 13:06:20.786832 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.786808 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb"] Oct 30 13:06:20.788402 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:06:20.788374 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb607f9a7_6ef2_4ea2_bdb4_352b2312966d.slice/crio-5fb19caa85339fb4ec8e6ebe82d456484cc9b386f339201021ff911b97ff4c7c WatchSource:0}: Error finding container 5fb19caa85339fb4ec8e6ebe82d456484cc9b386f339201021ff911b97ff4c7c: Status 404 returned error can't find the container with id 5fb19caa85339fb4ec8e6ebe82d456484cc9b386f339201021ff911b97ff4c7c Oct 30 13:06:20.790915 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.790882 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:06:20.790980 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.790950 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:06:20.791021 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:20.790981 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:06:21.055728 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:21.055644 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea9f444-sxqcc" Oct 30 13:06:21.055728 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:21.055666 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea9f444-sxqcc" event={"ID":"f0df24ea-4b3c-42b3-9283-cecc6b2858ed","Type":"ContainerDied","Data":"127957e71eb4c586f603321f56d7cca934bb1fc29d2d6ca74c4ae4ab815624fb"} Oct 30 13:06:21.055912 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:21.055728 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="127957e71eb4c586f603321f56d7cca934bb1fc29d2d6ca74c4ae4ab815624fb" Oct 30 13:06:21.057691 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:21.057663 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" event={"ID":"b607f9a7-6ef2-4ea2-bdb4-352b2312966d","Type":"ContainerStarted","Data":"7905db159264c464bfdd28ddb020ee47a98d62f341fb933cb51740e930840481"} Oct 30 13:06:21.057802 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:21.057713 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" event={"ID":"b607f9a7-6ef2-4ea2-bdb4-352b2312966d","Type":"ContainerStarted","Data":"5fb19caa85339fb4ec8e6ebe82d456484cc9b386f339201021ff911b97ff4c7c"} Oct 30 13:06:21.106749 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:21.106703 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" podStartSLOduration=1.106689932 podStartE2EDuration="1.106689932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:21.104536817 +0000 UTC m=+837.573952581" watchObservedRunningTime="2026-10-30 13:06:21.106689932 +0000 UTC m=+837.576105747" Oct 30 13:06:21.468491 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:21.468459 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" Oct 30 13:06:21.469948 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:21.469924 5922 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 13:06:21.470071 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:21.469973 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" podUID="b607f9a7-6ef2-4ea2-bdb4-352b2312966d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 13:06:21.787821 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:21.787745 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:22.476637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:22.468978 5922 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 13:06:22.476637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:22.469038 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" podUID="b607f9a7-6ef2-4ea2-bdb4-352b2312966d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 13:06:23.478636 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:23.469660 5922 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 13:06:23.478636 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:23.469734 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" podUID="b607f9a7-6ef2-4ea2-bdb4-352b2312966d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 13:06:23.553344 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:06:23.553310 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:06:23.590218 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:23.590187 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:24.471445 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:24.470973 5922 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 13:06:24.471445 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:24.471035 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" podUID="b607f9a7-6ef2-4ea2-bdb4-352b2312966d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 13:06:25.475636 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:25.471976 5922 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 13:06:25.475636 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:25.472045 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" podUID="b607f9a7-6ef2-4ea2-bdb4-352b2312966d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 13:06:25.706636 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:25.706545 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:26.475989 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:26.475952 5922 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 13:06:26.476564 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:26.476528 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" podUID="b607f9a7-6ef2-4ea2-bdb4-352b2312966d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 13:06:26.783572 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:26.783489 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" Oct 30 13:06:26.804107 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:26.804078 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" Oct 30 13:06:27.146643 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:27.146539 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" Oct 30 13:06:27.469098 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:27.469071 5922 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 13:06:27.469251 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:27.469116 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" podUID="b607f9a7-6ef2-4ea2-bdb4-352b2312966d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 13:06:27.549991 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:27.549958 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:28.473132 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:28.473103 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" Oct 30 13:06:28.473499 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:28.473474 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" Oct 30 13:06:28.474160 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:28.474143 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" Oct 30 13:06:29.372046 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:29.372012 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:31.058371 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:31.058335 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:32.454174 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:32.454136 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:32.878541 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:32.878466 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:34.315219 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:34.315189 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks"] Oct 30 13:06:34.315876 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:34.315858 5922 cpu_manager.go:414] "containerMap: removing container" podUID="f0df24ea-4b3c-42b3-9283-cecc6b2858ed" containerName="revoke-keys" Oct 30 13:06:34.315876 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:34.315876 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0df24ea-4b3c-42b3-9283-cecc6b2858ed" containerName="revoke-keys" Oct 30 13:06:34.315994 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:34.315984 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0df24ea-4b3c-42b3-9283-cecc6b2858ed" containerName="revoke-keys" Oct 30 13:06:34.320810 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:34.320791 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" Oct 30 13:06:34.344406 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:34.344382 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks"] Oct 30 13:06:34.438518 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:34.438491 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0132701-3b77-4cce-9989-a0d03ac479f3-istiod-ca-cert\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks\" (UID: \"d0132701-3b77-4cce-9989-a0d03ac479f3\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" Oct 30 13:06:34.438646 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:34.438523 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0132701-3b77-4cce-9989-a0d03ac479f3-istio-data\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks\" (UID: \"d0132701-3b77-4cce-9989-a0d03ac479f3\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" Oct 30 13:06:34.438646 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:34.438550 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0132701-3b77-4cce-9989-a0d03ac479f3-istio-podinfo\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks\" (UID: \"d0132701-3b77-4cce-9989-a0d03ac479f3\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" Oct 30 13:06:34.438742 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:34.438666 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0132701-3b77-4cce-9989-a0d03ac479f3-workload-certs\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks\" (UID: \"d0132701-3b77-4cce-9989-a0d03ac479f3\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" Oct 30 13:06:34.438777 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:34.438765 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0132701-3b77-4cce-9989-a0d03ac479f3-workload-socket\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks\" (UID: \"d0132701-3b77-4cce-9989-a0d03ac479f3\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" Oct 30 13:06:34.438815 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:34.438788 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0132701-3b77-4cce-9989-a0d03ac479f3-credential-socket\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks\" (UID: \"d0132701-3b77-4cce-9989-a0d03ac479f3\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" Oct 30 13:06:34.438865 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:34.438815 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5fs7\" (UniqueName: \"kubernetes.io/projected/d0132701-3b77-4cce-9989-a0d03ac479f3-kube-api-access-g5fs7\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks\" (UID: \"d0132701-3b77-4cce-9989-a0d03ac479f3\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" Oct 30 13:06:34.438903 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:34.438896 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0132701-3b77-4cce-9989-a0d03ac479f3-istio-token\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks\" (UID: \"d0132701-3b77-4cce-9989-a0d03ac479f3\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" Oct 30 13:06:34.438936 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:34.438931 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0132701-3b77-4cce-9989-a0d03ac479f3-istio-envoy\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks\" (UID: \"d0132701-3b77-4cce-9989-a0d03ac479f3\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" Oct 30 13:06:34.540101 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:34.540071 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0132701-3b77-4cce-9989-a0d03ac479f3-istiod-ca-cert\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks\" (UID: \"d0132701-3b77-4cce-9989-a0d03ac479f3\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" Oct 30 13:06:34.540223 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:34.540115 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0132701-3b77-4cce-9989-a0d03ac479f3-istio-data\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks\" (UID: \"d0132701-3b77-4cce-9989-a0d03ac479f3\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" Oct 30 13:06:34.540223 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:34.540176 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0132701-3b77-4cce-9989-a0d03ac479f3-istio-podinfo\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks\" (UID: \"d0132701-3b77-4cce-9989-a0d03ac479f3\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" Oct 30 13:06:34.540223 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:34.540209 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0132701-3b77-4cce-9989-a0d03ac479f3-workload-certs\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks\" (UID: \"d0132701-3b77-4cce-9989-a0d03ac479f3\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" Oct 30 13:06:34.540443 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:34.540298 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0132701-3b77-4cce-9989-a0d03ac479f3-workload-socket\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks\" (UID: \"d0132701-3b77-4cce-9989-a0d03ac479f3\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" Oct 30 13:06:34.540443 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:34.540324 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0132701-3b77-4cce-9989-a0d03ac479f3-credential-socket\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks\" (UID: \"d0132701-3b77-4cce-9989-a0d03ac479f3\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" Oct 30 13:06:34.540443 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:34.540353 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5fs7\" (UniqueName: \"kubernetes.io/projected/d0132701-3b77-4cce-9989-a0d03ac479f3-kube-api-access-g5fs7\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks\" (UID: \"d0132701-3b77-4cce-9989-a0d03ac479f3\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" Oct 30 13:06:34.540587 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:34.540448 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0132701-3b77-4cce-9989-a0d03ac479f3-istio-token\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks\" (UID: \"d0132701-3b77-4cce-9989-a0d03ac479f3\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" Oct 30 13:06:34.540587 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:34.540527 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0132701-3b77-4cce-9989-a0d03ac479f3-istio-envoy\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks\" (UID: \"d0132701-3b77-4cce-9989-a0d03ac479f3\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" Oct 30 13:06:34.540587 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:34.540531 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0132701-3b77-4cce-9989-a0d03ac479f3-istio-data\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks\" (UID: \"d0132701-3b77-4cce-9989-a0d03ac479f3\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" Oct 30 13:06:34.540772 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:34.540697 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0132701-3b77-4cce-9989-a0d03ac479f3-workload-certs\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks\" (UID: \"d0132701-3b77-4cce-9989-a0d03ac479f3\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" Oct 30 13:06:34.540899 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:34.540829 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0132701-3b77-4cce-9989-a0d03ac479f3-credential-socket\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks\" (UID: \"d0132701-3b77-4cce-9989-a0d03ac479f3\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" Oct 30 13:06:34.541052 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:34.540999 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0132701-3b77-4cce-9989-a0d03ac479f3-workload-socket\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks\" (UID: \"d0132701-3b77-4cce-9989-a0d03ac479f3\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" Oct 30 13:06:34.541159 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:34.541078 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0132701-3b77-4cce-9989-a0d03ac479f3-istiod-ca-cert\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks\" (UID: \"d0132701-3b77-4cce-9989-a0d03ac479f3\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" Oct 30 13:06:34.542861 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:34.542838 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0132701-3b77-4cce-9989-a0d03ac479f3-istio-envoy\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks\" (UID: \"d0132701-3b77-4cce-9989-a0d03ac479f3\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" Oct 30 13:06:34.543022 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:34.543002 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0132701-3b77-4cce-9989-a0d03ac479f3-istio-podinfo\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks\" (UID: \"d0132701-3b77-4cce-9989-a0d03ac479f3\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" Oct 30 13:06:34.566064 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:34.566008 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0132701-3b77-4cce-9989-a0d03ac479f3-istio-token\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks\" (UID: \"d0132701-3b77-4cce-9989-a0d03ac479f3\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" Oct 30 13:06:34.566064 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:34.566041 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5fs7\" (UniqueName: \"kubernetes.io/projected/d0132701-3b77-4cce-9989-a0d03ac479f3-kube-api-access-g5fs7\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks\" (UID: \"d0132701-3b77-4cce-9989-a0d03ac479f3\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" Oct 30 13:06:34.592304 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:34.592279 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:34.634659 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:34.634384 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" Oct 30 13:06:34.992144 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:34.992120 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks"] Oct 30 13:06:34.993986 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:06:34.993961 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0132701_3b77_4cce_9989_a0d03ac479f3.slice/crio-f7c1e374ece95af5c4b408676e8ef3367cc3fbffedc52e1933d5f5255fc58a75 WatchSource:0}: Error finding container f7c1e374ece95af5c4b408676e8ef3367cc3fbffedc52e1933d5f5255fc58a75: Status 404 returned error can't find the container with id f7c1e374ece95af5c4b408676e8ef3367cc3fbffedc52e1933d5f5255fc58a75 Oct 30 13:06:34.996315 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:34.996285 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:06:34.996396 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:34.996344 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:06:34.996396 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:34.996373 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:06:35.062347 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:35.062323 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3465b539-kjn9s"] Oct 30 13:06:35.068098 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:35.068081 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3465b539-kjn9s" Oct 30 13:06:35.085950 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:35.085927 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3465b539-kjn9s"] Oct 30 13:06:35.095921 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:35.095897 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:06:35.096050 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:35.095975 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:06:35.096050 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:35.096023 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:06:35.096174 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:35.096102 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:06:35.151387 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:35.151359 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bcabc28-e50c-4e82-bd30-79a43685ae14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3465b539-kjn9s\" (UID: \"8bcabc28-e50c-4e82-bd30-79a43685ae14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3465b539-kjn9s" Oct 30 13:06:35.151564 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:35.151475 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bcabc28-e50c-4e82-bd30-79a43685ae14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3465b539-kjn9s\" (UID: \"8bcabc28-e50c-4e82-bd30-79a43685ae14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3465b539-kjn9s" Oct 30 13:06:35.151564 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:35.151498 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n96df\" (UniqueName: \"kubernetes.io/projected/8bcabc28-e50c-4e82-bd30-79a43685ae14-kube-api-access-n96df\") pod \"maas-api-revoke-sub-3465b539-kjn9s\" (UID: \"8bcabc28-e50c-4e82-bd30-79a43685ae14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3465b539-kjn9s" Oct 30 13:06:35.186404 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:35.186375 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" event={"ID":"d0132701-3b77-4cce-9989-a0d03ac479f3","Type":"ContainerStarted","Data":"2ead0939b95900ae542eba74cecb73cc7b39574b7015929f9fef7f6bc367af75"} Oct 30 13:06:35.186542 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:35.186424 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" event={"ID":"d0132701-3b77-4cce-9989-a0d03ac479f3","Type":"ContainerStarted","Data":"f7c1e374ece95af5c4b408676e8ef3367cc3fbffedc52e1933d5f5255fc58a75"} Oct 30 13:06:35.234545 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:35.234502 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" podStartSLOduration=1.234487648 podStartE2EDuration="1.234487648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:35.231490248 +0000 UTC m=+851.700906022" watchObservedRunningTime="2026-10-30 13:06:35.234487648 +0000 UTC m=+851.703903393" Oct 30 13:06:35.252481 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:35.252420 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bcabc28-e50c-4e82-bd30-79a43685ae14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3465b539-kjn9s\" (UID: \"8bcabc28-e50c-4e82-bd30-79a43685ae14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3465b539-kjn9s" Oct 30 13:06:35.252481 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:35.252455 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n96df\" (UniqueName: \"kubernetes.io/projected/8bcabc28-e50c-4e82-bd30-79a43685ae14-kube-api-access-n96df\") pod \"maas-api-revoke-sub-3465b539-kjn9s\" (UID: \"8bcabc28-e50c-4e82-bd30-79a43685ae14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3465b539-kjn9s" Oct 30 13:06:35.252698 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:35.252598 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bcabc28-e50c-4e82-bd30-79a43685ae14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3465b539-kjn9s\" (UID: \"8bcabc28-e50c-4e82-bd30-79a43685ae14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3465b539-kjn9s" Oct 30 13:06:35.253261 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:35.253239 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bcabc28-e50c-4e82-bd30-79a43685ae14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3465b539-kjn9s\" (UID: \"8bcabc28-e50c-4e82-bd30-79a43685ae14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3465b539-kjn9s" Oct 30 13:06:35.254762 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:35.254745 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bcabc28-e50c-4e82-bd30-79a43685ae14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3465b539-kjn9s\" (UID: \"8bcabc28-e50c-4e82-bd30-79a43685ae14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3465b539-kjn9s" Oct 30 13:06:35.277546 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:35.277526 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n96df\" (UniqueName: \"kubernetes.io/projected/8bcabc28-e50c-4e82-bd30-79a43685ae14-kube-api-access-n96df\") pod \"maas-api-revoke-sub-3465b539-kjn9s\" (UID: \"8bcabc28-e50c-4e82-bd30-79a43685ae14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3465b539-kjn9s" Oct 30 13:06:35.382109 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:35.382078 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3465b539-kjn9s" Oct 30 13:06:35.635420 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:35.635336 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" Oct 30 13:06:35.692570 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:35.692542 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3465b539-kjn9s"] Oct 30 13:06:35.694152 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:06:35.694119 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bcabc28_e50c_4e82_bd30_79a43685ae14.slice/crio-0173a7f9a1a7594eb730300415b5a4924e4753db30b6abe2a0b4a91f7271a09f WatchSource:0}: Error finding container 0173a7f9a1a7594eb730300415b5a4924e4753db30b6abe2a0b4a91f7271a09f: Status 404 returned error can't find the container with id 0173a7f9a1a7594eb730300415b5a4924e4753db30b6abe2a0b4a91f7271a09f Oct 30 13:06:36.195110 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:36.195081 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3465b539-kjn9s" event={"ID":"8bcabc28-e50c-4e82-bd30-79a43685ae14","Type":"ContainerStarted","Data":"0173a7f9a1a7594eb730300415b5a4924e4753db30b6abe2a0b4a91f7271a09f"} Oct 30 13:06:36.441331 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:36.441246 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:36.637832 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:36.637802 5922 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 13:06:36.638023 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:36.637858 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" podUID="d0132701-3b77-4cce-9989-a0d03ac479f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 13:06:37.157480 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:37.157401 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:37.203706 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:37.203673 5922 generic.go:356] "Generic (PLEG): container finished" podID="8bcabc28-e50c-4e82-bd30-79a43685ae14" containerID="dbd4f36d944fd7fa66bba1fdb3699f5f7656ac7f12afae4dc4e201a99f1b4560" exitCode=0 Oct 30 13:06:37.203865 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:37.203713 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3465b539-kjn9s" event={"ID":"8bcabc28-e50c-4e82-bd30-79a43685ae14","Type":"ContainerDied","Data":"dbd4f36d944fd7fa66bba1fdb3699f5f7656ac7f12afae4dc4e201a99f1b4560"} Oct 30 13:06:37.635567 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:37.635532 5922 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 13:06:37.636044 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:37.635582 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" podUID="d0132701-3b77-4cce-9989-a0d03ac479f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 13:06:38.401991 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:38.401966 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3465b539-kjn9s" Oct 30 13:06:38.493351 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:38.493262 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:38.516268 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:38.516239 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bcabc28-e50c-4e82-bd30-79a43685ae14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bcabc28-e50c-4e82-bd30-79a43685ae14-maas-api-xks-ca\") pod \"8bcabc28-e50c-4e82-bd30-79a43685ae14\" (UID: \"8bcabc28-e50c-4e82-bd30-79a43685ae14\") " Oct 30 13:06:38.516396 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:38.516289 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bcabc28-e50c-4e82-bd30-79a43685ae14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bcabc28-e50c-4e82-bd30-79a43685ae14-maas-api-openshift-ca\") pod \"8bcabc28-e50c-4e82-bd30-79a43685ae14\" (UID: \"8bcabc28-e50c-4e82-bd30-79a43685ae14\") " Oct 30 13:06:38.516396 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:38.516368 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bcabc28-e50c-4e82-bd30-79a43685ae14-kube-api-access-n96df\" (UniqueName: \"kubernetes.io/projected/8bcabc28-e50c-4e82-bd30-79a43685ae14-kube-api-access-n96df\") pod \"8bcabc28-e50c-4e82-bd30-79a43685ae14\" (UID: \"8bcabc28-e50c-4e82-bd30-79a43685ae14\") " Oct 30 13:06:38.516678 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:38.516649 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bcabc28-e50c-4e82-bd30-79a43685ae14-maas-api-openshift-ca" pod "8bcabc28-e50c-4e82-bd30-79a43685ae14" (UID: "8bcabc28-e50c-4e82-bd30-79a43685ae14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:38.517007 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:38.516989 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bcabc28-e50c-4e82-bd30-79a43685ae14-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:06:38.518389 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:38.518364 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bcabc28-e50c-4e82-bd30-79a43685ae14-kube-api-access-n96df" pod "8bcabc28-e50c-4e82-bd30-79a43685ae14" (UID: "8bcabc28-e50c-4e82-bd30-79a43685ae14"). InnerVolumeSpecName "kube-api-access-n96df". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:38.518389 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:38.518370 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bcabc28-e50c-4e82-bd30-79a43685ae14-maas-api-xks-ca" pod "8bcabc28-e50c-4e82-bd30-79a43685ae14" (UID: "8bcabc28-e50c-4e82-bd30-79a43685ae14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:06:38.617813 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:38.617778 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bcabc28-e50c-4e82-bd30-79a43685ae14-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:06:38.617813 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:38.617805 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n96df\" (UniqueName: \"kubernetes.io/projected/8bcabc28-e50c-4e82-bd30-79a43685ae14-kube-api-access-n96df\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:06:38.635759 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:38.635728 5922 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 13:06:38.636125 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:38.635779 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" podUID="d0132701-3b77-4cce-9989-a0d03ac479f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 13:06:39.217973 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:39.217940 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3465b539-kjn9s" Oct 30 13:06:39.217973 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:39.217967 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3465b539-kjn9s" event={"ID":"8bcabc28-e50c-4e82-bd30-79a43685ae14","Type":"ContainerDied","Data":"0173a7f9a1a7594eb730300415b5a4924e4753db30b6abe2a0b4a91f7271a09f"} Oct 30 13:06:39.218181 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:39.218019 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0173a7f9a1a7594eb730300415b5a4924e4753db30b6abe2a0b4a91f7271a09f" Oct 30 13:06:39.589039 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:39.588960 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1b3044b-8p7s2"] Oct 30 13:06:39.589543 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:39.589528 5922 cpu_manager.go:414] "containerMap: removing container" podUID="8bcabc28-e50c-4e82-bd30-79a43685ae14" containerName="revoke-keys" Oct 30 13:06:39.589586 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:39.589545 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bcabc28-e50c-4e82-bd30-79a43685ae14" containerName="revoke-keys" Oct 30 13:06:39.589685 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:39.589674 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bcabc28-e50c-4e82-bd30-79a43685ae14" containerName="revoke-keys" Oct 30 13:06:39.592958 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:39.592943 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b3044b-8p7s2" Oct 30 13:06:39.602140 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:39.602118 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1b3044b-8p7s2"] Oct 30 13:06:39.610343 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:39.610319 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:06:39.610450 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:39.610407 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:06:39.610450 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:39.610430 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:06:39.610553 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:39.610472 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:06:39.636202 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:39.636162 5922 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 13:06:39.636557 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:39.636231 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" podUID="d0132701-3b77-4cce-9989-a0d03ac479f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 13:06:39.731226 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:39.731196 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/692066d9-7507-4e4f-95aa-60b4fef097cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1b3044b-8p7s2\" (UID: \"692066d9-7507-4e4f-95aa-60b4fef097cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b3044b-8p7s2" Oct 30 13:06:39.731376 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:39.731337 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwsvp\" (UniqueName: \"kubernetes.io/projected/692066d9-7507-4e4f-95aa-60b4fef097cb-kube-api-access-fwsvp\") pod \"maas-api-revoke-sub-f1b3044b-8p7s2\" (UID: \"692066d9-7507-4e4f-95aa-60b4fef097cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b3044b-8p7s2" Oct 30 13:06:39.731598 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:39.731580 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/692066d9-7507-4e4f-95aa-60b4fef097cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1b3044b-8p7s2\" (UID: \"692066d9-7507-4e4f-95aa-60b4fef097cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b3044b-8p7s2" Oct 30 13:06:39.832602 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:39.832566 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/692066d9-7507-4e4f-95aa-60b4fef097cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1b3044b-8p7s2\" (UID: \"692066d9-7507-4e4f-95aa-60b4fef097cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b3044b-8p7s2" Oct 30 13:06:39.832815 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:39.832752 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/692066d9-7507-4e4f-95aa-60b4fef097cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1b3044b-8p7s2\" (UID: \"692066d9-7507-4e4f-95aa-60b4fef097cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b3044b-8p7s2" Oct 30 13:06:39.832815 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:39.832808 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwsvp\" (UniqueName: \"kubernetes.io/projected/692066d9-7507-4e4f-95aa-60b4fef097cb-kube-api-access-fwsvp\") pod \"maas-api-revoke-sub-f1b3044b-8p7s2\" (UID: \"692066d9-7507-4e4f-95aa-60b4fef097cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b3044b-8p7s2" Oct 30 13:06:39.833250 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:39.833229 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/692066d9-7507-4e4f-95aa-60b4fef097cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1b3044b-8p7s2\" (UID: \"692066d9-7507-4e4f-95aa-60b4fef097cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b3044b-8p7s2" Oct 30 13:06:39.835135 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:39.835117 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/692066d9-7507-4e4f-95aa-60b4fef097cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1b3044b-8p7s2\" (UID: \"692066d9-7507-4e4f-95aa-60b4fef097cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b3044b-8p7s2" Oct 30 13:06:39.857395 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:39.857344 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwsvp\" (UniqueName: \"kubernetes.io/projected/692066d9-7507-4e4f-95aa-60b4fef097cb-kube-api-access-fwsvp\") pod \"maas-api-revoke-sub-f1b3044b-8p7s2\" (UID: \"692066d9-7507-4e4f-95aa-60b4fef097cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b3044b-8p7s2" Oct 30 13:06:39.908156 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:39.908128 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b3044b-8p7s2" Oct 30 13:06:40.230117 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:40.230092 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1b3044b-8p7s2"] Oct 30 13:06:40.231648 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:06:40.231603 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod692066d9_7507_4e4f_95aa_60b4fef097cb.slice/crio-d25f631d03e7010762fe3cb196190c99bde919851d7f042e3f6787b2ed460e96 WatchSource:0}: Error finding container d25f631d03e7010762fe3cb196190c99bde919851d7f042e3f6787b2ed460e96: Status 404 returned error can't find the container with id d25f631d03e7010762fe3cb196190c99bde919851d7f042e3f6787b2ed460e96 Oct 30 13:06:40.584634 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:40.582207 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:40.635658 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:40.635594 5922 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 13:06:40.635889 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:40.635679 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" podUID="d0132701-3b77-4cce-9989-a0d03ac479f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 13:06:41.232361 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:41.232333 5922 generic.go:356] "Generic (PLEG): container finished" podID="692066d9-7507-4e4f-95aa-60b4fef097cb" containerID="a981deff70dc47bde8f69479c6412db86f286d6ec6bd893da134fa0f03c33421" exitCode=0 Oct 30 13:06:41.232759 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:41.232421 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b3044b-8p7s2" event={"ID":"692066d9-7507-4e4f-95aa-60b4fef097cb","Type":"ContainerDied","Data":"a981deff70dc47bde8f69479c6412db86f286d6ec6bd893da134fa0f03c33421"} Oct 30 13:06:41.232759 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:41.232522 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b3044b-8p7s2" event={"ID":"692066d9-7507-4e4f-95aa-60b4fef097cb","Type":"ContainerStarted","Data":"d25f631d03e7010762fe3cb196190c99bde919851d7f042e3f6787b2ed460e96"} Oct 30 13:06:41.491085 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:41.491053 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:06:41.635750 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:41.635721 5922 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 13:06:41.635929 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:41.635769 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" podUID="d0132701-3b77-4cce-9989-a0d03ac479f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 13:06:42.378169 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:42.376441 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:42.453743 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:42.453712 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b3044b-8p7s2" Oct 30 13:06:42.478519 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:42.478487 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/692066d9-7507-4e4f-95aa-60b4fef097cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/692066d9-7507-4e4f-95aa-60b4fef097cb-maas-api-xks-ca\") pod \"692066d9-7507-4e4f-95aa-60b4fef097cb\" (UID: \"692066d9-7507-4e4f-95aa-60b4fef097cb\") " Oct 30 13:06:42.478736 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:42.478622 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/692066d9-7507-4e4f-95aa-60b4fef097cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/692066d9-7507-4e4f-95aa-60b4fef097cb-maas-api-openshift-ca\") pod \"692066d9-7507-4e4f-95aa-60b4fef097cb\" (UID: \"692066d9-7507-4e4f-95aa-60b4fef097cb\") " Oct 30 13:06:42.478736 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:42.478686 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/692066d9-7507-4e4f-95aa-60b4fef097cb-kube-api-access-fwsvp\" (UniqueName: \"kubernetes.io/projected/692066d9-7507-4e4f-95aa-60b4fef097cb-kube-api-access-fwsvp\") pod \"692066d9-7507-4e4f-95aa-60b4fef097cb\" (UID: \"692066d9-7507-4e4f-95aa-60b4fef097cb\") " Oct 30 13:06:42.479268 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:42.479233 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/692066d9-7507-4e4f-95aa-60b4fef097cb-maas-api-openshift-ca" pod "692066d9-7507-4e4f-95aa-60b4fef097cb" (UID: "692066d9-7507-4e4f-95aa-60b4fef097cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:42.481200 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:42.481175 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/692066d9-7507-4e4f-95aa-60b4fef097cb-kube-api-access-fwsvp" pod "692066d9-7507-4e4f-95aa-60b4fef097cb" (UID: "692066d9-7507-4e4f-95aa-60b4fef097cb"). InnerVolumeSpecName "kube-api-access-fwsvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:42.481576 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:42.481558 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/692066d9-7507-4e4f-95aa-60b4fef097cb-maas-api-xks-ca" pod "692066d9-7507-4e4f-95aa-60b4fef097cb" (UID: "692066d9-7507-4e4f-95aa-60b4fef097cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:06:42.579735 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:42.579664 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/692066d9-7507-4e4f-95aa-60b4fef097cb-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:06:42.579735 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:42.579692 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwsvp\" (UniqueName: \"kubernetes.io/projected/692066d9-7507-4e4f-95aa-60b4fef097cb-kube-api-access-fwsvp\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:06:42.579735 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:42.579703 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/692066d9-7507-4e4f-95aa-60b4fef097cb-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:06:42.640416 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:42.640389 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" Oct 30 13:06:42.640861 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:42.640842 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" Oct 30 13:06:42.641419 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:42.641397 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" Oct 30 13:06:43.252755 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:43.252723 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b3044b-8p7s2" Oct 30 13:06:43.252961 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:43.252756 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b3044b-8p7s2" event={"ID":"692066d9-7507-4e4f-95aa-60b4fef097cb","Type":"ContainerDied","Data":"d25f631d03e7010762fe3cb196190c99bde919851d7f042e3f6787b2ed460e96"} Oct 30 13:06:43.252961 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:43.252814 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d25f631d03e7010762fe3cb196190c99bde919851d7f042e3f6787b2ed460e96" Oct 30 13:06:44.193466 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:44.193427 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:46.302188 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:46.302156 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:48.324548 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:48.324518 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:49.265698 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.265664 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63cdf4c0-px7v6"] Oct 30 13:06:49.266252 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.266239 5922 cpu_manager.go:414] "containerMap: removing container" podUID="692066d9-7507-4e4f-95aa-60b4fef097cb" containerName="revoke-keys" Oct 30 13:06:49.266314 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.266254 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="692066d9-7507-4e4f-95aa-60b4fef097cb" containerName="revoke-keys" Oct 30 13:06:49.266355 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.266345 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="692066d9-7507-4e4f-95aa-60b4fef097cb" containerName="revoke-keys" Oct 30 13:06:49.270056 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.270040 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63cdf4c0-px7v6" Oct 30 13:06:49.291960 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.291916 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63cdf4c0-px7v6"] Oct 30 13:06:49.298788 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.298765 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:06:49.298899 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.298835 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:06:49.298899 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.298874 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:06:49.298969 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.298900 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:06:49.357988 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.357955 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6rlm\" (UniqueName: \"kubernetes.io/projected/6ed1ede4-062b-4b91-b25c-8f583e120427-kube-api-access-j6rlm\") pod \"maas-api-revoke-sub-63cdf4c0-px7v6\" (UID: \"6ed1ede4-062b-4b91-b25c-8f583e120427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63cdf4c0-px7v6" Oct 30 13:06:49.358398 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.358145 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ed1ede4-062b-4b91-b25c-8f583e120427-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63cdf4c0-px7v6\" (UID: \"6ed1ede4-062b-4b91-b25c-8f583e120427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63cdf4c0-px7v6" Oct 30 13:06:49.358398 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.358288 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ed1ede4-062b-4b91-b25c-8f583e120427-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63cdf4c0-px7v6\" (UID: \"6ed1ede4-062b-4b91-b25c-8f583e120427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63cdf4c0-px7v6" Oct 30 13:06:49.372172 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.372143 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m"] Oct 30 13:06:49.376378 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.376360 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" Oct 30 13:06:49.400044 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.400015 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m"] Oct 30 13:06:49.459842 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.459817 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6rlm\" (UniqueName: \"kubernetes.io/projected/6ed1ede4-062b-4b91-b25c-8f583e120427-kube-api-access-j6rlm\") pod \"maas-api-revoke-sub-63cdf4c0-px7v6\" (UID: \"6ed1ede4-062b-4b91-b25c-8f583e120427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63cdf4c0-px7v6" Oct 30 13:06:49.460009 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.459849 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d034a1b-2482-4f51-b89c-6300d660b7ee-istio-data\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m\" (UID: \"7d034a1b-2482-4f51-b89c-6300d660b7ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" Oct 30 13:06:49.460009 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.459871 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d034a1b-2482-4f51-b89c-6300d660b7ee-workload-socket\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m\" (UID: \"7d034a1b-2482-4f51-b89c-6300d660b7ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" Oct 30 13:06:49.460009 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.459907 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d034a1b-2482-4f51-b89c-6300d660b7ee-istio-envoy\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m\" (UID: \"7d034a1b-2482-4f51-b89c-6300d660b7ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" Oct 30 13:06:49.460167 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.460076 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d034a1b-2482-4f51-b89c-6300d660b7ee-credential-socket\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m\" (UID: \"7d034a1b-2482-4f51-b89c-6300d660b7ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" Oct 30 13:06:49.460167 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.460150 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skrnn\" (UniqueName: \"kubernetes.io/projected/7d034a1b-2482-4f51-b89c-6300d660b7ee-kube-api-access-skrnn\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m\" (UID: \"7d034a1b-2482-4f51-b89c-6300d660b7ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" Oct 30 13:06:49.460249 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.460177 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ed1ede4-062b-4b91-b25c-8f583e120427-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63cdf4c0-px7v6\" (UID: \"6ed1ede4-062b-4b91-b25c-8f583e120427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63cdf4c0-px7v6" Oct 30 13:06:49.460249 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.460192 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d034a1b-2482-4f51-b89c-6300d660b7ee-istio-token\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m\" (UID: \"7d034a1b-2482-4f51-b89c-6300d660b7ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" Oct 30 13:06:49.460249 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.460209 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d034a1b-2482-4f51-b89c-6300d660b7ee-istiod-ca-cert\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m\" (UID: \"7d034a1b-2482-4f51-b89c-6300d660b7ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" Oct 30 13:06:49.460405 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.460387 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ed1ede4-062b-4b91-b25c-8f583e120427-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63cdf4c0-px7v6\" (UID: \"6ed1ede4-062b-4b91-b25c-8f583e120427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63cdf4c0-px7v6" Oct 30 13:06:49.460459 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.460439 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d034a1b-2482-4f51-b89c-6300d660b7ee-istio-podinfo\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m\" (UID: \"7d034a1b-2482-4f51-b89c-6300d660b7ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" Oct 30 13:06:49.460520 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.460503 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d034a1b-2482-4f51-b89c-6300d660b7ee-workload-certs\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m\" (UID: \"7d034a1b-2482-4f51-b89c-6300d660b7ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" Oct 30 13:06:49.461012 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.460984 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ed1ede4-062b-4b91-b25c-8f583e120427-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63cdf4c0-px7v6\" (UID: \"6ed1ede4-062b-4b91-b25c-8f583e120427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63cdf4c0-px7v6" Oct 30 13:06:49.462413 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.462393 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ed1ede4-062b-4b91-b25c-8f583e120427-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63cdf4c0-px7v6\" (UID: \"6ed1ede4-062b-4b91-b25c-8f583e120427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63cdf4c0-px7v6" Oct 30 13:06:49.484360 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.484331 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6rlm\" (UniqueName: \"kubernetes.io/projected/6ed1ede4-062b-4b91-b25c-8f583e120427-kube-api-access-j6rlm\") pod \"maas-api-revoke-sub-63cdf4c0-px7v6\" (UID: \"6ed1ede4-062b-4b91-b25c-8f583e120427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63cdf4c0-px7v6" Oct 30 13:06:49.561944 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.561883 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d034a1b-2482-4f51-b89c-6300d660b7ee-istio-podinfo\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m\" (UID: \"7d034a1b-2482-4f51-b89c-6300d660b7ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" Oct 30 13:06:49.561944 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.561927 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d034a1b-2482-4f51-b89c-6300d660b7ee-workload-certs\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m\" (UID: \"7d034a1b-2482-4f51-b89c-6300d660b7ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" Oct 30 13:06:49.562101 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.561968 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d034a1b-2482-4f51-b89c-6300d660b7ee-istio-data\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m\" (UID: \"7d034a1b-2482-4f51-b89c-6300d660b7ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" Oct 30 13:06:49.562101 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.561999 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d034a1b-2482-4f51-b89c-6300d660b7ee-workload-socket\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m\" (UID: \"7d034a1b-2482-4f51-b89c-6300d660b7ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" Oct 30 13:06:49.562101 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.562051 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d034a1b-2482-4f51-b89c-6300d660b7ee-istio-envoy\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m\" (UID: \"7d034a1b-2482-4f51-b89c-6300d660b7ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" Oct 30 13:06:49.562250 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.562130 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d034a1b-2482-4f51-b89c-6300d660b7ee-credential-socket\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m\" (UID: \"7d034a1b-2482-4f51-b89c-6300d660b7ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" Oct 30 13:06:49.562307 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.562273 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skrnn\" (UniqueName: \"kubernetes.io/projected/7d034a1b-2482-4f51-b89c-6300d660b7ee-kube-api-access-skrnn\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m\" (UID: \"7d034a1b-2482-4f51-b89c-6300d660b7ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" Oct 30 13:06:49.562360 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.562328 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d034a1b-2482-4f51-b89c-6300d660b7ee-istio-token\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m\" (UID: \"7d034a1b-2482-4f51-b89c-6300d660b7ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" Oct 30 13:06:49.562409 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.562377 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d034a1b-2482-4f51-b89c-6300d660b7ee-workload-certs\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m\" (UID: \"7d034a1b-2482-4f51-b89c-6300d660b7ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" Oct 30 13:06:49.562456 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.562381 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d034a1b-2482-4f51-b89c-6300d660b7ee-istiod-ca-cert\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m\" (UID: \"7d034a1b-2482-4f51-b89c-6300d660b7ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" Oct 30 13:06:49.562721 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.562695 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d034a1b-2482-4f51-b89c-6300d660b7ee-istio-data\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m\" (UID: \"7d034a1b-2482-4f51-b89c-6300d660b7ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" Oct 30 13:06:49.562810 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.562749 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d034a1b-2482-4f51-b89c-6300d660b7ee-workload-socket\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m\" (UID: \"7d034a1b-2482-4f51-b89c-6300d660b7ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" Oct 30 13:06:49.562868 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.562801 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d034a1b-2482-4f51-b89c-6300d660b7ee-credential-socket\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m\" (UID: \"7d034a1b-2482-4f51-b89c-6300d660b7ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" Oct 30 13:06:49.563112 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.563092 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d034a1b-2482-4f51-b89c-6300d660b7ee-istiod-ca-cert\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m\" (UID: \"7d034a1b-2482-4f51-b89c-6300d660b7ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" Oct 30 13:06:49.564149 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.564132 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d034a1b-2482-4f51-b89c-6300d660b7ee-istio-podinfo\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m\" (UID: \"7d034a1b-2482-4f51-b89c-6300d660b7ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" Oct 30 13:06:49.564326 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.564307 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d034a1b-2482-4f51-b89c-6300d660b7ee-istio-envoy\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m\" (UID: \"7d034a1b-2482-4f51-b89c-6300d660b7ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" Oct 30 13:06:49.583702 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.583683 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63cdf4c0-px7v6" Oct 30 13:06:49.593089 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.593070 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skrnn\" (UniqueName: \"kubernetes.io/projected/7d034a1b-2482-4f51-b89c-6300d660b7ee-kube-api-access-skrnn\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m\" (UID: \"7d034a1b-2482-4f51-b89c-6300d660b7ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" Oct 30 13:06:49.593176 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.593094 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d034a1b-2482-4f51-b89c-6300d660b7ee-istio-token\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m\" (UID: \"7d034a1b-2482-4f51-b89c-6300d660b7ee\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" Oct 30 13:06:49.693498 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.693469 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" Oct 30 13:06:49.760507 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.760479 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn"] Oct 30 13:06:49.766191 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.766168 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" Oct 30 13:06:49.791475 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.791451 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn"] Oct 30 13:06:49.867926 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.867853 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c61d2f25-754d-4538-b154-b9095bffe96f-istio-podinfo\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn\" (UID: \"c61d2f25-754d-4538-b154-b9095bffe96f\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" Oct 30 13:06:49.867926 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.867892 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c61d2f25-754d-4538-b154-b9095bffe96f-istiod-ca-cert\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn\" (UID: \"c61d2f25-754d-4538-b154-b9095bffe96f\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" Oct 30 13:06:49.868087 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.868024 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c61d2f25-754d-4538-b154-b9095bffe96f-workload-socket\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn\" (UID: \"c61d2f25-754d-4538-b154-b9095bffe96f\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" Oct 30 13:06:49.868087 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.868075 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c61d2f25-754d-4538-b154-b9095bffe96f-workload-certs\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn\" (UID: \"c61d2f25-754d-4538-b154-b9095bffe96f\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" Oct 30 13:06:49.868209 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.868188 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c61d2f25-754d-4538-b154-b9095bffe96f-credential-socket\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn\" (UID: \"c61d2f25-754d-4538-b154-b9095bffe96f\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" Oct 30 13:06:49.868276 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.868225 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c61d2f25-754d-4538-b154-b9095bffe96f-istio-token\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn\" (UID: \"c61d2f25-754d-4538-b154-b9095bffe96f\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" Oct 30 13:06:49.868276 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.868254 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbvm7\" (UniqueName: \"kubernetes.io/projected/c61d2f25-754d-4538-b154-b9095bffe96f-kube-api-access-jbvm7\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn\" (UID: \"c61d2f25-754d-4538-b154-b9095bffe96f\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" Oct 30 13:06:49.868454 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.868428 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c61d2f25-754d-4538-b154-b9095bffe96f-istio-envoy\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn\" (UID: \"c61d2f25-754d-4538-b154-b9095bffe96f\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" Oct 30 13:06:49.868510 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.868482 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c61d2f25-754d-4538-b154-b9095bffe96f-istio-data\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn\" (UID: \"c61d2f25-754d-4538-b154-b9095bffe96f\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" Oct 30 13:06:49.888052 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.888032 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63cdf4c0-px7v6"] Oct 30 13:06:49.889520 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:06:49.889498 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed1ede4_062b_4b91_b25c_8f583e120427.slice/crio-ee3cfc0d2b9b35386d9c6f9086a39d40fdbbfe8a95a4a36811f0f1176e68b66a WatchSource:0}: Error finding container ee3cfc0d2b9b35386d9c6f9086a39d40fdbbfe8a95a4a36811f0f1176e68b66a: Status 404 returned error can't find the container with id ee3cfc0d2b9b35386d9c6f9086a39d40fdbbfe8a95a4a36811f0f1176e68b66a Oct 30 13:06:49.973120 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.973094 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c61d2f25-754d-4538-b154-b9095bffe96f-workload-socket\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn\" (UID: \"c61d2f25-754d-4538-b154-b9095bffe96f\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" Oct 30 13:06:49.973288 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.973151 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c61d2f25-754d-4538-b154-b9095bffe96f-workload-certs\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn\" (UID: \"c61d2f25-754d-4538-b154-b9095bffe96f\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" Oct 30 13:06:49.973288 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.973188 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c61d2f25-754d-4538-b154-b9095bffe96f-credential-socket\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn\" (UID: \"c61d2f25-754d-4538-b154-b9095bffe96f\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" Oct 30 13:06:49.973288 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.973208 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c61d2f25-754d-4538-b154-b9095bffe96f-istio-token\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn\" (UID: \"c61d2f25-754d-4538-b154-b9095bffe96f\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" Oct 30 13:06:49.973288 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.973225 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbvm7\" (UniqueName: \"kubernetes.io/projected/c61d2f25-754d-4538-b154-b9095bffe96f-kube-api-access-jbvm7\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn\" (UID: \"c61d2f25-754d-4538-b154-b9095bffe96f\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" Oct 30 13:06:49.973515 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.973289 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c61d2f25-754d-4538-b154-b9095bffe96f-istio-envoy\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn\" (UID: \"c61d2f25-754d-4538-b154-b9095bffe96f\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" Oct 30 13:06:49.973515 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.973309 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c61d2f25-754d-4538-b154-b9095bffe96f-istio-data\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn\" (UID: \"c61d2f25-754d-4538-b154-b9095bffe96f\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" Oct 30 13:06:49.973515 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.973397 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c61d2f25-754d-4538-b154-b9095bffe96f-istio-podinfo\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn\" (UID: \"c61d2f25-754d-4538-b154-b9095bffe96f\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" Oct 30 13:06:49.973515 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.973437 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c61d2f25-754d-4538-b154-b9095bffe96f-istiod-ca-cert\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn\" (UID: \"c61d2f25-754d-4538-b154-b9095bffe96f\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" Oct 30 13:06:49.973714 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.973521 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c61d2f25-754d-4538-b154-b9095bffe96f-workload-socket\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn\" (UID: \"c61d2f25-754d-4538-b154-b9095bffe96f\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" Oct 30 13:06:49.973714 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.973542 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c61d2f25-754d-4538-b154-b9095bffe96f-workload-certs\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn\" (UID: \"c61d2f25-754d-4538-b154-b9095bffe96f\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" Oct 30 13:06:49.973714 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.973576 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c61d2f25-754d-4538-b154-b9095bffe96f-credential-socket\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn\" (UID: \"c61d2f25-754d-4538-b154-b9095bffe96f\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" Oct 30 13:06:49.973998 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.973974 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c61d2f25-754d-4538-b154-b9095bffe96f-istio-data\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn\" (UID: \"c61d2f25-754d-4538-b154-b9095bffe96f\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" Oct 30 13:06:49.974212 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.974191 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c61d2f25-754d-4538-b154-b9095bffe96f-istiod-ca-cert\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn\" (UID: \"c61d2f25-754d-4538-b154-b9095bffe96f\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" Oct 30 13:06:49.975587 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.975553 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c61d2f25-754d-4538-b154-b9095bffe96f-istio-envoy\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn\" (UID: \"c61d2f25-754d-4538-b154-b9095bffe96f\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" Oct 30 13:06:49.975822 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.975804 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c61d2f25-754d-4538-b154-b9095bffe96f-istio-podinfo\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn\" (UID: \"c61d2f25-754d-4538-b154-b9095bffe96f\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" Oct 30 13:06:49.997821 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.997761 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c61d2f25-754d-4538-b154-b9095bffe96f-istio-token\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn\" (UID: \"c61d2f25-754d-4538-b154-b9095bffe96f\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" Oct 30 13:06:49.998075 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:49.998051 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbvm7\" (UniqueName: \"kubernetes.io/projected/c61d2f25-754d-4538-b154-b9095bffe96f-kube-api-access-jbvm7\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn\" (UID: \"c61d2f25-754d-4538-b154-b9095bffe96f\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" Oct 30 13:06:50.002861 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:50.002841 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m"] Oct 30 13:06:50.004853 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:06:50.004823 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d034a1b_2482_4f51_b89c_6300d660b7ee.slice/crio-15ba2a9876f623fe541b1a2e7f02c11138ccc805e70bc631838e1831147cb608 WatchSource:0}: Error finding container 15ba2a9876f623fe541b1a2e7f02c11138ccc805e70bc631838e1831147cb608: Status 404 returned error can't find the container with id 15ba2a9876f623fe541b1a2e7f02c11138ccc805e70bc631838e1831147cb608 Oct 30 13:06:50.007327 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:50.007295 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:06:50.007401 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:50.007357 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:06:50.007401 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:50.007386 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:06:50.083276 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:50.083252 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" Oct 30 13:06:50.304960 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:50.304858 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:50.308244 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:50.308216 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63cdf4c0-px7v6" event={"ID":"6ed1ede4-062b-4b91-b25c-8f583e120427","Type":"ContainerStarted","Data":"9834914807a62848a58c7ecec261a51ff8e0619cfd1e27a2c7d161bf53eff8be"} Oct 30 13:06:50.308375 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:50.308262 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63cdf4c0-px7v6" event={"ID":"6ed1ede4-062b-4b91-b25c-8f583e120427","Type":"ContainerStarted","Data":"ee3cfc0d2b9b35386d9c6f9086a39d40fdbbfe8a95a4a36811f0f1176e68b66a"} Oct 30 13:06:50.309865 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:50.309841 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" event={"ID":"7d034a1b-2482-4f51-b89c-6300d660b7ee","Type":"ContainerStarted","Data":"3bc9462e86526c0163c87edb7331ffec30154f643fe4ba31aebcfad8f7d9a8cf"} Oct 30 13:06:50.309979 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:50.309879 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" event={"ID":"7d034a1b-2482-4f51-b89c-6300d660b7ee","Type":"ContainerStarted","Data":"15ba2a9876f623fe541b1a2e7f02c11138ccc805e70bc631838e1831147cb608"} Oct 30 13:06:50.352440 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:50.352379 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63cdf4c0-px7v6" podStartSLOduration=1.352359673 podStartE2EDuration="1.352359673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:50.349198762 +0000 UTC m=+866.818614556" watchObservedRunningTime="2026-10-30 13:06:50.352359673 +0000 UTC m=+866.821775421" Oct 30 13:06:50.397981 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:50.397886 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" podStartSLOduration=1.397871425 podStartE2EDuration="1.397871425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:50.394801879 +0000 UTC m=+866.864217649" watchObservedRunningTime="2026-10-30 13:06:50.397871425 +0000 UTC m=+866.867287169" Oct 30 13:06:50.403912 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:50.403884 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn"] Oct 30 13:06:50.405743 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:06:50.405713 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc61d2f25_754d_4538_b154_b9095bffe96f.slice/crio-36952ec869368eef6359d65005d5f58fec9c3575ad3471f33cadce7a8827fe1b WatchSource:0}: Error finding container 36952ec869368eef6359d65005d5f58fec9c3575ad3471f33cadce7a8827fe1b: Status 404 returned error can't find the container with id 36952ec869368eef6359d65005d5f58fec9c3575ad3471f33cadce7a8827fe1b Oct 30 13:06:50.415189 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:50.415146 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:06:50.415291 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:50.415236 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:06:50.415291 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:50.415283 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:06:50.694352 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:50.694322 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" Oct 30 13:06:51.325336 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:51.325294 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" event={"ID":"c61d2f25-754d-4538-b154-b9095bffe96f","Type":"ContainerStarted","Data":"2185979abda3a74fd942f13b3aa4f6a5289a25f8a8812fe823a9390e92563364"} Oct 30 13:06:51.325537 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:51.325389 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" event={"ID":"c61d2f25-754d-4538-b154-b9095bffe96f","Type":"ContainerStarted","Data":"36952ec869368eef6359d65005d5f58fec9c3575ad3471f33cadce7a8827fe1b"} Oct 30 13:06:51.327093 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:51.327063 5922 generic.go:356] "Generic (PLEG): container finished" podID="6ed1ede4-062b-4b91-b25c-8f583e120427" containerID="9834914807a62848a58c7ecec261a51ff8e0619cfd1e27a2c7d161bf53eff8be" exitCode=0 Oct 30 13:06:51.327204 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:51.327135 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63cdf4c0-px7v6" event={"ID":"6ed1ede4-062b-4b91-b25c-8f583e120427","Type":"ContainerDied","Data":"9834914807a62848a58c7ecec261a51ff8e0619cfd1e27a2c7d161bf53eff8be"} Oct 30 13:06:51.387807 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:51.387741 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" podStartSLOduration=2.387726377 podStartE2EDuration="2.387726377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:51.386946631 +0000 UTC m=+867.856362429" watchObservedRunningTime="2026-10-30 13:06:51.387726377 +0000 UTC m=+867.857142116" Oct 30 13:06:51.701764 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:51.701723 5922 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 13:06:51.702354 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:51.702317 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" podUID="7d034a1b-2482-4f51-b89c-6300d660b7ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 13:06:52.089063 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:52.086913 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" Oct 30 13:06:52.093637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:52.090706 5922 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 13:06:52.093637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:52.090764 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" podUID="c61d2f25-754d-4538-b154-b9095bffe96f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 13:06:52.317337 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:52.317294 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:52.654455 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:52.654429 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63cdf4c0-px7v6" Oct 30 13:06:52.717689 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:52.707939 5922 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 13:06:52.717689 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:52.707995 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" podUID="7d034a1b-2482-4f51-b89c-6300d660b7ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 13:06:52.730608 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:52.730576 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ed1ede4-062b-4b91-b25c-8f583e120427-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ed1ede4-062b-4b91-b25c-8f583e120427-maas-api-xks-ca\") pod \"6ed1ede4-062b-4b91-b25c-8f583e120427\" (UID: \"6ed1ede4-062b-4b91-b25c-8f583e120427\") " Oct 30 13:06:52.730856 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:52.730835 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ed1ede4-062b-4b91-b25c-8f583e120427-kube-api-access-j6rlm\" (UniqueName: \"kubernetes.io/projected/6ed1ede4-062b-4b91-b25c-8f583e120427-kube-api-access-j6rlm\") pod \"6ed1ede4-062b-4b91-b25c-8f583e120427\" (UID: \"6ed1ede4-062b-4b91-b25c-8f583e120427\") " Oct 30 13:06:52.731186 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:52.731166 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ed1ede4-062b-4b91-b25c-8f583e120427-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ed1ede4-062b-4b91-b25c-8f583e120427-maas-api-openshift-ca\") pod \"6ed1ede4-062b-4b91-b25c-8f583e120427\" (UID: \"6ed1ede4-062b-4b91-b25c-8f583e120427\") " Oct 30 13:06:52.731583 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:52.731558 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ed1ede4-062b-4b91-b25c-8f583e120427-maas-api-openshift-ca" pod "6ed1ede4-062b-4b91-b25c-8f583e120427" (UID: "6ed1ede4-062b-4b91-b25c-8f583e120427"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:52.732517 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:52.732499 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ed1ede4-062b-4b91-b25c-8f583e120427-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:06:52.733272 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:52.733247 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ed1ede4-062b-4b91-b25c-8f583e120427-kube-api-access-j6rlm" pod "6ed1ede4-062b-4b91-b25c-8f583e120427" (UID: "6ed1ede4-062b-4b91-b25c-8f583e120427"). InnerVolumeSpecName "kube-api-access-j6rlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:52.733355 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:52.733247 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ed1ede4-062b-4b91-b25c-8f583e120427-maas-api-xks-ca" pod "6ed1ede4-062b-4b91-b25c-8f583e120427" (UID: "6ed1ede4-062b-4b91-b25c-8f583e120427"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:06:52.834312 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:52.834281 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ed1ede4-062b-4b91-b25c-8f583e120427-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:06:52.834312 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:52.834312 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6rlm\" (UniqueName: \"kubernetes.io/projected/6ed1ede4-062b-4b91-b25c-8f583e120427-kube-api-access-j6rlm\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:06:52.940537 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:52.940505 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3adc8c0-5hkn5"] Oct 30 13:06:52.941310 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:52.941289 5922 cpu_manager.go:414] "containerMap: removing container" podUID="6ed1ede4-062b-4b91-b25c-8f583e120427" containerName="revoke-keys" Oct 30 13:06:52.941419 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:52.941409 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ed1ede4-062b-4b91-b25c-8f583e120427" containerName="revoke-keys" Oct 30 13:06:52.941595 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:52.941586 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ed1ede4-062b-4b91-b25c-8f583e120427" containerName="revoke-keys" Oct 30 13:06:52.945312 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:52.945294 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3adc8c0-5hkn5" Oct 30 13:06:52.985685 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:52.984889 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3adc8c0-5hkn5"] Oct 30 13:06:53.037203 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:53.037161 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3adc8c0-5hkn5\" (UID: \"1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3adc8c0-5hkn5" Oct 30 13:06:53.037203 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:53.037198 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vmlk\" (UniqueName: \"kubernetes.io/projected/1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646-kube-api-access-7vmlk\") pod \"maas-api-revoke-sub-f3adc8c0-5hkn5\" (UID: \"1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3adc8c0-5hkn5" Oct 30 13:06:53.037455 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:53.037246 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3adc8c0-5hkn5\" (UID: \"1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3adc8c0-5hkn5" Oct 30 13:06:53.105637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:53.104942 5922 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 13:06:53.105637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:53.105002 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" podUID="c61d2f25-754d-4538-b154-b9095bffe96f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 13:06:53.138999 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:53.138959 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3adc8c0-5hkn5\" (UID: \"1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3adc8c0-5hkn5" Oct 30 13:06:53.138999 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:53.139003 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vmlk\" (UniqueName: \"kubernetes.io/projected/1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646-kube-api-access-7vmlk\") pod \"maas-api-revoke-sub-f3adc8c0-5hkn5\" (UID: \"1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3adc8c0-5hkn5" Oct 30 13:06:53.139197 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:53.139071 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3adc8c0-5hkn5\" (UID: \"1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3adc8c0-5hkn5" Oct 30 13:06:53.140244 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:53.140215 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3adc8c0-5hkn5\" (UID: \"1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3adc8c0-5hkn5" Oct 30 13:06:53.141928 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:53.141904 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3adc8c0-5hkn5\" (UID: \"1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3adc8c0-5hkn5" Oct 30 13:06:53.211638 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:53.208447 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vmlk\" (UniqueName: \"kubernetes.io/projected/1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646-kube-api-access-7vmlk\") pod \"maas-api-revoke-sub-f3adc8c0-5hkn5\" (UID: \"1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3adc8c0-5hkn5" Oct 30 13:06:53.258652 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:53.258578 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3adc8c0-5hkn5" Oct 30 13:06:53.342595 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:53.342567 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63cdf4c0-px7v6" Oct 30 13:06:53.342758 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:53.342647 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63cdf4c0-px7v6" event={"ID":"6ed1ede4-062b-4b91-b25c-8f583e120427","Type":"ContainerDied","Data":"ee3cfc0d2b9b35386d9c6f9086a39d40fdbbfe8a95a4a36811f0f1176e68b66a"} Oct 30 13:06:53.342758 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:53.342687 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee3cfc0d2b9b35386d9c6f9086a39d40fdbbfe8a95a4a36811f0f1176e68b66a" Oct 30 13:06:53.552889 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:06:53.552808 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:06:53.633769 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:53.633686 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3adc8c0-5hkn5"] Oct 30 13:06:53.638124 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:06:53.638098 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dfb2b3b_76f0_4bcb_b0f9_eb39ae32b646.slice/crio-7a805de6108a0a14d9d291fb738b3d94984ec55e9be9a3e5e1787b5ecbf14a2b WatchSource:0}: Error finding container 7a805de6108a0a14d9d291fb738b3d94984ec55e9be9a3e5e1787b5ecbf14a2b: Status 404 returned error can't find the container with id 7a805de6108a0a14d9d291fb738b3d94984ec55e9be9a3e5e1787b5ecbf14a2b Oct 30 13:06:53.697646 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:53.696988 5922 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 13:06:53.697646 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:53.697047 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" podUID="7d034a1b-2482-4f51-b89c-6300d660b7ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 13:06:54.113641 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:54.103952 5922 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 13:06:54.113641 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:54.104009 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" podUID="c61d2f25-754d-4538-b154-b9095bffe96f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 13:06:54.182683 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:54.181151 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:54.357183 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:54.357139 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3adc8c0-5hkn5" event={"ID":"1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646","Type":"ContainerStarted","Data":"afd8d18a813e42506379cef3857a06d5e147b9dbcc79c40a25c918ca279828dd"} Oct 30 13:06:54.357372 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:54.357242 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3adc8c0-5hkn5" event={"ID":"1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646","Type":"ContainerStarted","Data":"7a805de6108a0a14d9d291fb738b3d94984ec55e9be9a3e5e1787b5ecbf14a2b"} Oct 30 13:06:54.412636 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:54.412283 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3adc8c0-5hkn5" podStartSLOduration=2.412261338 podStartE2EDuration="2.412261338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:54.40200597 +0000 UTC m=+870.871421745" watchObservedRunningTime="2026-10-30 13:06:54.412261338 +0000 UTC m=+870.881677076" Oct 30 13:06:54.697637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:54.694963 5922 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 13:06:54.697637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:54.695036 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" podUID="7d034a1b-2482-4f51-b89c-6300d660b7ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 13:06:55.096695 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:55.086711 5922 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 13:06:55.096695 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:55.086782 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" podUID="c61d2f25-754d-4538-b154-b9095bffe96f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 13:06:55.373923 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:55.373833 5922 generic.go:356] "Generic (PLEG): container finished" podID="1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646" containerID="afd8d18a813e42506379cef3857a06d5e147b9dbcc79c40a25c918ca279828dd" exitCode=0 Oct 30 13:06:55.374565 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:55.374538 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3adc8c0-5hkn5" event={"ID":"1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646","Type":"ContainerDied","Data":"afd8d18a813e42506379cef3857a06d5e147b9dbcc79c40a25c918ca279828dd"} Oct 30 13:06:55.726093 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:55.725962 5922 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 13:06:55.726914 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:55.726882 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" podUID="7d034a1b-2482-4f51-b89c-6300d660b7ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 13:06:56.089690 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:56.086216 5922 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 13:06:56.089690 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:56.086308 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" podUID="c61d2f25-754d-4538-b154-b9095bffe96f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 13:06:56.130733 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:56.130671 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:56.658050 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:56.657986 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3adc8c0-5hkn5" Oct 30 13:06:56.697396 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:56.697355 5922 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 13:06:56.698112 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:56.697672 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" podUID="7d034a1b-2482-4f51-b89c-6300d660b7ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 13:06:56.758967 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:56.755718 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646-maas-api-xks-ca\") pod \"1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646\" (UID: \"1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646\") " Oct 30 13:06:56.759438 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:56.759392 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646-maas-api-openshift-ca\") pod \"1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646\" (UID: \"1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646\") " Oct 30 13:06:56.760230 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:56.760193 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646-kube-api-access-7vmlk\" (UniqueName: \"kubernetes.io/projected/1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646-kube-api-access-7vmlk\") pod \"1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646\" (UID: \"1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646\") " Oct 30 13:06:56.797910 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:56.775826 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646-maas-api-openshift-ca" pod "1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646" (UID: "1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:56.866857 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:56.862538 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:06:56.869271 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:56.867203 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646-kube-api-access-7vmlk" pod "1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646" (UID: "1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646"). InnerVolumeSpecName "kube-api-access-7vmlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:56.928695 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:56.927768 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646-maas-api-xks-ca" pod "1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646" (UID: "1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:06:56.987466 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:56.987434 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vmlk\" (UniqueName: \"kubernetes.io/projected/1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646-kube-api-access-7vmlk\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:06:56.992744 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:56.992722 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:06:57.086657 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:57.085315 5922 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 13:06:57.086657 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:57.085386 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" podUID="c61d2f25-754d-4538-b154-b9095bffe96f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 13:06:57.445633 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:57.442333 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3adc8c0-5hkn5" event={"ID":"1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646","Type":"ContainerDied","Data":"7a805de6108a0a14d9d291fb738b3d94984ec55e9be9a3e5e1787b5ecbf14a2b"} Oct 30 13:06:57.445633 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:57.442445 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a805de6108a0a14d9d291fb738b3d94984ec55e9be9a3e5e1787b5ecbf14a2b" Oct 30 13:06:57.445633 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:57.442516 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3adc8c0-5hkn5" Oct 30 13:06:58.087910 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:58.087871 5922 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 13:06:58.094739 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:58.094706 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" podUID="c61d2f25-754d-4538-b154-b9095bffe96f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 13:06:58.692637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:58.689862 5922 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:06:58.692637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:58.689922 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" podUID="d0132701-3b77-4cce-9989-a0d03ac479f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:06:58.722643 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:58.718684 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" podUID="b487a069-f519-45a3-b1d8-f32f2431797b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:06:58.834730 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:58.834687 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" podUID="5f353928-8cb4-49a3-b999-1b90e6cb31bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:06:58.887674 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:58.887534 5922 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:06:58.887674 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:58.887600 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" podUID="7d034a1b-2482-4f51-b89c-6300d660b7ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:06:58.923830 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:58.923789 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" podUID="a85de36b-f8c8-4ade-a468-c114080cb312" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:06:58.947787 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:58.947682 5922 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" podUID="5f353928-8cb4-49a3-b999-1b90e6cb31bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:06:58.967486 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:58.967444 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" podUID="47517373-759d-4eeb-ab5e-6150b7e82bb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:06:59.041749 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:59.041701 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" podUID="dfd83cc8-6503-45a8-8260-c0edb8752471" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:06:59.163637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:59.152296 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:59.163637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:59.152388 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" podUID="dad5bdad-cd6f-41f1-a124-540c3524ead7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:06:59.536639 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:59.532961 5922 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:06:59.536639 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:59.533025 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" podUID="b607f9a7-6ef2-4ea2-bdb4-352b2312966d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:06:59.638635 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:59.637690 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" podUID="c0ea7fde-2ac5-432d-bb9b-534024186ac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:06:59.725637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:59.720089 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" podUID="b487a069-f519-45a3-b1d8-f32f2431797b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:06:59.725637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:59.720826 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" Oct 30 13:06:59.798689 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:59.797840 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" podUID="35619ab9-cbeb-45da-8506-3af63a273177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:06:59.841634 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:59.838683 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" podUID="5f353928-8cb4-49a3-b999-1b90e6cb31bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 13:06:59.841634 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:59.839076 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" Oct 30 13:06:59.872124 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:59.871131 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" podUID="14ae0aea-473f-4ca0-bcc0-f617b9839c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded" Oct 30 13:06:59.888715 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:59.879946 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" Oct 30 13:06:59.907084 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:59.905400 5922 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:06:59.907084 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:59.905460 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" podUID="7d034a1b-2482-4f51-b89c-6300d660b7ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:06:59.944102 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:59.926472 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" podUID="a85de36b-f8c8-4ade-a468-c114080cb312" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:06:59.944102 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:59.926893 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" Oct 30 13:07:00.006631 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:59.997685 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" podUID="47517373-759d-4eeb-ab5e-6150b7e82bb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:00.006631 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:06:59.998108 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" Oct 30 13:07:00.006631 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:00.001742 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" Oct 30 13:07:00.047501 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:00.045599 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" Oct 30 13:07:00.047501 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:00.046421 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" podUID="dfd83cc8-6503-45a8-8260-c0edb8752471" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:00.047501 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:00.046746 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" Oct 30 13:07:00.094771 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:00.094691 5922 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:07:00.095010 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:00.094982 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" podUID="c61d2f25-754d-4538-b154-b9095bffe96f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:00.154058 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:00.152727 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" Oct 30 13:07:00.176745 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:00.176715 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" Oct 30 13:07:00.946753 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:00.946718 5922 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:07:00.946959 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:00.946771 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" podUID="7d034a1b-2482-4f51-b89c-6300d660b7ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:01.135756 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:01.135711 5922 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:07:01.135936 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:01.135766 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" podUID="c61d2f25-754d-4538-b154-b9095bffe96f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:01.155976 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:01.155946 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" Oct 30 13:07:01.156849 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:01.156821 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" Oct 30 13:07:01.158065 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:01.157900 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" Oct 30 13:07:01.441296 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:01.441189 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" Oct 30 13:07:01.441834 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:01.441788 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" Oct 30 13:07:01.443052 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:01.443032 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" Oct 30 13:07:01.724012 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:01.723984 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:01.876069 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:01.876032 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0b5e49f-bn8ws"] Oct 30 13:07:01.876636 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:01.876607 5922 cpu_manager.go:414] "containerMap: removing container" podUID="1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646" containerName="revoke-keys" Oct 30 13:07:01.876678 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:01.876637 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646" containerName="revoke-keys" Oct 30 13:07:01.876790 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:01.876780 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646" containerName="revoke-keys" Oct 30 13:07:01.880382 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:01.880359 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b5e49f-bn8ws" Oct 30 13:07:01.896726 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:01.896699 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:07:01.896898 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:01.896848 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:07:01.897122 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:01.897102 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:07:01.897292 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:01.897267 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:07:01.900176 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:01.900153 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0b5e49f-bn8ws"] Oct 30 13:07:01.976658 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:01.976562 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cab4ff5-dfd0-4a13-b88c-f895992dd9c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0b5e49f-bn8ws\" (UID: \"3cab4ff5-dfd0-4a13-b88c-f895992dd9c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b5e49f-bn8ws" Oct 30 13:07:01.976803 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:01.976738 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cab4ff5-dfd0-4a13-b88c-f895992dd9c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0b5e49f-bn8ws\" (UID: \"3cab4ff5-dfd0-4a13-b88c-f895992dd9c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b5e49f-bn8ws" Oct 30 13:07:01.976864 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:01.976806 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzkwc\" (UniqueName: \"kubernetes.io/projected/3cab4ff5-dfd0-4a13-b88c-f895992dd9c2-kube-api-access-fzkwc\") pod \"maas-api-revoke-sub-d0b5e49f-bn8ws\" (UID: \"3cab4ff5-dfd0-4a13-b88c-f895992dd9c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b5e49f-bn8ws" Oct 30 13:07:02.077784 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:02.077751 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzkwc\" (UniqueName: \"kubernetes.io/projected/3cab4ff5-dfd0-4a13-b88c-f895992dd9c2-kube-api-access-fzkwc\") pod \"maas-api-revoke-sub-d0b5e49f-bn8ws\" (UID: \"3cab4ff5-dfd0-4a13-b88c-f895992dd9c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b5e49f-bn8ws" Oct 30 13:07:02.077939 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:02.077841 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cab4ff5-dfd0-4a13-b88c-f895992dd9c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0b5e49f-bn8ws\" (UID: \"3cab4ff5-dfd0-4a13-b88c-f895992dd9c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b5e49f-bn8ws" Oct 30 13:07:02.077995 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:02.077963 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cab4ff5-dfd0-4a13-b88c-f895992dd9c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0b5e49f-bn8ws\" (UID: \"3cab4ff5-dfd0-4a13-b88c-f895992dd9c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b5e49f-bn8ws" Oct 30 13:07:02.078602 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:02.078579 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cab4ff5-dfd0-4a13-b88c-f895992dd9c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0b5e49f-bn8ws\" (UID: \"3cab4ff5-dfd0-4a13-b88c-f895992dd9c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b5e49f-bn8ws" Oct 30 13:07:02.080540 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:02.080517 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cab4ff5-dfd0-4a13-b88c-f895992dd9c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0b5e49f-bn8ws\" (UID: \"3cab4ff5-dfd0-4a13-b88c-f895992dd9c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b5e49f-bn8ws" Oct 30 13:07:02.105280 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:02.105258 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzkwc\" (UniqueName: \"kubernetes.io/projected/3cab4ff5-dfd0-4a13-b88c-f895992dd9c2-kube-api-access-fzkwc\") pod \"maas-api-revoke-sub-d0b5e49f-bn8ws\" (UID: \"3cab4ff5-dfd0-4a13-b88c-f895992dd9c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b5e49f-bn8ws" Oct 30 13:07:02.194074 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:02.194040 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b5e49f-bn8ws" Oct 30 13:07:02.497434 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:02.497406 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0b5e49f-bn8ws"] Oct 30 13:07:02.498836 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:07:02.498806 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cab4ff5_dfd0_4a13_b88c_f895992dd9c2.slice/crio-8c2cf1b18e3181f887a64d5910d73755f24a01423962f61e7e57459209b7e1c3 WatchSource:0}: Error finding container 8c2cf1b18e3181f887a64d5910d73755f24a01423962f61e7e57459209b7e1c3: Status 404 returned error can't find the container with id 8c2cf1b18e3181f887a64d5910d73755f24a01423962f61e7e57459209b7e1c3 Oct 30 13:07:02.611746 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:02.611711 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b5e49f-bn8ws" event={"ID":"3cab4ff5-dfd0-4a13-b88c-f895992dd9c2","Type":"ContainerStarted","Data":"8c2cf1b18e3181f887a64d5910d73755f24a01423962f61e7e57459209b7e1c3"} Oct 30 13:07:03.588519 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:03.588441 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:03.621705 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:03.621673 5922 generic.go:356] "Generic (PLEG): container finished" podID="3cab4ff5-dfd0-4a13-b88c-f895992dd9c2" containerID="a2b6a02a9de636e02e42af6069b9525e97760c746e4dc9c188d09a08e2b7e4c9" exitCode=0 Oct 30 13:07:03.621845 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:03.621709 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b5e49f-bn8ws" event={"ID":"3cab4ff5-dfd0-4a13-b88c-f895992dd9c2","Type":"ContainerDied","Data":"a2b6a02a9de636e02e42af6069b9525e97760c746e4dc9c188d09a08e2b7e4c9"} Oct 30 13:07:03.713051 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:03.713015 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e7b6e1d-shd4w"] Oct 30 13:07:03.718268 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:03.718241 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7b6e1d-shd4w" Oct 30 13:07:03.735424 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:03.735397 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e7b6e1d-shd4w"] Oct 30 13:07:03.803257 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:03.803221 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw9fn\" (UniqueName: \"kubernetes.io/projected/a3547727-fc7f-4d93-8d50-0a21fd349aa6-kube-api-access-dw9fn\") pod \"maas-api-revoke-sub-0e7b6e1d-shd4w\" (UID: \"a3547727-fc7f-4d93-8d50-0a21fd349aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7b6e1d-shd4w" Oct 30 13:07:03.803443 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:03.803267 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3547727-fc7f-4d93-8d50-0a21fd349aa6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e7b6e1d-shd4w\" (UID: \"a3547727-fc7f-4d93-8d50-0a21fd349aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7b6e1d-shd4w" Oct 30 13:07:03.803443 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:03.803343 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3547727-fc7f-4d93-8d50-0a21fd349aa6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e7b6e1d-shd4w\" (UID: \"a3547727-fc7f-4d93-8d50-0a21fd349aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7b6e1d-shd4w" Oct 30 13:07:03.904383 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:03.904313 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw9fn\" (UniqueName: \"kubernetes.io/projected/a3547727-fc7f-4d93-8d50-0a21fd349aa6-kube-api-access-dw9fn\") pod \"maas-api-revoke-sub-0e7b6e1d-shd4w\" (UID: \"a3547727-fc7f-4d93-8d50-0a21fd349aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7b6e1d-shd4w" Oct 30 13:07:03.904383 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:03.904345 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3547727-fc7f-4d93-8d50-0a21fd349aa6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e7b6e1d-shd4w\" (UID: \"a3547727-fc7f-4d93-8d50-0a21fd349aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7b6e1d-shd4w" Oct 30 13:07:03.904383 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:03.904366 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3547727-fc7f-4d93-8d50-0a21fd349aa6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e7b6e1d-shd4w\" (UID: \"a3547727-fc7f-4d93-8d50-0a21fd349aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7b6e1d-shd4w" Oct 30 13:07:03.905060 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:03.905039 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3547727-fc7f-4d93-8d50-0a21fd349aa6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e7b6e1d-shd4w\" (UID: \"a3547727-fc7f-4d93-8d50-0a21fd349aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7b6e1d-shd4w" Oct 30 13:07:03.906647 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:03.906601 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3547727-fc7f-4d93-8d50-0a21fd349aa6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e7b6e1d-shd4w\" (UID: \"a3547727-fc7f-4d93-8d50-0a21fd349aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7b6e1d-shd4w" Oct 30 13:07:03.929025 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:03.929006 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw9fn\" (UniqueName: \"kubernetes.io/projected/a3547727-fc7f-4d93-8d50-0a21fd349aa6-kube-api-access-dw9fn\") pod \"maas-api-revoke-sub-0e7b6e1d-shd4w\" (UID: \"a3547727-fc7f-4d93-8d50-0a21fd349aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7b6e1d-shd4w" Oct 30 13:07:04.033294 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:04.033269 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7b6e1d-shd4w" Oct 30 13:07:04.339939 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:07:04.339907 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3547727_fc7f_4d93_8d50_0a21fd349aa6.slice/crio-0d8da105f72fa105397702119e2d3628005af34049d6b9ec76ffcb3cdccabbbb WatchSource:0}: Error finding container 0d8da105f72fa105397702119e2d3628005af34049d6b9ec76ffcb3cdccabbbb: Status 404 returned error can't find the container with id 0d8da105f72fa105397702119e2d3628005af34049d6b9ec76ffcb3cdccabbbb Oct 30 13:07:04.340575 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:04.340552 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e7b6e1d-shd4w"] Oct 30 13:07:04.630367 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:04.630300 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7b6e1d-shd4w" event={"ID":"a3547727-fc7f-4d93-8d50-0a21fd349aa6","Type":"ContainerStarted","Data":"0d8da105f72fa105397702119e2d3628005af34049d6b9ec76ffcb3cdccabbbb"} Oct 30 13:07:04.796958 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:04.796926 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b5e49f-bn8ws" Oct 30 13:07:04.917038 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:04.916964 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cab4ff5-dfd0-4a13-b88c-f895992dd9c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cab4ff5-dfd0-4a13-b88c-f895992dd9c2-maas-api-openshift-ca\") pod \"3cab4ff5-dfd0-4a13-b88c-f895992dd9c2\" (UID: \"3cab4ff5-dfd0-4a13-b88c-f895992dd9c2\") " Oct 30 13:07:04.917201 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:04.917050 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cab4ff5-dfd0-4a13-b88c-f895992dd9c2-kube-api-access-fzkwc\" (UniqueName: \"kubernetes.io/projected/3cab4ff5-dfd0-4a13-b88c-f895992dd9c2-kube-api-access-fzkwc\") pod \"3cab4ff5-dfd0-4a13-b88c-f895992dd9c2\" (UID: \"3cab4ff5-dfd0-4a13-b88c-f895992dd9c2\") " Oct 30 13:07:04.917201 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:04.917174 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cab4ff5-dfd0-4a13-b88c-f895992dd9c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cab4ff5-dfd0-4a13-b88c-f895992dd9c2-maas-api-xks-ca\") pod \"3cab4ff5-dfd0-4a13-b88c-f895992dd9c2\" (UID: \"3cab4ff5-dfd0-4a13-b88c-f895992dd9c2\") " Oct 30 13:07:04.917366 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:04.917341 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cab4ff5-dfd0-4a13-b88c-f895992dd9c2-maas-api-openshift-ca" pod "3cab4ff5-dfd0-4a13-b88c-f895992dd9c2" (UID: "3cab4ff5-dfd0-4a13-b88c-f895992dd9c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:04.917698 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:04.917673 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cab4ff5-dfd0-4a13-b88c-f895992dd9c2-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:07:04.921795 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:04.921767 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cab4ff5-dfd0-4a13-b88c-f895992dd9c2-maas-api-xks-ca" pod "3cab4ff5-dfd0-4a13-b88c-f895992dd9c2" (UID: "3cab4ff5-dfd0-4a13-b88c-f895992dd9c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:04.922636 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:04.922539 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cab4ff5-dfd0-4a13-b88c-f895992dd9c2-kube-api-access-fzkwc" pod "3cab4ff5-dfd0-4a13-b88c-f895992dd9c2" (UID: "3cab4ff5-dfd0-4a13-b88c-f895992dd9c2"). InnerVolumeSpecName "kube-api-access-fzkwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:05.018895 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:05.018868 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzkwc\" (UniqueName: \"kubernetes.io/projected/3cab4ff5-dfd0-4a13-b88c-f895992dd9c2-kube-api-access-fzkwc\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:07:05.018895 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:05.018891 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cab4ff5-dfd0-4a13-b88c-f895992dd9c2-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:07:05.418993 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:05.418964 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:05.645077 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:05.644981 5922 generic.go:356] "Generic (PLEG): container finished" podID="a3547727-fc7f-4d93-8d50-0a21fd349aa6" containerID="05f3b66efdb1445b8c49c480c917130f001150a198662694f29edd2f8c234823" exitCode=0 Oct 30 13:07:05.646680 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:05.646653 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b5e49f-bn8ws" Oct 30 13:07:05.647106 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:05.647080 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7b6e1d-shd4w" event={"ID":"a3547727-fc7f-4d93-8d50-0a21fd349aa6","Type":"ContainerDied","Data":"05f3b66efdb1445b8c49c480c917130f001150a198662694f29edd2f8c234823"} Oct 30 13:07:05.647151 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:05.647135 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b5e49f-bn8ws" event={"ID":"3cab4ff5-dfd0-4a13-b88c-f895992dd9c2","Type":"ContainerDied","Data":"8c2cf1b18e3181f887a64d5910d73755f24a01423962f61e7e57459209b7e1c3"} Oct 30 13:07:05.647186 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:05.647160 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c2cf1b18e3181f887a64d5910d73755f24a01423962f61e7e57459209b7e1c3" Oct 30 13:07:06.795902 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:06.795876 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7b6e1d-shd4w" Oct 30 13:07:06.942964 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:06.942931 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3547727-fc7f-4d93-8d50-0a21fd349aa6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3547727-fc7f-4d93-8d50-0a21fd349aa6-maas-api-xks-ca\") pod \"a3547727-fc7f-4d93-8d50-0a21fd349aa6\" (UID: \"a3547727-fc7f-4d93-8d50-0a21fd349aa6\") " Oct 30 13:07:06.943157 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:06.942992 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3547727-fc7f-4d93-8d50-0a21fd349aa6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3547727-fc7f-4d93-8d50-0a21fd349aa6-maas-api-openshift-ca\") pod \"a3547727-fc7f-4d93-8d50-0a21fd349aa6\" (UID: \"a3547727-fc7f-4d93-8d50-0a21fd349aa6\") " Oct 30 13:07:06.943224 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:06.943172 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3547727-fc7f-4d93-8d50-0a21fd349aa6-kube-api-access-dw9fn\" (UniqueName: \"kubernetes.io/projected/a3547727-fc7f-4d93-8d50-0a21fd349aa6-kube-api-access-dw9fn\") pod \"a3547727-fc7f-4d93-8d50-0a21fd349aa6\" (UID: \"a3547727-fc7f-4d93-8d50-0a21fd349aa6\") " Oct 30 13:07:06.943384 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:06.943356 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3547727-fc7f-4d93-8d50-0a21fd349aa6-maas-api-openshift-ca" pod "a3547727-fc7f-4d93-8d50-0a21fd349aa6" (UID: "a3547727-fc7f-4d93-8d50-0a21fd349aa6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:06.943649 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:06.943632 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3547727-fc7f-4d93-8d50-0a21fd349aa6-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:07:06.945030 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:06.944998 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3547727-fc7f-4d93-8d50-0a21fd349aa6-maas-api-xks-ca" pod "a3547727-fc7f-4d93-8d50-0a21fd349aa6" (UID: "a3547727-fc7f-4d93-8d50-0a21fd349aa6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:06.945184 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:06.945163 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3547727-fc7f-4d93-8d50-0a21fd349aa6-kube-api-access-dw9fn" pod "a3547727-fc7f-4d93-8d50-0a21fd349aa6" (UID: "a3547727-fc7f-4d93-8d50-0a21fd349aa6"). InnerVolumeSpecName "kube-api-access-dw9fn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:07.044394 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:07.044360 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw9fn\" (UniqueName: \"kubernetes.io/projected/a3547727-fc7f-4d93-8d50-0a21fd349aa6-kube-api-access-dw9fn\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:07:07.044394 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:07.044388 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3547727-fc7f-4d93-8d50-0a21fd349aa6-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:07:07.189496 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:07.189457 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:07.661419 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:07.661391 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7b6e1d-shd4w" Oct 30 13:07:07.661575 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:07.661432 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7b6e1d-shd4w" event={"ID":"a3547727-fc7f-4d93-8d50-0a21fd349aa6","Type":"ContainerDied","Data":"0d8da105f72fa105397702119e2d3628005af34049d6b9ec76ffcb3cdccabbbb"} Oct 30 13:07:07.661575 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:07.661473 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d8da105f72fa105397702119e2d3628005af34049d6b9ec76ffcb3cdccabbbb" Oct 30 13:07:09.110406 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:09.110375 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:10.886109 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:10.886070 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:12.591582 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:12.591545 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:14.083519 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:14.083491 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:07:14.388766 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:14.388688 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:16.576635 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:16.576144 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:17.838859 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:17.838827 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c763adca-hn8ln"] Oct 30 13:07:17.839395 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:17.839382 5922 cpu_manager.go:414] "containerMap: removing container" podUID="3cab4ff5-dfd0-4a13-b88c-f895992dd9c2" containerName="revoke-keys" Oct 30 13:07:17.839454 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:17.839396 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cab4ff5-dfd0-4a13-b88c-f895992dd9c2" containerName="revoke-keys" Oct 30 13:07:17.839454 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:17.839432 5922 cpu_manager.go:414] "containerMap: removing container" podUID="a3547727-fc7f-4d93-8d50-0a21fd349aa6" containerName="revoke-keys" Oct 30 13:07:17.839454 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:17.839437 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3547727-fc7f-4d93-8d50-0a21fd349aa6" containerName="revoke-keys" Oct 30 13:07:17.839546 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:17.839516 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3547727-fc7f-4d93-8d50-0a21fd349aa6" containerName="revoke-keys" Oct 30 13:07:17.839546 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:17.839524 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cab4ff5-dfd0-4a13-b88c-f895992dd9c2" containerName="revoke-keys" Oct 30 13:07:17.843149 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:17.843133 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c763adca-hn8ln" Oct 30 13:07:17.862985 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:17.862958 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c763adca-hn8ln"] Oct 30 13:07:17.870247 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:17.870206 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:07:17.870247 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:17.870243 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:07:17.870449 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:17.870343 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:07:17.870641 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:17.870605 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:07:18.005206 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:18.005181 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjpj5\" (UniqueName: \"kubernetes.io/projected/967aa55f-1503-41bd-8089-a909532dd160-kube-api-access-wjpj5\") pod \"maas-api-revoke-sub-c763adca-hn8ln\" (UID: \"967aa55f-1503-41bd-8089-a909532dd160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c763adca-hn8ln" Oct 30 13:07:18.005402 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:18.005379 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/967aa55f-1503-41bd-8089-a909532dd160-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c763adca-hn8ln\" (UID: \"967aa55f-1503-41bd-8089-a909532dd160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c763adca-hn8ln" Oct 30 13:07:18.005555 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:18.005538 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/967aa55f-1503-41bd-8089-a909532dd160-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c763adca-hn8ln\" (UID: \"967aa55f-1503-41bd-8089-a909532dd160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c763adca-hn8ln" Oct 30 13:07:18.106325 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:18.106259 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/967aa55f-1503-41bd-8089-a909532dd160-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c763adca-hn8ln\" (UID: \"967aa55f-1503-41bd-8089-a909532dd160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c763adca-hn8ln" Oct 30 13:07:18.106429 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:18.106340 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/967aa55f-1503-41bd-8089-a909532dd160-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c763adca-hn8ln\" (UID: \"967aa55f-1503-41bd-8089-a909532dd160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c763adca-hn8ln" Oct 30 13:07:18.106429 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:18.106369 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjpj5\" (UniqueName: \"kubernetes.io/projected/967aa55f-1503-41bd-8089-a909532dd160-kube-api-access-wjpj5\") pod \"maas-api-revoke-sub-c763adca-hn8ln\" (UID: \"967aa55f-1503-41bd-8089-a909532dd160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c763adca-hn8ln" Oct 30 13:07:18.106880 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:18.106858 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/967aa55f-1503-41bd-8089-a909532dd160-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c763adca-hn8ln\" (UID: \"967aa55f-1503-41bd-8089-a909532dd160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c763adca-hn8ln" Oct 30 13:07:18.108600 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:18.108582 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/967aa55f-1503-41bd-8089-a909532dd160-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c763adca-hn8ln\" (UID: \"967aa55f-1503-41bd-8089-a909532dd160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c763adca-hn8ln" Oct 30 13:07:18.131127 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:18.131099 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjpj5\" (UniqueName: \"kubernetes.io/projected/967aa55f-1503-41bd-8089-a909532dd160-kube-api-access-wjpj5\") pod \"maas-api-revoke-sub-c763adca-hn8ln\" (UID: \"967aa55f-1503-41bd-8089-a909532dd160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c763adca-hn8ln" Oct 30 13:07:18.157132 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:18.157113 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c763adca-hn8ln" Oct 30 13:07:18.322683 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:18.322644 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:18.458324 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:18.458299 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c763adca-hn8ln"] Oct 30 13:07:18.460409 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:07:18.460373 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod967aa55f_1503_41bd_8089_a909532dd160.slice/crio-22c3037b7e1951a6fba848460e29380c8dd585974d35cdacc15b756da8f20619 WatchSource:0}: Error finding container 22c3037b7e1951a6fba848460e29380c8dd585974d35cdacc15b756da8f20619: Status 404 returned error can't find the container with id 22c3037b7e1951a6fba848460e29380c8dd585974d35cdacc15b756da8f20619 Oct 30 13:07:18.760580 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:18.760504 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c763adca-hn8ln" event={"ID":"967aa55f-1503-41bd-8089-a909532dd160","Type":"ContainerStarted","Data":"22c3037b7e1951a6fba848460e29380c8dd585974d35cdacc15b756da8f20619"} Oct 30 13:07:19.767930 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:19.767901 5922 generic.go:356] "Generic (PLEG): container finished" podID="967aa55f-1503-41bd-8089-a909532dd160" containerID="6da806f8a77b0e89131b12ece6fe4d5faff08b0c0afe4df77527b4623e77bb07" exitCode=0 Oct 30 13:07:19.768289 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:19.767986 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c763adca-hn8ln" event={"ID":"967aa55f-1503-41bd-8089-a909532dd160","Type":"ContainerDied","Data":"6da806f8a77b0e89131b12ece6fe4d5faff08b0c0afe4df77527b4623e77bb07"} Oct 30 13:07:19.882844 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:19.882812 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf"] Oct 30 13:07:19.887219 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:19.887193 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" Oct 30 13:07:19.912716 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:19.912688 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf"] Oct 30 13:07:20.029535 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.029456 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee561c5-3aa7-4021-9d3e-05e7636d19be-credential-socket\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf\" (UID: \"aee561c5-3aa7-4021-9d3e-05e7636d19be\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" Oct 30 13:07:20.029700 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.029587 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aee561c5-3aa7-4021-9d3e-05e7636d19be-istio-token\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf\" (UID: \"aee561c5-3aa7-4021-9d3e-05e7636d19be\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" Oct 30 13:07:20.029700 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.029633 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aee561c5-3aa7-4021-9d3e-05e7636d19be-workload-certs\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf\" (UID: \"aee561c5-3aa7-4021-9d3e-05e7636d19be\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" Oct 30 13:07:20.029700 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.029659 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee561c5-3aa7-4021-9d3e-05e7636d19be-workload-socket\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf\" (UID: \"aee561c5-3aa7-4021-9d3e-05e7636d19be\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" Oct 30 13:07:20.029700 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.029678 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aee561c5-3aa7-4021-9d3e-05e7636d19be-istio-envoy\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf\" (UID: \"aee561c5-3aa7-4021-9d3e-05e7636d19be\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" Oct 30 13:07:20.029700 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.029699 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aee561c5-3aa7-4021-9d3e-05e7636d19be-istio-data\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf\" (UID: \"aee561c5-3aa7-4021-9d3e-05e7636d19be\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" Oct 30 13:07:20.029901 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.029756 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aee561c5-3aa7-4021-9d3e-05e7636d19be-istio-podinfo\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf\" (UID: \"aee561c5-3aa7-4021-9d3e-05e7636d19be\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" Oct 30 13:07:20.029901 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.029828 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aee561c5-3aa7-4021-9d3e-05e7636d19be-istiod-ca-cert\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf\" (UID: \"aee561c5-3aa7-4021-9d3e-05e7636d19be\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" Oct 30 13:07:20.029994 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.029975 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sm57\" (UniqueName: \"kubernetes.io/projected/aee561c5-3aa7-4021-9d3e-05e7636d19be-kube-api-access-8sm57\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf\" (UID: \"aee561c5-3aa7-4021-9d3e-05e7636d19be\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" Oct 30 13:07:20.089878 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.089834 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:20.131284 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.131253 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee561c5-3aa7-4021-9d3e-05e7636d19be-credential-socket\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf\" (UID: \"aee561c5-3aa7-4021-9d3e-05e7636d19be\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" Oct 30 13:07:20.131457 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.131396 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aee561c5-3aa7-4021-9d3e-05e7636d19be-istio-token\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf\" (UID: \"aee561c5-3aa7-4021-9d3e-05e7636d19be\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" Oct 30 13:07:20.131457 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.131433 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aee561c5-3aa7-4021-9d3e-05e7636d19be-workload-certs\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf\" (UID: \"aee561c5-3aa7-4021-9d3e-05e7636d19be\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" Oct 30 13:07:20.131581 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.131470 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee561c5-3aa7-4021-9d3e-05e7636d19be-workload-socket\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf\" (UID: \"aee561c5-3aa7-4021-9d3e-05e7636d19be\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" Oct 30 13:07:20.131581 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.131494 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aee561c5-3aa7-4021-9d3e-05e7636d19be-istio-envoy\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf\" (UID: \"aee561c5-3aa7-4021-9d3e-05e7636d19be\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" Oct 30 13:07:20.131581 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.131517 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aee561c5-3aa7-4021-9d3e-05e7636d19be-istio-data\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf\" (UID: \"aee561c5-3aa7-4021-9d3e-05e7636d19be\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" Oct 30 13:07:20.131581 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.131536 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aee561c5-3aa7-4021-9d3e-05e7636d19be-istio-podinfo\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf\" (UID: \"aee561c5-3aa7-4021-9d3e-05e7636d19be\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" Oct 30 13:07:20.131581 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.131562 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aee561c5-3aa7-4021-9d3e-05e7636d19be-istiod-ca-cert\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf\" (UID: \"aee561c5-3aa7-4021-9d3e-05e7636d19be\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" Oct 30 13:07:20.131827 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.131607 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sm57\" (UniqueName: \"kubernetes.io/projected/aee561c5-3aa7-4021-9d3e-05e7636d19be-kube-api-access-8sm57\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf\" (UID: \"aee561c5-3aa7-4021-9d3e-05e7636d19be\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" Oct 30 13:07:20.132465 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.132438 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee561c5-3aa7-4021-9d3e-05e7636d19be-credential-socket\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf\" (UID: \"aee561c5-3aa7-4021-9d3e-05e7636d19be\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" Oct 30 13:07:20.132715 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.132479 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee561c5-3aa7-4021-9d3e-05e7636d19be-workload-socket\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf\" (UID: \"aee561c5-3aa7-4021-9d3e-05e7636d19be\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" Oct 30 13:07:20.132849 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.132835 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aee561c5-3aa7-4021-9d3e-05e7636d19be-workload-certs\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf\" (UID: \"aee561c5-3aa7-4021-9d3e-05e7636d19be\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" Oct 30 13:07:20.133164 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.133140 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aee561c5-3aa7-4021-9d3e-05e7636d19be-istio-data\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf\" (UID: \"aee561c5-3aa7-4021-9d3e-05e7636d19be\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" Oct 30 13:07:20.133251 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.133176 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aee561c5-3aa7-4021-9d3e-05e7636d19be-istiod-ca-cert\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf\" (UID: \"aee561c5-3aa7-4021-9d3e-05e7636d19be\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" Oct 30 13:07:20.134732 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.134708 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aee561c5-3aa7-4021-9d3e-05e7636d19be-istio-envoy\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf\" (UID: \"aee561c5-3aa7-4021-9d3e-05e7636d19be\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" Oct 30 13:07:20.134841 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.134816 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aee561c5-3aa7-4021-9d3e-05e7636d19be-istio-podinfo\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf\" (UID: \"aee561c5-3aa7-4021-9d3e-05e7636d19be\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" Oct 30 13:07:20.157706 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.157672 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aee561c5-3aa7-4021-9d3e-05e7636d19be-istio-token\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf\" (UID: \"aee561c5-3aa7-4021-9d3e-05e7636d19be\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" Oct 30 13:07:20.160787 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.160756 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sm57\" (UniqueName: \"kubernetes.io/projected/aee561c5-3aa7-4021-9d3e-05e7636d19be-kube-api-access-8sm57\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf\" (UID: \"aee561c5-3aa7-4021-9d3e-05e7636d19be\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" Oct 30 13:07:20.203030 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.203002 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" Oct 30 13:07:20.396337 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.396262 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24"] Oct 30 13:07:20.400738 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.400721 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" Oct 30 13:07:20.453700 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.453668 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24"] Oct 30 13:07:20.537022 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.536992 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1830fbd9-7c51-40d0-b6df-49a723185784-istio-data\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24\" (UID: \"1830fbd9-7c51-40d0-b6df-49a723185784\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" Oct 30 13:07:20.537176 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.537119 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1830fbd9-7c51-40d0-b6df-49a723185784-credential-socket\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24\" (UID: \"1830fbd9-7c51-40d0-b6df-49a723185784\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" Oct 30 13:07:20.537176 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.537152 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1830fbd9-7c51-40d0-b6df-49a723185784-workload-socket\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24\" (UID: \"1830fbd9-7c51-40d0-b6df-49a723185784\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" Oct 30 13:07:20.537273 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.537218 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1830fbd9-7c51-40d0-b6df-49a723185784-istio-token\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24\" (UID: \"1830fbd9-7c51-40d0-b6df-49a723185784\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" Oct 30 13:07:20.537308 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.537275 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1830fbd9-7c51-40d0-b6df-49a723185784-istio-envoy\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24\" (UID: \"1830fbd9-7c51-40d0-b6df-49a723185784\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" Oct 30 13:07:20.537341 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.537320 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1830fbd9-7c51-40d0-b6df-49a723185784-istio-podinfo\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24\" (UID: \"1830fbd9-7c51-40d0-b6df-49a723185784\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" Oct 30 13:07:20.537387 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.537374 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2frl6\" (UniqueName: \"kubernetes.io/projected/1830fbd9-7c51-40d0-b6df-49a723185784-kube-api-access-2frl6\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24\" (UID: \"1830fbd9-7c51-40d0-b6df-49a723185784\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" Oct 30 13:07:20.537432 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.537404 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1830fbd9-7c51-40d0-b6df-49a723185784-workload-certs\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24\" (UID: \"1830fbd9-7c51-40d0-b6df-49a723185784\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" Oct 30 13:07:20.537489 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.537477 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1830fbd9-7c51-40d0-b6df-49a723185784-istiod-ca-cert\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24\" (UID: \"1830fbd9-7c51-40d0-b6df-49a723185784\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" Oct 30 13:07:20.559994 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.557764 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf"] Oct 30 13:07:20.563497 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.563464 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:07:20.563596 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.563527 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:07:20.563596 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.563559 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:07:20.638362 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.638338 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1830fbd9-7c51-40d0-b6df-49a723185784-istio-envoy\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24\" (UID: \"1830fbd9-7c51-40d0-b6df-49a723185784\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" Oct 30 13:07:20.638471 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.638366 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1830fbd9-7c51-40d0-b6df-49a723185784-istio-podinfo\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24\" (UID: \"1830fbd9-7c51-40d0-b6df-49a723185784\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" Oct 30 13:07:20.638471 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.638384 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2frl6\" (UniqueName: \"kubernetes.io/projected/1830fbd9-7c51-40d0-b6df-49a723185784-kube-api-access-2frl6\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24\" (UID: \"1830fbd9-7c51-40d0-b6df-49a723185784\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" Oct 30 13:07:20.638471 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.638411 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1830fbd9-7c51-40d0-b6df-49a723185784-workload-certs\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24\" (UID: \"1830fbd9-7c51-40d0-b6df-49a723185784\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" Oct 30 13:07:20.638471 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.638451 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1830fbd9-7c51-40d0-b6df-49a723185784-istiod-ca-cert\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24\" (UID: \"1830fbd9-7c51-40d0-b6df-49a723185784\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" Oct 30 13:07:20.638745 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.638516 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1830fbd9-7c51-40d0-b6df-49a723185784-istio-data\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24\" (UID: \"1830fbd9-7c51-40d0-b6df-49a723185784\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" Oct 30 13:07:20.638745 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.638649 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1830fbd9-7c51-40d0-b6df-49a723185784-credential-socket\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24\" (UID: \"1830fbd9-7c51-40d0-b6df-49a723185784\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" Oct 30 13:07:20.638745 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.638685 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1830fbd9-7c51-40d0-b6df-49a723185784-workload-socket\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24\" (UID: \"1830fbd9-7c51-40d0-b6df-49a723185784\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" Oct 30 13:07:20.638903 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.638748 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1830fbd9-7c51-40d0-b6df-49a723185784-istio-token\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24\" (UID: \"1830fbd9-7c51-40d0-b6df-49a723185784\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" Oct 30 13:07:20.638903 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.638841 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1830fbd9-7c51-40d0-b6df-49a723185784-workload-certs\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24\" (UID: \"1830fbd9-7c51-40d0-b6df-49a723185784\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" Oct 30 13:07:20.639072 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.639051 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1830fbd9-7c51-40d0-b6df-49a723185784-istio-data\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24\" (UID: \"1830fbd9-7c51-40d0-b6df-49a723185784\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" Oct 30 13:07:20.639131 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.639087 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1830fbd9-7c51-40d0-b6df-49a723185784-workload-socket\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24\" (UID: \"1830fbd9-7c51-40d0-b6df-49a723185784\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" Oct 30 13:07:20.639582 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.639559 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1830fbd9-7c51-40d0-b6df-49a723185784-istiod-ca-cert\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24\" (UID: \"1830fbd9-7c51-40d0-b6df-49a723185784\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" Oct 30 13:07:20.639836 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.639814 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1830fbd9-7c51-40d0-b6df-49a723185784-credential-socket\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24\" (UID: \"1830fbd9-7c51-40d0-b6df-49a723185784\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" Oct 30 13:07:20.640703 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.640685 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1830fbd9-7c51-40d0-b6df-49a723185784-istio-envoy\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24\" (UID: \"1830fbd9-7c51-40d0-b6df-49a723185784\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" Oct 30 13:07:20.641446 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.641426 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1830fbd9-7c51-40d0-b6df-49a723185784-istio-podinfo\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24\" (UID: \"1830fbd9-7c51-40d0-b6df-49a723185784\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" Oct 30 13:07:20.665113 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.665044 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1830fbd9-7c51-40d0-b6df-49a723185784-istio-token\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24\" (UID: \"1830fbd9-7c51-40d0-b6df-49a723185784\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" Oct 30 13:07:20.665233 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.665191 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2frl6\" (UniqueName: \"kubernetes.io/projected/1830fbd9-7c51-40d0-b6df-49a723185784-kube-api-access-2frl6\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24\" (UID: \"1830fbd9-7c51-40d0-b6df-49a723185784\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" Oct 30 13:07:20.716024 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.715996 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" Oct 30 13:07:20.782265 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.782173 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" event={"ID":"aee561c5-3aa7-4021-9d3e-05e7636d19be","Type":"ContainerStarted","Data":"c46065deda8846da87056fdbf8225588e3a49592466cad9ef3c9931d49bb883f"} Oct 30 13:07:20.782265 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.782237 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" event={"ID":"aee561c5-3aa7-4021-9d3e-05e7636d19be","Type":"ContainerStarted","Data":"9b70974994632fb095a8defd4430013ec523854e530f5a0f4f71a52f3e27c7c0"} Oct 30 13:07:20.969766 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:20.969738 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c763adca-hn8ln" Oct 30 13:07:21.017282 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:21.017229 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" podStartSLOduration=2.017211147 podStartE2EDuration="2.017211147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:20.845821289 +0000 UTC m=+897.315237033" watchObservedRunningTime="2026-10-30 13:07:21.017211147 +0000 UTC m=+897.486626890" Oct 30 13:07:21.039975 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:21.039934 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24"] Oct 30 13:07:21.046359 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:07:21.046326 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1830fbd9_7c51_40d0_b6df_49a723185784.slice/crio-cbfa5e1a2db9048521eecfbd94901f28693d8cff184d8b2472230b78d5cf1b2d WatchSource:0}: Error finding container cbfa5e1a2db9048521eecfbd94901f28693d8cff184d8b2472230b78d5cf1b2d: Status 404 returned error can't find the container with id cbfa5e1a2db9048521eecfbd94901f28693d8cff184d8b2472230b78d5cf1b2d Oct 30 13:07:21.050906 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:21.050869 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:07:21.051120 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:21.051094 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:07:21.051260 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:21.051242 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:07:21.145578 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:21.145551 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/967aa55f-1503-41bd-8089-a909532dd160-kube-api-access-wjpj5\" (UniqueName: \"kubernetes.io/projected/967aa55f-1503-41bd-8089-a909532dd160-kube-api-access-wjpj5\") pod \"967aa55f-1503-41bd-8089-a909532dd160\" (UID: \"967aa55f-1503-41bd-8089-a909532dd160\") " Oct 30 13:07:21.145821 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:21.145800 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/967aa55f-1503-41bd-8089-a909532dd160-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/967aa55f-1503-41bd-8089-a909532dd160-maas-api-openshift-ca\") pod \"967aa55f-1503-41bd-8089-a909532dd160\" (UID: \"967aa55f-1503-41bd-8089-a909532dd160\") " Oct 30 13:07:21.146092 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:21.146044 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/967aa55f-1503-41bd-8089-a909532dd160-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/967aa55f-1503-41bd-8089-a909532dd160-maas-api-xks-ca\") pod \"967aa55f-1503-41bd-8089-a909532dd160\" (UID: \"967aa55f-1503-41bd-8089-a909532dd160\") " Oct 30 13:07:21.146195 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:21.146145 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/967aa55f-1503-41bd-8089-a909532dd160-maas-api-openshift-ca" pod "967aa55f-1503-41bd-8089-a909532dd160" (UID: "967aa55f-1503-41bd-8089-a909532dd160"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:21.146584 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:21.146565 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/967aa55f-1503-41bd-8089-a909532dd160-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:07:21.148163 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:21.148134 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/967aa55f-1503-41bd-8089-a909532dd160-kube-api-access-wjpj5" pod "967aa55f-1503-41bd-8089-a909532dd160" (UID: "967aa55f-1503-41bd-8089-a909532dd160"). InnerVolumeSpecName "kube-api-access-wjpj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:21.148347 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:21.148330 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/967aa55f-1503-41bd-8089-a909532dd160-maas-api-xks-ca" pod "967aa55f-1503-41bd-8089-a909532dd160" (UID: "967aa55f-1503-41bd-8089-a909532dd160"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:21.204824 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:21.204791 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" Oct 30 13:07:21.205941 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:21.205915 5922 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 13:07:21.206058 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:21.205962 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" podUID="aee561c5-3aa7-4021-9d3e-05e7636d19be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 13:07:21.248463 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:21.248391 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/967aa55f-1503-41bd-8089-a909532dd160-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:07:21.248463 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:21.248429 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjpj5\" (UniqueName: \"kubernetes.io/projected/967aa55f-1503-41bd-8089-a909532dd160-kube-api-access-wjpj5\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:07:21.818876 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:21.818845 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c763adca-hn8ln" Oct 30 13:07:21.818876 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:21.818854 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c763adca-hn8ln" event={"ID":"967aa55f-1503-41bd-8089-a909532dd160","Type":"ContainerDied","Data":"22c3037b7e1951a6fba848460e29380c8dd585974d35cdacc15b756da8f20619"} Oct 30 13:07:21.819416 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:21.818913 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22c3037b7e1951a6fba848460e29380c8dd585974d35cdacc15b756da8f20619" Oct 30 13:07:21.821790 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:21.821751 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" event={"ID":"1830fbd9-7c51-40d0-b6df-49a723185784","Type":"ContainerStarted","Data":"629fe8b00af9f7879c5086af3fa9c9cb0bf1701e99a7af8dd6a41b1a9204ed05"} Oct 30 13:07:21.822114 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:21.822077 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" event={"ID":"1830fbd9-7c51-40d0-b6df-49a723185784","Type":"ContainerStarted","Data":"cbfa5e1a2db9048521eecfbd94901f28693d8cff184d8b2472230b78d5cf1b2d"} Oct 30 13:07:21.914697 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:21.914386 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" podStartSLOduration=1.914362783 podStartE2EDuration="1.914362783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:21.89791931 +0000 UTC m=+898.367335072" watchObservedRunningTime="2026-10-30 13:07:21.914362783 +0000 UTC m=+898.383778529" Oct 30 13:07:22.215679 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:22.213982 5922 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 13:07:22.215679 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:22.214039 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" podUID="aee561c5-3aa7-4021-9d3e-05e7636d19be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 13:07:22.550516 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:22.550436 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:22.716826 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:22.716790 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" Oct 30 13:07:22.718715 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:22.718685 5922 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 13:07:22.718869 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:22.718740 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" podUID="1830fbd9-7c51-40d0-b6df-49a723185784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 13:07:23.205640 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:23.204956 5922 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 13:07:23.205640 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:23.205016 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" podUID="aee561c5-3aa7-4021-9d3e-05e7636d19be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 13:07:23.553528 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:07:23.553437 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:07:23.716637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:23.716549 5922 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 13:07:23.716637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:23.716599 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" podUID="1830fbd9-7c51-40d0-b6df-49a723185784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 13:07:23.749064 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:23.749031 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-530952d9-qjk4g"] Oct 30 13:07:23.750016 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:23.749991 5922 cpu_manager.go:414] "containerMap: removing container" podUID="967aa55f-1503-41bd-8089-a909532dd160" containerName="revoke-keys" Oct 30 13:07:23.750016 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:23.750016 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="967aa55f-1503-41bd-8089-a909532dd160" containerName="revoke-keys" Oct 30 13:07:23.750171 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:23.750163 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="967aa55f-1503-41bd-8089-a909532dd160" containerName="revoke-keys" Oct 30 13:07:23.759379 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:23.759350 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-530952d9-qjk4g" Oct 30 13:07:23.844718 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:23.831533 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:07:23.844718 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:23.838674 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:07:23.844718 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:23.840748 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:07:23.844718 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:23.841497 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:07:23.844718 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:23.843335 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddd94e09-f766-4de6-b4f5-0a2fc927c3db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-530952d9-qjk4g\" (UID: \"ddd94e09-f766-4de6-b4f5-0a2fc927c3db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-530952d9-qjk4g" Oct 30 13:07:23.844718 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:23.843467 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddd94e09-f766-4de6-b4f5-0a2fc927c3db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-530952d9-qjk4g\" (UID: \"ddd94e09-f766-4de6-b4f5-0a2fc927c3db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-530952d9-qjk4g" Oct 30 13:07:23.844718 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:23.843509 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5w7w\" (UniqueName: \"kubernetes.io/projected/ddd94e09-f766-4de6-b4f5-0a2fc927c3db-kube-api-access-l5w7w\") pod \"maas-api-revoke-sub-530952d9-qjk4g\" (UID: \"ddd94e09-f766-4de6-b4f5-0a2fc927c3db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-530952d9-qjk4g" Oct 30 13:07:23.849979 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:23.849950 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-530952d9-qjk4g"] Oct 30 13:07:23.945213 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:23.945181 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddd94e09-f766-4de6-b4f5-0a2fc927c3db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-530952d9-qjk4g\" (UID: \"ddd94e09-f766-4de6-b4f5-0a2fc927c3db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-530952d9-qjk4g" Oct 30 13:07:23.945406 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:23.945312 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddd94e09-f766-4de6-b4f5-0a2fc927c3db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-530952d9-qjk4g\" (UID: \"ddd94e09-f766-4de6-b4f5-0a2fc927c3db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-530952d9-qjk4g" Oct 30 13:07:23.945406 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:23.945355 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5w7w\" (UniqueName: \"kubernetes.io/projected/ddd94e09-f766-4de6-b4f5-0a2fc927c3db-kube-api-access-l5w7w\") pod \"maas-api-revoke-sub-530952d9-qjk4g\" (UID: \"ddd94e09-f766-4de6-b4f5-0a2fc927c3db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-530952d9-qjk4g" Oct 30 13:07:23.945927 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:23.945903 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddd94e09-f766-4de6-b4f5-0a2fc927c3db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-530952d9-qjk4g\" (UID: \"ddd94e09-f766-4de6-b4f5-0a2fc927c3db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-530952d9-qjk4g" Oct 30 13:07:23.947566 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:23.947538 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddd94e09-f766-4de6-b4f5-0a2fc927c3db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-530952d9-qjk4g\" (UID: \"ddd94e09-f766-4de6-b4f5-0a2fc927c3db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-530952d9-qjk4g" Oct 30 13:07:24.015089 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:24.014911 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5w7w\" (UniqueName: \"kubernetes.io/projected/ddd94e09-f766-4de6-b4f5-0a2fc927c3db-kube-api-access-l5w7w\") pod \"maas-api-revoke-sub-530952d9-qjk4g\" (UID: \"ddd94e09-f766-4de6-b4f5-0a2fc927c3db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-530952d9-qjk4g" Oct 30 13:07:24.153351 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:24.153273 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-530952d9-qjk4g" Oct 30 13:07:24.220639 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:24.218980 5922 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 13:07:24.220639 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:24.219052 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" podUID="aee561c5-3aa7-4021-9d3e-05e7636d19be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 13:07:24.558570 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:24.558535 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-530952d9-qjk4g"] Oct 30 13:07:24.717345 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:24.717309 5922 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 13:07:24.717546 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:24.717374 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" podUID="1830fbd9-7c51-40d0-b6df-49a723185784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 13:07:24.923648 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:24.913385 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-530952d9-qjk4g" event={"ID":"ddd94e09-f766-4de6-b4f5-0a2fc927c3db","Type":"ContainerStarted","Data":"7e9496bf9d2a1a0f11d4ae5d92468b15086cfe1bca23df9f352d553040983d9c"} Oct 30 13:07:24.923648 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:24.918876 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:25.213799 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:25.207989 5922 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 13:07:25.213799 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:25.208049 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" podUID="aee561c5-3aa7-4021-9d3e-05e7636d19be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 13:07:25.718672 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:25.718162 5922 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 13:07:25.718672 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:25.718232 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" podUID="1830fbd9-7c51-40d0-b6df-49a723185784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 13:07:25.928386 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:25.926936 5922 generic.go:356] "Generic (PLEG): container finished" podID="ddd94e09-f766-4de6-b4f5-0a2fc927c3db" containerID="6565ef857cce9880b37d7d74e4d3199ac01756395112e2f1fbf6429ec77602a2" exitCode=0 Oct 30 13:07:25.928386 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:25.927059 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-530952d9-qjk4g" event={"ID":"ddd94e09-f766-4de6-b4f5-0a2fc927c3db","Type":"ContainerDied","Data":"6565ef857cce9880b37d7d74e4d3199ac01756395112e2f1fbf6429ec77602a2"} Oct 30 13:07:26.213638 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:26.208511 5922 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 13:07:26.213638 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:26.208572 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" podUID="aee561c5-3aa7-4021-9d3e-05e7636d19be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 13:07:26.717636 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:26.717033 5922 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 13:07:26.717636 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:26.717096 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" podUID="1830fbd9-7c51-40d0-b6df-49a723185784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 13:07:27.148120 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:27.147847 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-530952d9-qjk4g" Oct 30 13:07:27.203901 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:27.203865 5922 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 13:07:27.204048 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:27.203929 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" podUID="aee561c5-3aa7-4021-9d3e-05e7636d19be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 13:07:27.239188 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:27.239153 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddd94e09-f766-4de6-b4f5-0a2fc927c3db-kube-api-access-l5w7w\" (UniqueName: \"kubernetes.io/projected/ddd94e09-f766-4de6-b4f5-0a2fc927c3db-kube-api-access-l5w7w\") pod \"ddd94e09-f766-4de6-b4f5-0a2fc927c3db\" (UID: \"ddd94e09-f766-4de6-b4f5-0a2fc927c3db\") " Oct 30 13:07:27.239378 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:27.239346 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddd94e09-f766-4de6-b4f5-0a2fc927c3db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddd94e09-f766-4de6-b4f5-0a2fc927c3db-maas-api-xks-ca\") pod \"ddd94e09-f766-4de6-b4f5-0a2fc927c3db\" (UID: \"ddd94e09-f766-4de6-b4f5-0a2fc927c3db\") " Oct 30 13:07:27.239546 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:27.239526 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddd94e09-f766-4de6-b4f5-0a2fc927c3db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddd94e09-f766-4de6-b4f5-0a2fc927c3db-maas-api-openshift-ca\") pod \"ddd94e09-f766-4de6-b4f5-0a2fc927c3db\" (UID: \"ddd94e09-f766-4de6-b4f5-0a2fc927c3db\") " Oct 30 13:07:27.239936 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:27.239909 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddd94e09-f766-4de6-b4f5-0a2fc927c3db-maas-api-openshift-ca" pod "ddd94e09-f766-4de6-b4f5-0a2fc927c3db" (UID: "ddd94e09-f766-4de6-b4f5-0a2fc927c3db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:27.240255 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:27.240203 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddd94e09-f766-4de6-b4f5-0a2fc927c3db-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:07:27.241544 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:27.241514 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddd94e09-f766-4de6-b4f5-0a2fc927c3db-kube-api-access-l5w7w" pod "ddd94e09-f766-4de6-b4f5-0a2fc927c3db" (UID: "ddd94e09-f766-4de6-b4f5-0a2fc927c3db"). InnerVolumeSpecName "kube-api-access-l5w7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:27.241783 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:27.241755 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddd94e09-f766-4de6-b4f5-0a2fc927c3db-maas-api-xks-ca" pod "ddd94e09-f766-4de6-b4f5-0a2fc927c3db" (UID: "ddd94e09-f766-4de6-b4f5-0a2fc927c3db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:27.342452 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:27.342380 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5w7w\" (UniqueName: \"kubernetes.io/projected/ddd94e09-f766-4de6-b4f5-0a2fc927c3db-kube-api-access-l5w7w\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:07:27.342452 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:27.342418 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddd94e09-f766-4de6-b4f5-0a2fc927c3db-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:07:27.716800 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:27.716762 5922 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 13:07:27.716996 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:27.716826 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" podUID="1830fbd9-7c51-40d0-b6df-49a723185784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 13:07:27.946761 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:27.946736 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-530952d9-qjk4g" Oct 30 13:07:27.946957 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:27.946778 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-530952d9-qjk4g" event={"ID":"ddd94e09-f766-4de6-b4f5-0a2fc927c3db","Type":"ContainerDied","Data":"7e9496bf9d2a1a0f11d4ae5d92468b15086cfe1bca23df9f352d553040983d9c"} Oct 30 13:07:27.946957 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:27.946838 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e9496bf9d2a1a0f11d4ae5d92468b15086cfe1bca23df9f352d553040983d9c" Oct 30 13:07:28.118257 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:28.118218 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:28.258505 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:28.258418 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" Oct 30 13:07:28.258967 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:28.258896 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" Oct 30 13:07:28.259819 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:28.259793 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" Oct 30 13:07:28.720528 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:28.720498 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" Oct 30 13:07:28.720979 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:28.720966 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" Oct 30 13:07:28.721760 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:28.721738 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" Oct 30 13:07:31.358415 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:31.358380 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:32.599971 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:32.599932 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:32.607255 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:32.607225 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a702fe8a-ggd29"] Oct 30 13:07:32.608216 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:32.608196 5922 cpu_manager.go:414] "containerMap: removing container" podUID="ddd94e09-f766-4de6-b4f5-0a2fc927c3db" containerName="revoke-keys" Oct 30 13:07:32.608263 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:32.608221 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddd94e09-f766-4de6-b4f5-0a2fc927c3db" containerName="revoke-keys" Oct 30 13:07:32.608426 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:32.608411 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddd94e09-f766-4de6-b4f5-0a2fc927c3db" containerName="revoke-keys" Oct 30 13:07:32.612846 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:32.612807 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a702fe8a-ggd29" Oct 30 13:07:32.629392 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:32.629370 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:07:32.631122 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:32.631097 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a702fe8a-ggd29"] Oct 30 13:07:32.648140 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:32.635958 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:07:32.648140 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:32.636084 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:07:32.648140 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:32.636174 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:07:32.730398 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:32.730357 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f068e94-22e4-4154-b2ca-fe19368f0c86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a702fe8a-ggd29\" (UID: \"0f068e94-22e4-4154-b2ca-fe19368f0c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a702fe8a-ggd29" Oct 30 13:07:32.730731 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:32.730702 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f068e94-22e4-4154-b2ca-fe19368f0c86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a702fe8a-ggd29\" (UID: \"0f068e94-22e4-4154-b2ca-fe19368f0c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a702fe8a-ggd29" Oct 30 13:07:32.730903 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:32.730883 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m58z4\" (UniqueName: \"kubernetes.io/projected/0f068e94-22e4-4154-b2ca-fe19368f0c86-kube-api-access-m58z4\") pod \"maas-api-revoke-sub-a702fe8a-ggd29\" (UID: \"0f068e94-22e4-4154-b2ca-fe19368f0c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a702fe8a-ggd29" Oct 30 13:07:32.832991 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:32.832941 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f068e94-22e4-4154-b2ca-fe19368f0c86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a702fe8a-ggd29\" (UID: \"0f068e94-22e4-4154-b2ca-fe19368f0c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a702fe8a-ggd29" Oct 30 13:07:32.833190 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:32.833026 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f068e94-22e4-4154-b2ca-fe19368f0c86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a702fe8a-ggd29\" (UID: \"0f068e94-22e4-4154-b2ca-fe19368f0c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a702fe8a-ggd29" Oct 30 13:07:32.833190 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:32.833096 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m58z4\" (UniqueName: \"kubernetes.io/projected/0f068e94-22e4-4154-b2ca-fe19368f0c86-kube-api-access-m58z4\") pod \"maas-api-revoke-sub-a702fe8a-ggd29\" (UID: \"0f068e94-22e4-4154-b2ca-fe19368f0c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a702fe8a-ggd29" Oct 30 13:07:32.834356 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:32.834325 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f068e94-22e4-4154-b2ca-fe19368f0c86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a702fe8a-ggd29\" (UID: \"0f068e94-22e4-4154-b2ca-fe19368f0c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a702fe8a-ggd29" Oct 30 13:07:32.836537 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:32.836511 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f068e94-22e4-4154-b2ca-fe19368f0c86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a702fe8a-ggd29\" (UID: \"0f068e94-22e4-4154-b2ca-fe19368f0c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a702fe8a-ggd29" Oct 30 13:07:32.889849 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:32.889353 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m58z4\" (UniqueName: \"kubernetes.io/projected/0f068e94-22e4-4154-b2ca-fe19368f0c86-kube-api-access-m58z4\") pod \"maas-api-revoke-sub-a702fe8a-ggd29\" (UID: \"0f068e94-22e4-4154-b2ca-fe19368f0c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a702fe8a-ggd29" Oct 30 13:07:32.934640 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:32.933840 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a702fe8a-ggd29" Oct 30 13:07:34.051992 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:34.049871 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" podUID="b487a069-f519-45a3-b1d8-f32f2431797b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 13:07:34.103642 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:34.089061 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" podUID="7751779d-b082-409a-a20f-ff0ff6263808" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:34.147726 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:34.144000 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" podUID="dad5bdad-cd6f-41f1-a124-540c3524ead7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:34.168013 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:34.167976 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" podUID="dfd83cc8-6503-45a8-8260-c0edb8752471" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:34.231104 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:34.222909 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" podUID="47517373-759d-4eeb-ab5e-6150b7e82bb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:34.301315 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:34.301287 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a702fe8a-ggd29"] Oct 30 13:07:35.198488 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:35.198456 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a702fe8a-ggd29" event={"ID":"0f068e94-22e4-4154-b2ca-fe19368f0c86","Type":"ContainerStarted","Data":"4f44d264324f44b0238491c456cf8a5eff3f65b9bcd21abec22b0d52d7bf0639"} Oct 30 13:07:35.199020 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:35.198507 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a702fe8a-ggd29" event={"ID":"0f068e94-22e4-4154-b2ca-fe19368f0c86","Type":"ContainerStarted","Data":"d6e294a6be079017fa467379c0c34bc25f166a41f0d899ed038015556d2f742b"} Oct 30 13:07:35.251359 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:35.251316 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a702fe8a-ggd29" podStartSLOduration=3.251304285 podStartE2EDuration="3.251304285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:35.247517636 +0000 UTC m=+911.716933382" watchObservedRunningTime="2026-10-30 13:07:35.251304285 +0000 UTC m=+911.720720063" Oct 30 13:07:35.901805 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:35.901719 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:36.208107 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:36.208070 5922 generic.go:356] "Generic (PLEG): container finished" podID="0f068e94-22e4-4154-b2ca-fe19368f0c86" containerID="4f44d264324f44b0238491c456cf8a5eff3f65b9bcd21abec22b0d52d7bf0639" exitCode=0 Oct 30 13:07:36.208510 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:36.208129 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a702fe8a-ggd29" event={"ID":"0f068e94-22e4-4154-b2ca-fe19368f0c86","Type":"ContainerDied","Data":"4f44d264324f44b0238491c456cf8a5eff3f65b9bcd21abec22b0d52d7bf0639"} Oct 30 13:07:37.369401 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:37.369374 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a702fe8a-ggd29" Oct 30 13:07:37.542723 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:37.542655 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f068e94-22e4-4154-b2ca-fe19368f0c86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f068e94-22e4-4154-b2ca-fe19368f0c86-maas-api-xks-ca\") pod \"0f068e94-22e4-4154-b2ca-fe19368f0c86\" (UID: \"0f068e94-22e4-4154-b2ca-fe19368f0c86\") " Oct 30 13:07:37.542723 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:37.542719 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f068e94-22e4-4154-b2ca-fe19368f0c86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f068e94-22e4-4154-b2ca-fe19368f0c86-maas-api-openshift-ca\") pod \"0f068e94-22e4-4154-b2ca-fe19368f0c86\" (UID: \"0f068e94-22e4-4154-b2ca-fe19368f0c86\") " Oct 30 13:07:37.542906 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:37.542780 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f068e94-22e4-4154-b2ca-fe19368f0c86-kube-api-access-m58z4\" (UniqueName: \"kubernetes.io/projected/0f068e94-22e4-4154-b2ca-fe19368f0c86-kube-api-access-m58z4\") pod \"0f068e94-22e4-4154-b2ca-fe19368f0c86\" (UID: \"0f068e94-22e4-4154-b2ca-fe19368f0c86\") " Oct 30 13:07:37.543059 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:37.543035 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f068e94-22e4-4154-b2ca-fe19368f0c86-maas-api-openshift-ca" pod "0f068e94-22e4-4154-b2ca-fe19368f0c86" (UID: "0f068e94-22e4-4154-b2ca-fe19368f0c86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:37.543263 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:37.543239 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f068e94-22e4-4154-b2ca-fe19368f0c86-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:07:37.544688 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:37.544665 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f068e94-22e4-4154-b2ca-fe19368f0c86-maas-api-xks-ca" pod "0f068e94-22e4-4154-b2ca-fe19368f0c86" (UID: "0f068e94-22e4-4154-b2ca-fe19368f0c86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:37.544787 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:37.544775 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f068e94-22e4-4154-b2ca-fe19368f0c86-kube-api-access-m58z4" pod "0f068e94-22e4-4154-b2ca-fe19368f0c86" (UID: "0f068e94-22e4-4154-b2ca-fe19368f0c86"). InnerVolumeSpecName "kube-api-access-m58z4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:37.644683 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:37.644657 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f068e94-22e4-4154-b2ca-fe19368f0c86-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:07:37.644683 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:37.644681 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m58z4\" (UniqueName: \"kubernetes.io/projected/0f068e94-22e4-4154-b2ca-fe19368f0c86-kube-api-access-m58z4\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:07:38.224529 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:38.224492 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a702fe8a-ggd29" event={"ID":"0f068e94-22e4-4154-b2ca-fe19368f0c86","Type":"ContainerDied","Data":"d6e294a6be079017fa467379c0c34bc25f166a41f0d899ed038015556d2f742b"} Oct 30 13:07:38.224688 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:38.224555 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6e294a6be079017fa467379c0c34bc25f166a41f0d899ed038015556d2f742b" Oct 30 13:07:38.224688 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:38.224506 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a702fe8a-ggd29" Oct 30 13:07:39.093846 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:39.093815 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:40.810944 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:40.810906 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04d0392c-j658p"] Oct 30 13:07:40.811517 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:40.811504 5922 cpu_manager.go:414] "containerMap: removing container" podUID="0f068e94-22e4-4154-b2ca-fe19368f0c86" containerName="revoke-keys" Oct 30 13:07:40.811565 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:40.811517 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f068e94-22e4-4154-b2ca-fe19368f0c86" containerName="revoke-keys" Oct 30 13:07:40.811641 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:40.811630 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f068e94-22e4-4154-b2ca-fe19368f0c86" containerName="revoke-keys" Oct 30 13:07:40.815141 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:40.815123 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d0392c-j658p" Oct 30 13:07:40.833664 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:40.833642 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04d0392c-j658p"] Oct 30 13:07:40.841196 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:40.841174 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:07:40.841325 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:40.841252 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:07:40.841325 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:40.841295 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:07:40.842556 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:40.842529 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:07:40.886855 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:40.886829 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3091a06-b26b-4793-8d07-4f1d96bc1aa4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04d0392c-j658p\" (UID: \"d3091a06-b26b-4793-8d07-4f1d96bc1aa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d0392c-j658p" Oct 30 13:07:40.886983 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:40.886881 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klld7\" (UniqueName: \"kubernetes.io/projected/d3091a06-b26b-4793-8d07-4f1d96bc1aa4-kube-api-access-klld7\") pod \"maas-api-revoke-sub-04d0392c-j658p\" (UID: \"d3091a06-b26b-4793-8d07-4f1d96bc1aa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d0392c-j658p" Oct 30 13:07:40.886983 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:40.886909 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3091a06-b26b-4793-8d07-4f1d96bc1aa4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04d0392c-j658p\" (UID: \"d3091a06-b26b-4793-8d07-4f1d96bc1aa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d0392c-j658p" Oct 30 13:07:40.986467 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:40.986436 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:40.988691 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:40.988669 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3091a06-b26b-4793-8d07-4f1d96bc1aa4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04d0392c-j658p\" (UID: \"d3091a06-b26b-4793-8d07-4f1d96bc1aa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d0392c-j658p" Oct 30 13:07:40.988843 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:40.988830 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3091a06-b26b-4793-8d07-4f1d96bc1aa4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04d0392c-j658p\" (UID: \"d3091a06-b26b-4793-8d07-4f1d96bc1aa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d0392c-j658p" Oct 30 13:07:40.988913 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:40.988874 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klld7\" (UniqueName: \"kubernetes.io/projected/d3091a06-b26b-4793-8d07-4f1d96bc1aa4-kube-api-access-klld7\") pod \"maas-api-revoke-sub-04d0392c-j658p\" (UID: \"d3091a06-b26b-4793-8d07-4f1d96bc1aa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d0392c-j658p" Oct 30 13:07:40.989324 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:40.989303 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3091a06-b26b-4793-8d07-4f1d96bc1aa4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04d0392c-j658p\" (UID: \"d3091a06-b26b-4793-8d07-4f1d96bc1aa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d0392c-j658p" Oct 30 13:07:40.991079 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:40.991059 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3091a06-b26b-4793-8d07-4f1d96bc1aa4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04d0392c-j658p\" (UID: \"d3091a06-b26b-4793-8d07-4f1d96bc1aa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d0392c-j658p" Oct 30 13:07:41.013759 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:41.013734 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klld7\" (UniqueName: \"kubernetes.io/projected/d3091a06-b26b-4793-8d07-4f1d96bc1aa4-kube-api-access-klld7\") pod \"maas-api-revoke-sub-04d0392c-j658p\" (UID: \"d3091a06-b26b-4793-8d07-4f1d96bc1aa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d0392c-j658p" Oct 30 13:07:41.128176 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:41.128094 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d0392c-j658p" Oct 30 13:07:41.462024 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:41.462001 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04d0392c-j658p"] Oct 30 13:07:41.463587 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:07:41.463552 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3091a06_b26b_4793_8d07_4f1d96bc1aa4.slice/crio-f20cba5da177b36137a382a9e781bfa46593f191912d9ac88a1940db84dbbfed WatchSource:0}: Error finding container f20cba5da177b36137a382a9e781bfa46593f191912d9ac88a1940db84dbbfed: Status 404 returned error can't find the container with id f20cba5da177b36137a382a9e781bfa46593f191912d9ac88a1940db84dbbfed Oct 30 13:07:42.257295 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:42.257261 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d0392c-j658p" event={"ID":"d3091a06-b26b-4793-8d07-4f1d96bc1aa4","Type":"ContainerStarted","Data":"5f1644984bcd48f7c34ca25f5e12b91575fa35dc87a25139c7c7a8b82c2b2abc"} Oct 30 13:07:42.257752 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:42.257310 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d0392c-j658p" event={"ID":"d3091a06-b26b-4793-8d07-4f1d96bc1aa4","Type":"ContainerStarted","Data":"f20cba5da177b36137a382a9e781bfa46593f191912d9ac88a1940db84dbbfed"} Oct 30 13:07:42.303121 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:42.303066 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d0392c-j658p" podStartSLOduration=2.303053689 podStartE2EDuration="2.303053689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:42.299944366 +0000 UTC m=+918.769360112" watchObservedRunningTime="2026-10-30 13:07:42.303053689 +0000 UTC m=+918.772469433" Oct 30 13:07:42.809844 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:42.809793 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:43.266357 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:43.266327 5922 generic.go:356] "Generic (PLEG): container finished" podID="d3091a06-b26b-4793-8d07-4f1d96bc1aa4" containerID="5f1644984bcd48f7c34ca25f5e12b91575fa35dc87a25139c7c7a8b82c2b2abc" exitCode=0 Oct 30 13:07:43.266818 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:43.266404 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d0392c-j658p" event={"ID":"d3091a06-b26b-4793-8d07-4f1d96bc1aa4","Type":"ContainerDied","Data":"5f1644984bcd48f7c34ca25f5e12b91575fa35dc87a25139c7c7a8b82c2b2abc"} Oct 30 13:07:43.606238 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:43.606165 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-527643e2-9knlt"] Oct 30 13:07:43.610226 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:43.610210 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-527643e2-9knlt" Oct 30 13:07:43.629534 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:43.629502 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-527643e2-9knlt"] Oct 30 13:07:43.724783 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:43.724751 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w89z2\" (UniqueName: \"kubernetes.io/projected/f871ce6c-6eae-42e7-840f-1c897f8c53ce-kube-api-access-w89z2\") pod \"maas-api-revoke-sub-527643e2-9knlt\" (UID: \"f871ce6c-6eae-42e7-840f-1c897f8c53ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-527643e2-9knlt" Oct 30 13:07:43.725001 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:43.724986 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f871ce6c-6eae-42e7-840f-1c897f8c53ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-527643e2-9knlt\" (UID: \"f871ce6c-6eae-42e7-840f-1c897f8c53ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-527643e2-9knlt" Oct 30 13:07:43.725045 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:43.725014 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f871ce6c-6eae-42e7-840f-1c897f8c53ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-527643e2-9knlt\" (UID: \"f871ce6c-6eae-42e7-840f-1c897f8c53ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-527643e2-9knlt" Oct 30 13:07:43.826341 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:43.826309 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f871ce6c-6eae-42e7-840f-1c897f8c53ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-527643e2-9knlt\" (UID: \"f871ce6c-6eae-42e7-840f-1c897f8c53ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-527643e2-9knlt" Oct 30 13:07:43.826494 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:43.826346 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f871ce6c-6eae-42e7-840f-1c897f8c53ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-527643e2-9knlt\" (UID: \"f871ce6c-6eae-42e7-840f-1c897f8c53ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-527643e2-9knlt" Oct 30 13:07:43.826494 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:43.826383 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w89z2\" (UniqueName: \"kubernetes.io/projected/f871ce6c-6eae-42e7-840f-1c897f8c53ce-kube-api-access-w89z2\") pod \"maas-api-revoke-sub-527643e2-9knlt\" (UID: \"f871ce6c-6eae-42e7-840f-1c897f8c53ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-527643e2-9knlt" Oct 30 13:07:43.827099 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:43.827074 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f871ce6c-6eae-42e7-840f-1c897f8c53ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-527643e2-9knlt\" (UID: \"f871ce6c-6eae-42e7-840f-1c897f8c53ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-527643e2-9knlt" Oct 30 13:07:43.828769 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:43.828744 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f871ce6c-6eae-42e7-840f-1c897f8c53ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-527643e2-9knlt\" (UID: \"f871ce6c-6eae-42e7-840f-1c897f8c53ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-527643e2-9knlt" Oct 30 13:07:43.851266 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:43.851240 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w89z2\" (UniqueName: \"kubernetes.io/projected/f871ce6c-6eae-42e7-840f-1c897f8c53ce-kube-api-access-w89z2\") pod \"maas-api-revoke-sub-527643e2-9knlt\" (UID: \"f871ce6c-6eae-42e7-840f-1c897f8c53ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-527643e2-9knlt" Oct 30 13:07:43.923990 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:43.923936 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-527643e2-9knlt" Oct 30 13:07:44.232853 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:44.232828 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-527643e2-9knlt"] Oct 30 13:07:44.234455 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:07:44.234429 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf871ce6c_6eae_42e7_840f_1c897f8c53ce.slice/crio-89bd365aabdde28ebc80367294f0fb74c6fce9afe820893d661da975145f7c49 WatchSource:0}: Error finding container 89bd365aabdde28ebc80367294f0fb74c6fce9afe820893d661da975145f7c49: Status 404 returned error can't find the container with id 89bd365aabdde28ebc80367294f0fb74c6fce9afe820893d661da975145f7c49 Oct 30 13:07:44.274653 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:44.274601 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-527643e2-9knlt" event={"ID":"f871ce6c-6eae-42e7-840f-1c897f8c53ce","Type":"ContainerStarted","Data":"89bd365aabdde28ebc80367294f0fb74c6fce9afe820893d661da975145f7c49"} Oct 30 13:07:44.399973 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:44.399950 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d0392c-j658p" Oct 30 13:07:44.434226 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:44.434202 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3091a06-b26b-4793-8d07-4f1d96bc1aa4-kube-api-access-klld7\" (UniqueName: \"kubernetes.io/projected/d3091a06-b26b-4793-8d07-4f1d96bc1aa4-kube-api-access-klld7\") pod \"d3091a06-b26b-4793-8d07-4f1d96bc1aa4\" (UID: \"d3091a06-b26b-4793-8d07-4f1d96bc1aa4\") " Oct 30 13:07:44.434380 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:44.434333 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3091a06-b26b-4793-8d07-4f1d96bc1aa4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3091a06-b26b-4793-8d07-4f1d96bc1aa4-maas-api-xks-ca\") pod \"d3091a06-b26b-4793-8d07-4f1d96bc1aa4\" (UID: \"d3091a06-b26b-4793-8d07-4f1d96bc1aa4\") " Oct 30 13:07:44.434496 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:44.434479 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3091a06-b26b-4793-8d07-4f1d96bc1aa4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3091a06-b26b-4793-8d07-4f1d96bc1aa4-maas-api-openshift-ca\") pod \"d3091a06-b26b-4793-8d07-4f1d96bc1aa4\" (UID: \"d3091a06-b26b-4793-8d07-4f1d96bc1aa4\") " Oct 30 13:07:44.434831 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:44.434800 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3091a06-b26b-4793-8d07-4f1d96bc1aa4-maas-api-openshift-ca" pod "d3091a06-b26b-4793-8d07-4f1d96bc1aa4" (UID: "d3091a06-b26b-4793-8d07-4f1d96bc1aa4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:44.435212 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:44.435190 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3091a06-b26b-4793-8d07-4f1d96bc1aa4-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:07:44.436201 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:44.436182 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3091a06-b26b-4793-8d07-4f1d96bc1aa4-kube-api-access-klld7" pod "d3091a06-b26b-4793-8d07-4f1d96bc1aa4" (UID: "d3091a06-b26b-4793-8d07-4f1d96bc1aa4"). InnerVolumeSpecName "kube-api-access-klld7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:44.436354 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:44.436340 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3091a06-b26b-4793-8d07-4f1d96bc1aa4-maas-api-xks-ca" pod "d3091a06-b26b-4793-8d07-4f1d96bc1aa4" (UID: "d3091a06-b26b-4793-8d07-4f1d96bc1aa4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:44.536705 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:44.536647 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klld7\" (UniqueName: \"kubernetes.io/projected/d3091a06-b26b-4793-8d07-4f1d96bc1aa4-kube-api-access-klld7\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:07:44.536705 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:44.536674 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3091a06-b26b-4793-8d07-4f1d96bc1aa4-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:07:44.780916 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:44.780875 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:45.283365 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:45.283338 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d0392c-j658p" Oct 30 13:07:45.283777 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:45.283367 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d0392c-j658p" event={"ID":"d3091a06-b26b-4793-8d07-4f1d96bc1aa4","Type":"ContainerDied","Data":"f20cba5da177b36137a382a9e781bfa46593f191912d9ac88a1940db84dbbfed"} Oct 30 13:07:45.283777 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:45.283433 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f20cba5da177b36137a382a9e781bfa46593f191912d9ac88a1940db84dbbfed" Oct 30 13:07:45.285150 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:45.285123 5922 generic.go:356] "Generic (PLEG): container finished" podID="f871ce6c-6eae-42e7-840f-1c897f8c53ce" containerID="80b183e3c01a9d4bf53c9646794d1830a9a483e0ce59758cf500b7d748003844" exitCode=0 Oct 30 13:07:45.285272 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:45.285165 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-527643e2-9knlt" event={"ID":"f871ce6c-6eae-42e7-840f-1c897f8c53ce","Type":"ContainerDied","Data":"80b183e3c01a9d4bf53c9646794d1830a9a483e0ce59758cf500b7d748003844"} Oct 30 13:07:46.448099 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:46.448075 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-527643e2-9knlt" Oct 30 13:07:46.517514 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:46.517471 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:46.561701 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:46.561623 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f871ce6c-6eae-42e7-840f-1c897f8c53ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f871ce6c-6eae-42e7-840f-1c897f8c53ce-maas-api-openshift-ca\") pod \"f871ce6c-6eae-42e7-840f-1c897f8c53ce\" (UID: \"f871ce6c-6eae-42e7-840f-1c897f8c53ce\") " Oct 30 13:07:46.561885 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:46.561868 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f871ce6c-6eae-42e7-840f-1c897f8c53ce-kube-api-access-w89z2\" (UniqueName: \"kubernetes.io/projected/f871ce6c-6eae-42e7-840f-1c897f8c53ce-kube-api-access-w89z2\") pod \"f871ce6c-6eae-42e7-840f-1c897f8c53ce\" (UID: \"f871ce6c-6eae-42e7-840f-1c897f8c53ce\") " Oct 30 13:07:46.561941 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:46.561918 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f871ce6c-6eae-42e7-840f-1c897f8c53ce-maas-api-openshift-ca" pod "f871ce6c-6eae-42e7-840f-1c897f8c53ce" (UID: "f871ce6c-6eae-42e7-840f-1c897f8c53ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:46.562005 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:46.561990 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f871ce6c-6eae-42e7-840f-1c897f8c53ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f871ce6c-6eae-42e7-840f-1c897f8c53ce-maas-api-xks-ca\") pod \"f871ce6c-6eae-42e7-840f-1c897f8c53ce\" (UID: \"f871ce6c-6eae-42e7-840f-1c897f8c53ce\") " Oct 30 13:07:46.562555 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:46.562525 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f871ce6c-6eae-42e7-840f-1c897f8c53ce-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:07:46.564358 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:46.564325 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f871ce6c-6eae-42e7-840f-1c897f8c53ce-maas-api-xks-ca" pod "f871ce6c-6eae-42e7-840f-1c897f8c53ce" (UID: "f871ce6c-6eae-42e7-840f-1c897f8c53ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:46.564687 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:46.564636 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f871ce6c-6eae-42e7-840f-1c897f8c53ce-kube-api-access-w89z2" pod "f871ce6c-6eae-42e7-840f-1c897f8c53ce" (UID: "f871ce6c-6eae-42e7-840f-1c897f8c53ce"). InnerVolumeSpecName "kube-api-access-w89z2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:46.664630 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:46.664590 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f871ce6c-6eae-42e7-840f-1c897f8c53ce-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:07:46.664630 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:46.664631 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w89z2\" (UniqueName: \"kubernetes.io/projected/f871ce6c-6eae-42e7-840f-1c897f8c53ce-kube-api-access-w89z2\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:07:47.300360 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:47.300333 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-527643e2-9knlt" Oct 30 13:07:47.300531 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:47.300415 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-527643e2-9knlt" event={"ID":"f871ce6c-6eae-42e7-840f-1c897f8c53ce","Type":"ContainerDied","Data":"89bd365aabdde28ebc80367294f0fb74c6fce9afe820893d661da975145f7c49"} Oct 30 13:07:47.300531 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:47.300460 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89bd365aabdde28ebc80367294f0fb74c6fce9afe820893d661da975145f7c49" Oct 30 13:07:48.285368 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:48.285329 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:50.083696 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:50.083659 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:50.951452 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:50.951414 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:07:51.144770 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:51.144736 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec450da7-jgsxh"] Oct 30 13:07:51.145642 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:51.145623 5922 cpu_manager.go:414] "containerMap: removing container" podUID="f871ce6c-6eae-42e7-840f-1c897f8c53ce" containerName="revoke-keys" Oct 30 13:07:51.145706 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:51.145643 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f871ce6c-6eae-42e7-840f-1c897f8c53ce" containerName="revoke-keys" Oct 30 13:07:51.145706 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:51.145697 5922 cpu_manager.go:414] "containerMap: removing container" podUID="d3091a06-b26b-4793-8d07-4f1d96bc1aa4" containerName="revoke-keys" Oct 30 13:07:51.145777 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:51.145706 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3091a06-b26b-4793-8d07-4f1d96bc1aa4" containerName="revoke-keys" Oct 30 13:07:51.145842 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:51.145831 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3091a06-b26b-4793-8d07-4f1d96bc1aa4" containerName="revoke-keys" Oct 30 13:07:51.145877 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:51.145850 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="f871ce6c-6eae-42e7-840f-1c897f8c53ce" containerName="revoke-keys" Oct 30 13:07:51.150516 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:51.150496 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec450da7-jgsxh" Oct 30 13:07:51.166850 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:51.166827 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:07:51.166961 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:51.166854 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:07:51.169187 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:51.169165 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec450da7-jgsxh"] Oct 30 13:07:51.176895 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:51.176858 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:07:51.177065 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:51.177040 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:07:51.218460 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:51.218390 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bec1a3c3-11b9-4a4d-ab7a-721ced5436c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec450da7-jgsxh\" (UID: \"bec1a3c3-11b9-4a4d-ab7a-721ced5436c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec450da7-jgsxh" Oct 30 13:07:51.218605 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:51.218586 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bec1a3c3-11b9-4a4d-ab7a-721ced5436c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec450da7-jgsxh\" (UID: \"bec1a3c3-11b9-4a4d-ab7a-721ced5436c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec450da7-jgsxh" Oct 30 13:07:51.218691 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:51.218642 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqgs5\" (UniqueName: \"kubernetes.io/projected/bec1a3c3-11b9-4a4d-ab7a-721ced5436c1-kube-api-access-lqgs5\") pod \"maas-api-revoke-sub-ec450da7-jgsxh\" (UID: \"bec1a3c3-11b9-4a4d-ab7a-721ced5436c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec450da7-jgsxh" Oct 30 13:07:51.320265 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:51.320237 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bec1a3c3-11b9-4a4d-ab7a-721ced5436c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec450da7-jgsxh\" (UID: \"bec1a3c3-11b9-4a4d-ab7a-721ced5436c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec450da7-jgsxh" Oct 30 13:07:51.320358 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:51.320273 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqgs5\" (UniqueName: \"kubernetes.io/projected/bec1a3c3-11b9-4a4d-ab7a-721ced5436c1-kube-api-access-lqgs5\") pod \"maas-api-revoke-sub-ec450da7-jgsxh\" (UID: \"bec1a3c3-11b9-4a4d-ab7a-721ced5436c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec450da7-jgsxh" Oct 30 13:07:51.320421 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:51.320405 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bec1a3c3-11b9-4a4d-ab7a-721ced5436c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec450da7-jgsxh\" (UID: \"bec1a3c3-11b9-4a4d-ab7a-721ced5436c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec450da7-jgsxh" Oct 30 13:07:51.321168 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:51.321143 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bec1a3c3-11b9-4a4d-ab7a-721ced5436c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec450da7-jgsxh\" (UID: \"bec1a3c3-11b9-4a4d-ab7a-721ced5436c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec450da7-jgsxh" Oct 30 13:07:51.322623 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:51.322593 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bec1a3c3-11b9-4a4d-ab7a-721ced5436c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec450da7-jgsxh\" (UID: \"bec1a3c3-11b9-4a4d-ab7a-721ced5436c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec450da7-jgsxh" Oct 30 13:07:51.356548 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:51.356526 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqgs5\" (UniqueName: \"kubernetes.io/projected/bec1a3c3-11b9-4a4d-ab7a-721ced5436c1-kube-api-access-lqgs5\") pod \"maas-api-revoke-sub-ec450da7-jgsxh\" (UID: \"bec1a3c3-11b9-4a4d-ab7a-721ced5436c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec450da7-jgsxh" Oct 30 13:07:51.376530 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:51.376504 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f38a233c-nglhq"] Oct 30 13:07:51.381156 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:51.381121 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f38a233c-nglhq" Oct 30 13:07:51.399471 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:51.399438 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f38a233c-nglhq"] Oct 30 13:07:51.465556 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:51.465532 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec450da7-jgsxh" Oct 30 13:07:51.523716 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:51.523683 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8md26\" (UniqueName: \"kubernetes.io/projected/e3cd207c-108f-4f32-9bed-fb30557b8ffa-kube-api-access-8md26\") pod \"maas-api-revoke-sub-f38a233c-nglhq\" (UID: \"e3cd207c-108f-4f32-9bed-fb30557b8ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f38a233c-nglhq" Oct 30 13:07:51.523840 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:51.523746 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3cd207c-108f-4f32-9bed-fb30557b8ffa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f38a233c-nglhq\" (UID: \"e3cd207c-108f-4f32-9bed-fb30557b8ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f38a233c-nglhq" Oct 30 13:07:51.523915 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:51.523842 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3cd207c-108f-4f32-9bed-fb30557b8ffa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f38a233c-nglhq\" (UID: \"e3cd207c-108f-4f32-9bed-fb30557b8ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f38a233c-nglhq" Oct 30 13:07:51.643111 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:51.626550 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8md26\" (UniqueName: \"kubernetes.io/projected/e3cd207c-108f-4f32-9bed-fb30557b8ffa-kube-api-access-8md26\") pod \"maas-api-revoke-sub-f38a233c-nglhq\" (UID: \"e3cd207c-108f-4f32-9bed-fb30557b8ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f38a233c-nglhq" Oct 30 13:07:51.643111 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:51.626674 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3cd207c-108f-4f32-9bed-fb30557b8ffa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f38a233c-nglhq\" (UID: \"e3cd207c-108f-4f32-9bed-fb30557b8ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f38a233c-nglhq" Oct 30 13:07:51.643111 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:51.626726 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3cd207c-108f-4f32-9bed-fb30557b8ffa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f38a233c-nglhq\" (UID: \"e3cd207c-108f-4f32-9bed-fb30557b8ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f38a233c-nglhq" Oct 30 13:07:51.643111 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:51.627665 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3cd207c-108f-4f32-9bed-fb30557b8ffa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f38a233c-nglhq\" (UID: \"e3cd207c-108f-4f32-9bed-fb30557b8ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f38a233c-nglhq" Oct 30 13:07:51.678641 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:51.678060 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3cd207c-108f-4f32-9bed-fb30557b8ffa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f38a233c-nglhq\" (UID: \"e3cd207c-108f-4f32-9bed-fb30557b8ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f38a233c-nglhq" Oct 30 13:07:51.683145 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:51.683098 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8md26\" (UniqueName: \"kubernetes.io/projected/e3cd207c-108f-4f32-9bed-fb30557b8ffa-kube-api-access-8md26\") pod \"maas-api-revoke-sub-f38a233c-nglhq\" (UID: \"e3cd207c-108f-4f32-9bed-fb30557b8ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f38a233c-nglhq" Oct 30 13:07:51.695325 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:51.695007 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f38a233c-nglhq" Oct 30 13:07:51.775258 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:51.775224 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec450da7-jgsxh"] Oct 30 13:07:52.021810 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:52.021780 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f38a233c-nglhq"] Oct 30 13:07:52.022868 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:07:52.022835 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3cd207c_108f_4f32_9bed_fb30557b8ffa.slice/crio-816e237ea39f75c5b4f553db5ec438b0e7d54fab8fa46f003ad18205a49fb917 WatchSource:0}: Error finding container 816e237ea39f75c5b4f553db5ec438b0e7d54fab8fa46f003ad18205a49fb917: Status 404 returned error can't find the container with id 816e237ea39f75c5b4f553db5ec438b0e7d54fab8fa46f003ad18205a49fb917 Oct 30 13:07:52.367642 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:52.366471 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec450da7-jgsxh" event={"ID":"bec1a3c3-11b9-4a4d-ab7a-721ced5436c1","Type":"ContainerStarted","Data":"c3ada22ba5c2915a0332b29b037517c2437df1036cfe42e14823324979659d72"} Oct 30 13:07:52.367642 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:52.366544 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec450da7-jgsxh" event={"ID":"bec1a3c3-11b9-4a4d-ab7a-721ced5436c1","Type":"ContainerStarted","Data":"30c55c4c61514f6368eb330aeaf648d1397707b0cc757ffc034cadc8bc1e1fff"} Oct 30 13:07:52.369908 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:52.369838 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f38a233c-nglhq" event={"ID":"e3cd207c-108f-4f32-9bed-fb30557b8ffa","Type":"ContainerStarted","Data":"816e237ea39f75c5b4f553db5ec438b0e7d54fab8fa46f003ad18205a49fb917"} Oct 30 13:07:52.380597 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:52.380571 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:52.417688 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:52.417602 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec450da7-jgsxh" podStartSLOduration=1.417584808 podStartE2EDuration="1.417584808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:52.409555812 +0000 UTC m=+928.878971559" watchObservedRunningTime="2026-10-30 13:07:52.417584808 +0000 UTC m=+928.887000553" Oct 30 13:07:53.459643 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:53.453992 5922 generic.go:356] "Generic (PLEG): container finished" podID="e3cd207c-108f-4f32-9bed-fb30557b8ffa" containerID="e3f9f578c707db53651a403ee001f6b32e904ebd02a19c94c9fb82342bc9463c" exitCode=0 Oct 30 13:07:53.459643 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:53.454095 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f38a233c-nglhq" event={"ID":"e3cd207c-108f-4f32-9bed-fb30557b8ffa","Type":"ContainerDied","Data":"e3f9f578c707db53651a403ee001f6b32e904ebd02a19c94c9fb82342bc9463c"} Oct 30 13:07:53.463681 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:53.463496 5922 generic.go:356] "Generic (PLEG): container finished" podID="bec1a3c3-11b9-4a4d-ab7a-721ced5436c1" containerID="c3ada22ba5c2915a0332b29b037517c2437df1036cfe42e14823324979659d72" exitCode=0 Oct 30 13:07:53.463681 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:53.463635 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec450da7-jgsxh" event={"ID":"bec1a3c3-11b9-4a4d-ab7a-721ced5436c1","Type":"ContainerDied","Data":"c3ada22ba5c2915a0332b29b037517c2437df1036cfe42e14823324979659d72"} Oct 30 13:07:53.838700 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:07:53.835792 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:07:53.884645 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:53.879695 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" podUID="14ae0aea-473f-4ca0-bcc0-f617b9839c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded" Oct 30 13:07:53.884645 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:53.880671 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" podUID="5f353928-8cb4-49a3-b999-1b90e6cb31bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:54.037637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:54.032684 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" podUID="b487a069-f519-45a3-b1d8-f32f2431797b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 13:07:54.204373 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:54.202678 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" podUID="47517373-759d-4eeb-ab5e-6150b7e82bb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 13:07:54.457889 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:54.457817 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af2d3dcf-jn6ph"] Oct 30 13:07:54.462845 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:54.462819 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2d3dcf-jn6ph" Oct 30 13:07:54.497186 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:54.494921 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af2d3dcf-jn6ph"] Oct 30 13:07:54.572670 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:54.572566 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2a01a75-f5d3-4ef1-8215-1f72d13743df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af2d3dcf-jn6ph\" (UID: \"f2a01a75-f5d3-4ef1-8215-1f72d13743df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2d3dcf-jn6ph" Oct 30 13:07:54.572806 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:54.572674 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gfcx\" (UniqueName: \"kubernetes.io/projected/f2a01a75-f5d3-4ef1-8215-1f72d13743df-kube-api-access-5gfcx\") pod \"maas-api-revoke-sub-af2d3dcf-jn6ph\" (UID: \"f2a01a75-f5d3-4ef1-8215-1f72d13743df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2d3dcf-jn6ph" Oct 30 13:07:54.572871 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:54.572821 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2a01a75-f5d3-4ef1-8215-1f72d13743df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af2d3dcf-jn6ph\" (UID: \"f2a01a75-f5d3-4ef1-8215-1f72d13743df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2d3dcf-jn6ph" Oct 30 13:07:54.674295 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:54.674037 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2a01a75-f5d3-4ef1-8215-1f72d13743df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af2d3dcf-jn6ph\" (UID: \"f2a01a75-f5d3-4ef1-8215-1f72d13743df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2d3dcf-jn6ph" Oct 30 13:07:54.677485 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:54.677453 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2a01a75-f5d3-4ef1-8215-1f72d13743df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af2d3dcf-jn6ph\" (UID: \"f2a01a75-f5d3-4ef1-8215-1f72d13743df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2d3dcf-jn6ph" Oct 30 13:07:54.677644 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:54.676314 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2a01a75-f5d3-4ef1-8215-1f72d13743df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af2d3dcf-jn6ph\" (UID: \"f2a01a75-f5d3-4ef1-8215-1f72d13743df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2d3dcf-jn6ph" Oct 30 13:07:54.677644 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:54.677630 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gfcx\" (UniqueName: \"kubernetes.io/projected/f2a01a75-f5d3-4ef1-8215-1f72d13743df-kube-api-access-5gfcx\") pod \"maas-api-revoke-sub-af2d3dcf-jn6ph\" (UID: \"f2a01a75-f5d3-4ef1-8215-1f72d13743df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2d3dcf-jn6ph" Oct 30 13:07:54.679525 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:54.679505 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2a01a75-f5d3-4ef1-8215-1f72d13743df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af2d3dcf-jn6ph\" (UID: \"f2a01a75-f5d3-4ef1-8215-1f72d13743df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2d3dcf-jn6ph" Oct 30 13:07:54.702222 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:54.702202 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gfcx\" (UniqueName: \"kubernetes.io/projected/f2a01a75-f5d3-4ef1-8215-1f72d13743df-kube-api-access-5gfcx\") pod \"maas-api-revoke-sub-af2d3dcf-jn6ph\" (UID: \"f2a01a75-f5d3-4ef1-8215-1f72d13743df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2d3dcf-jn6ph" Oct 30 13:07:54.730479 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:54.730458 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f38a233c-nglhq" Oct 30 13:07:54.732801 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:54.732782 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec450da7-jgsxh" Oct 30 13:07:54.778833 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:54.778807 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bec1a3c3-11b9-4a4d-ab7a-721ced5436c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bec1a3c3-11b9-4a4d-ab7a-721ced5436c1-maas-api-xks-ca\") pod \"bec1a3c3-11b9-4a4d-ab7a-721ced5436c1\" (UID: \"bec1a3c3-11b9-4a4d-ab7a-721ced5436c1\") " Oct 30 13:07:54.778933 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:54.778872 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bec1a3c3-11b9-4a4d-ab7a-721ced5436c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bec1a3c3-11b9-4a4d-ab7a-721ced5436c1-maas-api-openshift-ca\") pod \"bec1a3c3-11b9-4a4d-ab7a-721ced5436c1\" (UID: \"bec1a3c3-11b9-4a4d-ab7a-721ced5436c1\") " Oct 30 13:07:54.778933 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:54.778904 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3cd207c-108f-4f32-9bed-fb30557b8ffa-kube-api-access-8md26\" (UniqueName: \"kubernetes.io/projected/e3cd207c-108f-4f32-9bed-fb30557b8ffa-kube-api-access-8md26\") pod \"e3cd207c-108f-4f32-9bed-fb30557b8ffa\" (UID: \"e3cd207c-108f-4f32-9bed-fb30557b8ffa\") " Oct 30 13:07:54.779041 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:54.778970 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3cd207c-108f-4f32-9bed-fb30557b8ffa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3cd207c-108f-4f32-9bed-fb30557b8ffa-maas-api-xks-ca\") pod \"e3cd207c-108f-4f32-9bed-fb30557b8ffa\" (UID: \"e3cd207c-108f-4f32-9bed-fb30557b8ffa\") " Oct 30 13:07:54.779041 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:54.778997 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bec1a3c3-11b9-4a4d-ab7a-721ced5436c1-kube-api-access-lqgs5\" (UniqueName: \"kubernetes.io/projected/bec1a3c3-11b9-4a4d-ab7a-721ced5436c1-kube-api-access-lqgs5\") pod \"bec1a3c3-11b9-4a4d-ab7a-721ced5436c1\" (UID: \"bec1a3c3-11b9-4a4d-ab7a-721ced5436c1\") " Oct 30 13:07:54.779149 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:54.779128 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3cd207c-108f-4f32-9bed-fb30557b8ffa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3cd207c-108f-4f32-9bed-fb30557b8ffa-maas-api-openshift-ca\") pod \"e3cd207c-108f-4f32-9bed-fb30557b8ffa\" (UID: \"e3cd207c-108f-4f32-9bed-fb30557b8ffa\") " Oct 30 13:07:54.779322 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:54.779296 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bec1a3c3-11b9-4a4d-ab7a-721ced5436c1-maas-api-openshift-ca" pod "bec1a3c3-11b9-4a4d-ab7a-721ced5436c1" (UID: "bec1a3c3-11b9-4a4d-ab7a-721ced5436c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:54.780156 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:54.779788 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bec1a3c3-11b9-4a4d-ab7a-721ced5436c1-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:07:54.780156 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:54.780122 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3cd207c-108f-4f32-9bed-fb30557b8ffa-maas-api-openshift-ca" pod "e3cd207c-108f-4f32-9bed-fb30557b8ffa" (UID: "e3cd207c-108f-4f32-9bed-fb30557b8ffa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:54.781246 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:54.781220 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3cd207c-108f-4f32-9bed-fb30557b8ffa-kube-api-access-8md26" pod "e3cd207c-108f-4f32-9bed-fb30557b8ffa" (UID: "e3cd207c-108f-4f32-9bed-fb30557b8ffa"). InnerVolumeSpecName "kube-api-access-8md26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:54.781867 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:54.781842 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bec1a3c3-11b9-4a4d-ab7a-721ced5436c1-kube-api-access-lqgs5" pod "bec1a3c3-11b9-4a4d-ab7a-721ced5436c1" (UID: "bec1a3c3-11b9-4a4d-ab7a-721ced5436c1"). InnerVolumeSpecName "kube-api-access-lqgs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:54.782635 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:54.782510 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3cd207c-108f-4f32-9bed-fb30557b8ffa-maas-api-xks-ca" pod "e3cd207c-108f-4f32-9bed-fb30557b8ffa" (UID: "e3cd207c-108f-4f32-9bed-fb30557b8ffa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:54.782635 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:54.782510 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bec1a3c3-11b9-4a4d-ab7a-721ced5436c1-maas-api-xks-ca" pod "bec1a3c3-11b9-4a4d-ab7a-721ced5436c1" (UID: "bec1a3c3-11b9-4a4d-ab7a-721ced5436c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:54.790051 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:54.790027 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2d3dcf-jn6ph" Oct 30 13:07:54.881668 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:54.881638 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bec1a3c3-11b9-4a4d-ab7a-721ced5436c1-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:07:54.881835 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:54.881816 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8md26\" (UniqueName: \"kubernetes.io/projected/e3cd207c-108f-4f32-9bed-fb30557b8ffa-kube-api-access-8md26\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:07:54.881935 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:54.881922 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3cd207c-108f-4f32-9bed-fb30557b8ffa-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:07:54.882017 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:54.882006 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqgs5\" (UniqueName: \"kubernetes.io/projected/bec1a3c3-11b9-4a4d-ab7a-721ced5436c1-kube-api-access-lqgs5\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:07:54.882116 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:54.882103 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3cd207c-108f-4f32-9bed-fb30557b8ffa-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:07:55.108194 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:55.108170 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af2d3dcf-jn6ph"] Oct 30 13:07:55.110098 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:07:55.110073 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2a01a75_f5d3_4ef1_8215_1f72d13743df.slice/crio-f6a315fbddb161914d5cff66ce0f9f67b9aa42ac9528c9ed4ec08517e501e106 WatchSource:0}: Error finding container f6a315fbddb161914d5cff66ce0f9f67b9aa42ac9528c9ed4ec08517e501e106: Status 404 returned error can't find the container with id f6a315fbddb161914d5cff66ce0f9f67b9aa42ac9528c9ed4ec08517e501e106 Oct 30 13:07:55.483239 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:55.483213 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f38a233c-nglhq" Oct 30 13:07:55.483239 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:55.483220 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f38a233c-nglhq" event={"ID":"e3cd207c-108f-4f32-9bed-fb30557b8ffa","Type":"ContainerDied","Data":"816e237ea39f75c5b4f553db5ec438b0e7d54fab8fa46f003ad18205a49fb917"} Oct 30 13:07:55.483720 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:55.483265 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="816e237ea39f75c5b4f553db5ec438b0e7d54fab8fa46f003ad18205a49fb917" Oct 30 13:07:55.485563 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:55.485543 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec450da7-jgsxh" Oct 30 13:07:55.485563 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:55.485555 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec450da7-jgsxh" event={"ID":"bec1a3c3-11b9-4a4d-ab7a-721ced5436c1","Type":"ContainerDied","Data":"30c55c4c61514f6368eb330aeaf648d1397707b0cc757ffc034cadc8bc1e1fff"} Oct 30 13:07:55.485754 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:55.485588 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30c55c4c61514f6368eb330aeaf648d1397707b0cc757ffc034cadc8bc1e1fff" Oct 30 13:07:55.487388 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:55.487363 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2d3dcf-jn6ph" event={"ID":"f2a01a75-f5d3-4ef1-8215-1f72d13743df","Type":"ContainerStarted","Data":"66bfe534c8e9d2e3560bda1d20002abcf93ca6b7e703e5e4328e132895234ea0"} Oct 30 13:07:55.487503 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:55.487412 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2d3dcf-jn6ph" event={"ID":"f2a01a75-f5d3-4ef1-8215-1f72d13743df","Type":"ContainerStarted","Data":"f6a315fbddb161914d5cff66ce0f9f67b9aa42ac9528c9ed4ec08517e501e106"} Oct 30 13:07:55.536578 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:55.536535 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2d3dcf-jn6ph" podStartSLOduration=1.536524247 podStartE2EDuration="1.536524247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:55.535004698 +0000 UTC m=+932.004420462" watchObservedRunningTime="2026-10-30 13:07:55.536524247 +0000 UTC m=+932.005939989" Oct 30 13:07:56.401425 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:56.401327 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:56.496183 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:56.496148 5922 generic.go:356] "Generic (PLEG): container finished" podID="f2a01a75-f5d3-4ef1-8215-1f72d13743df" containerID="66bfe534c8e9d2e3560bda1d20002abcf93ca6b7e703e5e4328e132895234ea0" exitCode=0 Oct 30 13:07:56.496574 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:56.496236 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2d3dcf-jn6ph" event={"ID":"f2a01a75-f5d3-4ef1-8215-1f72d13743df","Type":"ContainerDied","Data":"66bfe534c8e9d2e3560bda1d20002abcf93ca6b7e703e5e4328e132895234ea0"} Oct 30 13:07:57.649295 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:57.649272 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2d3dcf-jn6ph" Oct 30 13:07:57.718912 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:57.718882 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2a01a75-f5d3-4ef1-8215-1f72d13743df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2a01a75-f5d3-4ef1-8215-1f72d13743df-maas-api-openshift-ca\") pod \"f2a01a75-f5d3-4ef1-8215-1f72d13743df\" (UID: \"f2a01a75-f5d3-4ef1-8215-1f72d13743df\") " Oct 30 13:07:57.719146 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:57.719125 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2a01a75-f5d3-4ef1-8215-1f72d13743df-kube-api-access-5gfcx\" (UniqueName: \"kubernetes.io/projected/f2a01a75-f5d3-4ef1-8215-1f72d13743df-kube-api-access-5gfcx\") pod \"f2a01a75-f5d3-4ef1-8215-1f72d13743df\" (UID: \"f2a01a75-f5d3-4ef1-8215-1f72d13743df\") " Oct 30 13:07:57.719236 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:57.719218 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2a01a75-f5d3-4ef1-8215-1f72d13743df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2a01a75-f5d3-4ef1-8215-1f72d13743df-maas-api-xks-ca\") pod \"f2a01a75-f5d3-4ef1-8215-1f72d13743df\" (UID: \"f2a01a75-f5d3-4ef1-8215-1f72d13743df\") " Oct 30 13:07:57.719305 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:57.719234 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2a01a75-f5d3-4ef1-8215-1f72d13743df-maas-api-openshift-ca" pod "f2a01a75-f5d3-4ef1-8215-1f72d13743df" (UID: "f2a01a75-f5d3-4ef1-8215-1f72d13743df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:57.719877 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:57.719858 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2a01a75-f5d3-4ef1-8215-1f72d13743df-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:07:57.721243 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:57.721219 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2a01a75-f5d3-4ef1-8215-1f72d13743df-maas-api-xks-ca" pod "f2a01a75-f5d3-4ef1-8215-1f72d13743df" (UID: "f2a01a75-f5d3-4ef1-8215-1f72d13743df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:57.721243 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:57.721220 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2a01a75-f5d3-4ef1-8215-1f72d13743df-kube-api-access-5gfcx" pod "f2a01a75-f5d3-4ef1-8215-1f72d13743df" (UID: "f2a01a75-f5d3-4ef1-8215-1f72d13743df"). InnerVolumeSpecName "kube-api-access-5gfcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:57.821128 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:57.821063 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gfcx\" (UniqueName: \"kubernetes.io/projected/f2a01a75-f5d3-4ef1-8215-1f72d13743df-kube-api-access-5gfcx\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:07:57.821128 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:57.821089 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2a01a75-f5d3-4ef1-8215-1f72d13743df-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:07:58.315971 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:58.315936 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:58.511316 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:58.511287 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2d3dcf-jn6ph" Oct 30 13:07:58.511316 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:58.511298 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2d3dcf-jn6ph" event={"ID":"f2a01a75-f5d3-4ef1-8215-1f72d13743df","Type":"ContainerDied","Data":"f6a315fbddb161914d5cff66ce0f9f67b9aa42ac9528c9ed4ec08517e501e106"} Oct 30 13:07:58.511522 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:07:58.511404 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6a315fbddb161914d5cff66ce0f9f67b9aa42ac9528c9ed4ec08517e501e106" Oct 30 13:08:00.099236 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:00.099189 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:01.995390 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:01.995355 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:03.791133 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:03.791100 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:06.050772 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:06.050726 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" podUID="a85de36b-f8c8-4ade-a468-c114080cb312" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 13:08:06.083784 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:06.083739 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" podUID="7751779d-b082-409a-a20f-ff0ff6263808" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:06.137991 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:06.137938 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" podUID="dad5bdad-cd6f-41f1-a124-540c3524ead7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:06.158146 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:06.158073 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" podUID="dfd83cc8-6503-45a8-8260-c0edb8752471" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:06.169052 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:06.168997 5922 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" podUID="7751779d-b082-409a-a20f-ff0ff6263808" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:06.186908 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:06.185694 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" podUID="47517373-759d-4eeb-ab5e-6150b7e82bb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 13:08:06.298041 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:06.297703 5922 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" podUID="47517373-759d-4eeb-ab5e-6150b7e82bb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:06.538636 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:06.534123 5922 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" podUID="14ae0aea-473f-4ca0-bcc0-f617b9839c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:06.656751 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:06.643443 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" podUID="c0ea7fde-2ac5-432d-bb9b-534024186ac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 13:08:06.809892 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:06.808719 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" podUID="35619ab9-cbeb-45da-8506-3af63a273177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:06.866715 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:06.866663 5922 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" podUID="c0ea7fde-2ac5-432d-bb9b-534024186ac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:06.872643 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:06.870683 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" podUID="14ae0aea-473f-4ca0-bcc0-f617b9839c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:06.886641 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:06.882682 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" podUID="5f353928-8cb4-49a3-b999-1b90e6cb31bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:07.003675 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:07.003394 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" podUID="b487a069-f519-45a3-b1d8-f32f2431797b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:07.056196 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:07.054295 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" podUID="a85de36b-f8c8-4ade-a468-c114080cb312" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:07.056196 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:07.054712 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" Oct 30 13:08:07.084821 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:07.084697 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" podUID="7751779d-b082-409a-a20f-ff0ff6263808" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 13:08:07.085236 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:07.085202 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" Oct 30 13:08:07.158651 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:07.143226 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" podUID="dad5bdad-cd6f-41f1-a124-540c3524ead7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:07.158651 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:07.143631 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" Oct 30 13:08:07.259636 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:07.255970 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" podUID="47517373-759d-4eeb-ab5e-6150b7e82bb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:07.259636 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:07.256374 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" Oct 30 13:08:07.307486 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:07.307444 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" podUID="dfd83cc8-6503-45a8-8260-c0edb8752471" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 13:08:07.325492 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:07.325464 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" Oct 30 13:08:07.643633 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:07.638347 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" Oct 30 13:08:07.807715 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:07.807675 5922 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" podUID="dad5bdad-cd6f-41f1-a124-540c3524ead7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": context deadline exceeded" Oct 30 13:08:07.929823 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:07.929725 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" podUID="14ae0aea-473f-4ca0-bcc0-f617b9839c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:07.930861 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:07.930836 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" Oct 30 13:08:08.038592 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:08.037830 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" Oct 30 13:08:08.055630 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:08.051118 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" Oct 30 13:08:08.063643 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:08.061896 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" Oct 30 13:08:08.063643 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:08.061946 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" Oct 30 13:08:08.065109 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:08.065091 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" Oct 30 13:08:09.411191 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:09.411160 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc9a7925-vnz2x"] Oct 30 13:08:09.411780 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:09.411766 5922 cpu_manager.go:414] "containerMap: removing container" podUID="bec1a3c3-11b9-4a4d-ab7a-721ced5436c1" containerName="revoke-keys" Oct 30 13:08:09.411842 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:09.411781 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bec1a3c3-11b9-4a4d-ab7a-721ced5436c1" containerName="revoke-keys" Oct 30 13:08:09.411842 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:09.411796 5922 cpu_manager.go:414] "containerMap: removing container" podUID="e3cd207c-108f-4f32-9bed-fb30557b8ffa" containerName="revoke-keys" Oct 30 13:08:09.411842 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:09.411801 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3cd207c-108f-4f32-9bed-fb30557b8ffa" containerName="revoke-keys" Oct 30 13:08:09.411842 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:09.411808 5922 cpu_manager.go:414] "containerMap: removing container" podUID="f2a01a75-f5d3-4ef1-8215-1f72d13743df" containerName="revoke-keys" Oct 30 13:08:09.411842 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:09.411813 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2a01a75-f5d3-4ef1-8215-1f72d13743df" containerName="revoke-keys" Oct 30 13:08:09.411988 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:09.411912 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="bec1a3c3-11b9-4a4d-ab7a-721ced5436c1" containerName="revoke-keys" Oct 30 13:08:09.411988 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:09.411920 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3cd207c-108f-4f32-9bed-fb30557b8ffa" containerName="revoke-keys" Oct 30 13:08:09.411988 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:09.411928 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2a01a75-f5d3-4ef1-8215-1f72d13743df" containerName="revoke-keys" Oct 30 13:08:09.416881 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:09.416863 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9a7925-vnz2x" Oct 30 13:08:09.433956 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:09.433933 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc9a7925-vnz2x"] Oct 30 13:08:09.441858 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:09.441837 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:08:09.442010 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:09.441882 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:08:09.442010 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:09.441953 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:08:09.444214 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:09.444197 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:08:09.559215 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:09.559191 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrwjq\" (UniqueName: \"kubernetes.io/projected/0b985705-3a9a-4ade-8cd5-a09e6d425f6e-kube-api-access-nrwjq\") pod \"maas-api-revoke-sub-bc9a7925-vnz2x\" (UID: \"0b985705-3a9a-4ade-8cd5-a09e6d425f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9a7925-vnz2x" Oct 30 13:08:09.559423 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:09.559403 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b985705-3a9a-4ade-8cd5-a09e6d425f6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc9a7925-vnz2x\" (UID: \"0b985705-3a9a-4ade-8cd5-a09e6d425f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9a7925-vnz2x" Oct 30 13:08:09.559507 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:09.559474 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b985705-3a9a-4ade-8cd5-a09e6d425f6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc9a7925-vnz2x\" (UID: \"0b985705-3a9a-4ade-8cd5-a09e6d425f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9a7925-vnz2x" Oct 30 13:08:09.660424 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:09.660389 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b985705-3a9a-4ade-8cd5-a09e6d425f6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc9a7925-vnz2x\" (UID: \"0b985705-3a9a-4ade-8cd5-a09e6d425f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9a7925-vnz2x" Oct 30 13:08:09.660567 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:09.660536 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b985705-3a9a-4ade-8cd5-a09e6d425f6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc9a7925-vnz2x\" (UID: \"0b985705-3a9a-4ade-8cd5-a09e6d425f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9a7925-vnz2x" Oct 30 13:08:09.660661 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:09.660644 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrwjq\" (UniqueName: \"kubernetes.io/projected/0b985705-3a9a-4ade-8cd5-a09e6d425f6e-kube-api-access-nrwjq\") pod \"maas-api-revoke-sub-bc9a7925-vnz2x\" (UID: \"0b985705-3a9a-4ade-8cd5-a09e6d425f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9a7925-vnz2x" Oct 30 13:08:09.661254 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:09.661201 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b985705-3a9a-4ade-8cd5-a09e6d425f6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc9a7925-vnz2x\" (UID: \"0b985705-3a9a-4ade-8cd5-a09e6d425f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9a7925-vnz2x" Oct 30 13:08:09.662842 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:09.662819 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b985705-3a9a-4ade-8cd5-a09e6d425f6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc9a7925-vnz2x\" (UID: \"0b985705-3a9a-4ade-8cd5-a09e6d425f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9a7925-vnz2x" Oct 30 13:08:09.685605 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:09.685580 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrwjq\" (UniqueName: \"kubernetes.io/projected/0b985705-3a9a-4ade-8cd5-a09e6d425f6e-kube-api-access-nrwjq\") pod \"maas-api-revoke-sub-bc9a7925-vnz2x\" (UID: \"0b985705-3a9a-4ade-8cd5-a09e6d425f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9a7925-vnz2x" Oct 30 13:08:09.730819 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:09.730791 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9a7925-vnz2x" Oct 30 13:08:10.045075 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:10.045037 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc9a7925-vnz2x"] Oct 30 13:08:10.340509 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:10.340434 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:10.368845 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:10.368816 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08d73678-qgj7k"] Oct 30 13:08:10.373103 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:10.373085 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d73678-qgj7k" Oct 30 13:08:10.392603 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:10.392579 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08d73678-qgj7k"] Oct 30 13:08:10.470407 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:10.470364 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ebdacb-1634-472a-b8d4-2785866325df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08d73678-qgj7k\" (UID: \"11ebdacb-1634-472a-b8d4-2785866325df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d73678-qgj7k" Oct 30 13:08:10.470922 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:10.470426 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ebdacb-1634-472a-b8d4-2785866325df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08d73678-qgj7k\" (UID: \"11ebdacb-1634-472a-b8d4-2785866325df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d73678-qgj7k" Oct 30 13:08:10.470922 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:10.470543 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwg99\" (UniqueName: \"kubernetes.io/projected/11ebdacb-1634-472a-b8d4-2785866325df-kube-api-access-pwg99\") pod \"maas-api-revoke-sub-08d73678-qgj7k\" (UID: \"11ebdacb-1634-472a-b8d4-2785866325df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d73678-qgj7k" Oct 30 13:08:10.571647 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:10.571581 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ebdacb-1634-472a-b8d4-2785866325df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08d73678-qgj7k\" (UID: \"11ebdacb-1634-472a-b8d4-2785866325df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d73678-qgj7k" Oct 30 13:08:10.571830 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:10.571705 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwg99\" (UniqueName: \"kubernetes.io/projected/11ebdacb-1634-472a-b8d4-2785866325df-kube-api-access-pwg99\") pod \"maas-api-revoke-sub-08d73678-qgj7k\" (UID: \"11ebdacb-1634-472a-b8d4-2785866325df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d73678-qgj7k" Oct 30 13:08:10.571952 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:10.571935 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ebdacb-1634-472a-b8d4-2785866325df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08d73678-qgj7k\" (UID: \"11ebdacb-1634-472a-b8d4-2785866325df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d73678-qgj7k" Oct 30 13:08:10.572259 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:10.572239 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ebdacb-1634-472a-b8d4-2785866325df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08d73678-qgj7k\" (UID: \"11ebdacb-1634-472a-b8d4-2785866325df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d73678-qgj7k" Oct 30 13:08:10.574164 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:10.574138 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ebdacb-1634-472a-b8d4-2785866325df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08d73678-qgj7k\" (UID: \"11ebdacb-1634-472a-b8d4-2785866325df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d73678-qgj7k" Oct 30 13:08:10.597881 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:10.597820 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwg99\" (UniqueName: \"kubernetes.io/projected/11ebdacb-1634-472a-b8d4-2785866325df-kube-api-access-pwg99\") pod \"maas-api-revoke-sub-08d73678-qgj7k\" (UID: \"11ebdacb-1634-472a-b8d4-2785866325df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d73678-qgj7k" Oct 30 13:08:10.687270 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:10.687170 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d73678-qgj7k" Oct 30 13:08:10.925586 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:10.925557 5922 generic.go:356] "Generic (PLEG): container finished" podID="0b985705-3a9a-4ade-8cd5-a09e6d425f6e" containerID="20896b15adbf980433327fe263fac620de8abab6eff0004b0d5a0b4f3fdf6f91" exitCode=0 Oct 30 13:08:10.925719 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:10.925649 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9a7925-vnz2x" event={"ID":"0b985705-3a9a-4ade-8cd5-a09e6d425f6e","Type":"ContainerDied","Data":"20896b15adbf980433327fe263fac620de8abab6eff0004b0d5a0b4f3fdf6f91"} Oct 30 13:08:10.925719 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:10.925709 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9a7925-vnz2x" event={"ID":"0b985705-3a9a-4ade-8cd5-a09e6d425f6e","Type":"ContainerStarted","Data":"42f9d1a2e943d76e0e97e345a945ed0b27d7fcbba4adf0323c508d091895c218"} Oct 30 13:08:10.997650 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:10.997589 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08d73678-qgj7k"] Oct 30 13:08:10.999276 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:08:10.999247 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11ebdacb_1634_472a_b8d4_2785866325df.slice/crio-589cb7bf5241dd8d6bbd6484b99d50c289d1b27b892beff24354328bb2c6a19b WatchSource:0}: Error finding container 589cb7bf5241dd8d6bbd6484b99d50c289d1b27b892beff24354328bb2c6a19b: Status 404 returned error can't find the container with id 589cb7bf5241dd8d6bbd6484b99d50c289d1b27b892beff24354328bb2c6a19b Oct 30 13:08:11.935575 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:11.935543 5922 generic.go:356] "Generic (PLEG): container finished" podID="11ebdacb-1634-472a-b8d4-2785866325df" containerID="806aa4480d659f24a872a769eb5d0e866fe714dd4f2086809e70d4091fcd6515" exitCode=0 Oct 30 13:08:11.935969 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:11.935637 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d73678-qgj7k" event={"ID":"11ebdacb-1634-472a-b8d4-2785866325df","Type":"ContainerDied","Data":"806aa4480d659f24a872a769eb5d0e866fe714dd4f2086809e70d4091fcd6515"} Oct 30 13:08:11.935969 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:11.935688 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d73678-qgj7k" event={"ID":"11ebdacb-1634-472a-b8d4-2785866325df","Type":"ContainerStarted","Data":"589cb7bf5241dd8d6bbd6484b99d50c289d1b27b892beff24354328bb2c6a19b"} Oct 30 13:08:12.089115 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:12.089089 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9a7925-vnz2x" Oct 30 13:08:12.191006 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:12.190978 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b985705-3a9a-4ade-8cd5-a09e6d425f6e-kube-api-access-nrwjq\" (UniqueName: \"kubernetes.io/projected/0b985705-3a9a-4ade-8cd5-a09e6d425f6e-kube-api-access-nrwjq\") pod \"0b985705-3a9a-4ade-8cd5-a09e6d425f6e\" (UID: \"0b985705-3a9a-4ade-8cd5-a09e6d425f6e\") " Oct 30 13:08:12.191158 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:12.191070 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b985705-3a9a-4ade-8cd5-a09e6d425f6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b985705-3a9a-4ade-8cd5-a09e6d425f6e-maas-api-xks-ca\") pod \"0b985705-3a9a-4ade-8cd5-a09e6d425f6e\" (UID: \"0b985705-3a9a-4ade-8cd5-a09e6d425f6e\") " Oct 30 13:08:12.191158 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:12.191101 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b985705-3a9a-4ade-8cd5-a09e6d425f6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b985705-3a9a-4ade-8cd5-a09e6d425f6e-maas-api-openshift-ca\") pod \"0b985705-3a9a-4ade-8cd5-a09e6d425f6e\" (UID: \"0b985705-3a9a-4ade-8cd5-a09e6d425f6e\") " Oct 30 13:08:12.191481 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:12.191454 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b985705-3a9a-4ade-8cd5-a09e6d425f6e-maas-api-openshift-ca" pod "0b985705-3a9a-4ade-8cd5-a09e6d425f6e" (UID: "0b985705-3a9a-4ade-8cd5-a09e6d425f6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:12.191598 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:12.191554 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b985705-3a9a-4ade-8cd5-a09e6d425f6e-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:12.193260 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:12.193233 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b985705-3a9a-4ade-8cd5-a09e6d425f6e-maas-api-xks-ca" pod "0b985705-3a9a-4ade-8cd5-a09e6d425f6e" (UID: "0b985705-3a9a-4ade-8cd5-a09e6d425f6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:12.193365 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:12.193291 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b985705-3a9a-4ade-8cd5-a09e6d425f6e-kube-api-access-nrwjq" pod "0b985705-3a9a-4ade-8cd5-a09e6d425f6e" (UID: "0b985705-3a9a-4ade-8cd5-a09e6d425f6e"). InnerVolumeSpecName "kube-api-access-nrwjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:12.193601 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:12.193577 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:12.292912 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:12.292873 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrwjq\" (UniqueName: \"kubernetes.io/projected/0b985705-3a9a-4ade-8cd5-a09e6d425f6e-kube-api-access-nrwjq\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:12.292912 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:12.292906 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b985705-3a9a-4ade-8cd5-a09e6d425f6e-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:12.944443 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:12.944412 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9a7925-vnz2x" Oct 30 13:08:12.944443 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:12.944435 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9a7925-vnz2x" event={"ID":"0b985705-3a9a-4ade-8cd5-a09e6d425f6e","Type":"ContainerDied","Data":"42f9d1a2e943d76e0e97e345a945ed0b27d7fcbba4adf0323c508d091895c218"} Oct 30 13:08:12.944901 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:12.944480 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42f9d1a2e943d76e0e97e345a945ed0b27d7fcbba4adf0323c508d091895c218" Oct 30 13:08:13.106824 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:13.106802 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d73678-qgj7k" Oct 30 13:08:13.204366 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:13.204300 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11ebdacb-1634-472a-b8d4-2785866325df-kube-api-access-pwg99\" (UniqueName: \"kubernetes.io/projected/11ebdacb-1634-472a-b8d4-2785866325df-kube-api-access-pwg99\") pod \"11ebdacb-1634-472a-b8d4-2785866325df\" (UID: \"11ebdacb-1634-472a-b8d4-2785866325df\") " Oct 30 13:08:13.204491 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:13.204436 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11ebdacb-1634-472a-b8d4-2785866325df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ebdacb-1634-472a-b8d4-2785866325df-maas-api-xks-ca\") pod \"11ebdacb-1634-472a-b8d4-2785866325df\" (UID: \"11ebdacb-1634-472a-b8d4-2785866325df\") " Oct 30 13:08:13.204650 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:13.204602 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11ebdacb-1634-472a-b8d4-2785866325df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ebdacb-1634-472a-b8d4-2785866325df-maas-api-openshift-ca\") pod \"11ebdacb-1634-472a-b8d4-2785866325df\" (UID: \"11ebdacb-1634-472a-b8d4-2785866325df\") " Oct 30 13:08:13.204935 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:13.204908 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11ebdacb-1634-472a-b8d4-2785866325df-maas-api-openshift-ca" pod "11ebdacb-1634-472a-b8d4-2785866325df" (UID: "11ebdacb-1634-472a-b8d4-2785866325df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:13.205163 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:13.205145 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ebdacb-1634-472a-b8d4-2785866325df-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:13.206520 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:13.206496 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11ebdacb-1634-472a-b8d4-2785866325df-maas-api-xks-ca" pod "11ebdacb-1634-472a-b8d4-2785866325df" (UID: "11ebdacb-1634-472a-b8d4-2785866325df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:13.206568 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:13.206550 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11ebdacb-1634-472a-b8d4-2785866325df-kube-api-access-pwg99" pod "11ebdacb-1634-472a-b8d4-2785866325df" (UID: "11ebdacb-1634-472a-b8d4-2785866325df"). InnerVolumeSpecName "kube-api-access-pwg99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:13.306782 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:13.306755 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwg99\" (UniqueName: \"kubernetes.io/projected/11ebdacb-1634-472a-b8d4-2785866325df-kube-api-access-pwg99\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:13.306782 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:13.306780 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ebdacb-1634-472a-b8d4-2785866325df-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:13.953019 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:13.952985 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d73678-qgj7k" event={"ID":"11ebdacb-1634-472a-b8d4-2785866325df","Type":"ContainerDied","Data":"589cb7bf5241dd8d6bbd6484b99d50c289d1b27b892beff24354328bb2c6a19b"} Oct 30 13:08:13.953019 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:13.953013 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d73678-qgj7k" Oct 30 13:08:13.953426 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:13.953041 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="589cb7bf5241dd8d6bbd6484b99d50c289d1b27b892beff24354328bb2c6a19b" Oct 30 13:08:14.117193 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:14.117164 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:14.129908 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:14.129883 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d30e8ab1-5ftgc"] Oct 30 13:08:14.130496 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:14.130482 5922 cpu_manager.go:414] "containerMap: removing container" podUID="11ebdacb-1634-472a-b8d4-2785866325df" containerName="revoke-keys" Oct 30 13:08:14.130537 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:14.130498 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11ebdacb-1634-472a-b8d4-2785866325df" containerName="revoke-keys" Oct 30 13:08:14.130537 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:14.130531 5922 cpu_manager.go:414] "containerMap: removing container" podUID="0b985705-3a9a-4ade-8cd5-a09e6d425f6e" containerName="revoke-keys" Oct 30 13:08:14.130537 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:14.130536 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b985705-3a9a-4ade-8cd5-a09e6d425f6e" containerName="revoke-keys" Oct 30 13:08:14.130678 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:14.130666 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="11ebdacb-1634-472a-b8d4-2785866325df" containerName="revoke-keys" Oct 30 13:08:14.130712 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:14.130687 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b985705-3a9a-4ade-8cd5-a09e6d425f6e" containerName="revoke-keys" Oct 30 13:08:14.134266 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:14.134250 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30e8ab1-5ftgc" Oct 30 13:08:14.152783 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:14.152755 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:08:14.153168 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:14.153143 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:08:14.153304 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:14.153215 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:08:14.156131 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:14.156109 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d30e8ab1-5ftgc"] Oct 30 13:08:14.161333 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:14.161276 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:08:14.216881 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:14.216800 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f106dfed-0dcd-4413-8647-1390c98dcdf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d30e8ab1-5ftgc\" (UID: \"f106dfed-0dcd-4413-8647-1390c98dcdf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30e8ab1-5ftgc" Oct 30 13:08:14.217044 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:14.216890 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8tmw\" (UniqueName: \"kubernetes.io/projected/f106dfed-0dcd-4413-8647-1390c98dcdf8-kube-api-access-l8tmw\") pod \"maas-api-revoke-sub-d30e8ab1-5ftgc\" (UID: \"f106dfed-0dcd-4413-8647-1390c98dcdf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30e8ab1-5ftgc" Oct 30 13:08:14.217261 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:14.217202 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f106dfed-0dcd-4413-8647-1390c98dcdf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d30e8ab1-5ftgc\" (UID: \"f106dfed-0dcd-4413-8647-1390c98dcdf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30e8ab1-5ftgc" Oct 30 13:08:14.318319 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:14.318285 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f106dfed-0dcd-4413-8647-1390c98dcdf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d30e8ab1-5ftgc\" (UID: \"f106dfed-0dcd-4413-8647-1390c98dcdf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30e8ab1-5ftgc" Oct 30 13:08:14.318528 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:14.318403 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f106dfed-0dcd-4413-8647-1390c98dcdf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d30e8ab1-5ftgc\" (UID: \"f106dfed-0dcd-4413-8647-1390c98dcdf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30e8ab1-5ftgc" Oct 30 13:08:14.318528 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:14.318471 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8tmw\" (UniqueName: \"kubernetes.io/projected/f106dfed-0dcd-4413-8647-1390c98dcdf8-kube-api-access-l8tmw\") pod \"maas-api-revoke-sub-d30e8ab1-5ftgc\" (UID: \"f106dfed-0dcd-4413-8647-1390c98dcdf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30e8ab1-5ftgc" Oct 30 13:08:14.318988 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:14.318971 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f106dfed-0dcd-4413-8647-1390c98dcdf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d30e8ab1-5ftgc\" (UID: \"f106dfed-0dcd-4413-8647-1390c98dcdf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30e8ab1-5ftgc" Oct 30 13:08:14.320497 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:14.320473 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f106dfed-0dcd-4413-8647-1390c98dcdf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d30e8ab1-5ftgc\" (UID: \"f106dfed-0dcd-4413-8647-1390c98dcdf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30e8ab1-5ftgc" Oct 30 13:08:14.342870 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:14.342848 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8tmw\" (UniqueName: \"kubernetes.io/projected/f106dfed-0dcd-4413-8647-1390c98dcdf8-kube-api-access-l8tmw\") pod \"maas-api-revoke-sub-d30e8ab1-5ftgc\" (UID: \"f106dfed-0dcd-4413-8647-1390c98dcdf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30e8ab1-5ftgc" Oct 30 13:08:14.454412 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:14.454386 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30e8ab1-5ftgc" Oct 30 13:08:14.759027 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:14.758996 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d30e8ab1-5ftgc"] Oct 30 13:08:14.760277 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:08:14.760253 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf106dfed_0dcd_4413_8647_1390c98dcdf8.slice/crio-142d9d8e992a8cea66fa5efd9a379b02db1ebcba352dc81b4c9722aa9e3f42bb WatchSource:0}: Error finding container 142d9d8e992a8cea66fa5efd9a379b02db1ebcba352dc81b4c9722aa9e3f42bb: Status 404 returned error can't find the container with id 142d9d8e992a8cea66fa5efd9a379b02db1ebcba352dc81b4c9722aa9e3f42bb Oct 30 13:08:14.976292 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:14.976252 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30e8ab1-5ftgc" event={"ID":"f106dfed-0dcd-4413-8647-1390c98dcdf8","Type":"ContainerStarted","Data":"142d9d8e992a8cea66fa5efd9a379b02db1ebcba352dc81b4c9722aa9e3f42bb"} Oct 30 13:08:15.985608 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:15.985578 5922 generic.go:356] "Generic (PLEG): container finished" podID="f106dfed-0dcd-4413-8647-1390c98dcdf8" containerID="c4c8ca3e9357ec8664ac8de299bcb171d9f7aa51db2a0cccca03468b026afb90" exitCode=0 Oct 30 13:08:15.985966 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:15.985669 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30e8ab1-5ftgc" event={"ID":"f106dfed-0dcd-4413-8647-1390c98dcdf8","Type":"ContainerDied","Data":"c4c8ca3e9357ec8664ac8de299bcb171d9f7aa51db2a0cccca03468b026afb90"} Oct 30 13:08:16.008073 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:16.008045 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:16.954707 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:16.954673 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06776799-9gfcb"] Oct 30 13:08:16.958898 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:16.958883 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06776799-9gfcb" Oct 30 13:08:16.981020 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:16.980992 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06776799-9gfcb"] Oct 30 13:08:17.056714 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:17.056398 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4deafea-cb6a-4c4a-b820-fcc454879ffa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06776799-9gfcb\" (UID: \"c4deafea-cb6a-4c4a-b820-fcc454879ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06776799-9gfcb" Oct 30 13:08:17.056714 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:17.056483 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x82dt\" (UniqueName: \"kubernetes.io/projected/c4deafea-cb6a-4c4a-b820-fcc454879ffa-kube-api-access-x82dt\") pod \"maas-api-revoke-sub-06776799-9gfcb\" (UID: \"c4deafea-cb6a-4c4a-b820-fcc454879ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06776799-9gfcb" Oct 30 13:08:17.056714 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:17.056512 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4deafea-cb6a-4c4a-b820-fcc454879ffa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06776799-9gfcb\" (UID: \"c4deafea-cb6a-4c4a-b820-fcc454879ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06776799-9gfcb" Oct 30 13:08:17.152382 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:17.152352 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30e8ab1-5ftgc" Oct 30 13:08:17.157700 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:17.157680 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x82dt\" (UniqueName: \"kubernetes.io/projected/c4deafea-cb6a-4c4a-b820-fcc454879ffa-kube-api-access-x82dt\") pod \"maas-api-revoke-sub-06776799-9gfcb\" (UID: \"c4deafea-cb6a-4c4a-b820-fcc454879ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06776799-9gfcb" Oct 30 13:08:17.157792 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:17.157707 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4deafea-cb6a-4c4a-b820-fcc454879ffa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06776799-9gfcb\" (UID: \"c4deafea-cb6a-4c4a-b820-fcc454879ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06776799-9gfcb" Oct 30 13:08:17.157936 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:17.157923 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4deafea-cb6a-4c4a-b820-fcc454879ffa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06776799-9gfcb\" (UID: \"c4deafea-cb6a-4c4a-b820-fcc454879ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06776799-9gfcb" Oct 30 13:08:17.158315 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:17.158297 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4deafea-cb6a-4c4a-b820-fcc454879ffa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06776799-9gfcb\" (UID: \"c4deafea-cb6a-4c4a-b820-fcc454879ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06776799-9gfcb" Oct 30 13:08:17.160160 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:17.160141 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4deafea-cb6a-4c4a-b820-fcc454879ffa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06776799-9gfcb\" (UID: \"c4deafea-cb6a-4c4a-b820-fcc454879ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06776799-9gfcb" Oct 30 13:08:17.190257 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:17.190236 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x82dt\" (UniqueName: \"kubernetes.io/projected/c4deafea-cb6a-4c4a-b820-fcc454879ffa-kube-api-access-x82dt\") pod \"maas-api-revoke-sub-06776799-9gfcb\" (UID: \"c4deafea-cb6a-4c4a-b820-fcc454879ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06776799-9gfcb" Oct 30 13:08:17.258974 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:17.258916 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f106dfed-0dcd-4413-8647-1390c98dcdf8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f106dfed-0dcd-4413-8647-1390c98dcdf8-maas-api-openshift-ca\") pod \"f106dfed-0dcd-4413-8647-1390c98dcdf8\" (UID: \"f106dfed-0dcd-4413-8647-1390c98dcdf8\") " Oct 30 13:08:17.258974 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:17.258957 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f106dfed-0dcd-4413-8647-1390c98dcdf8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f106dfed-0dcd-4413-8647-1390c98dcdf8-maas-api-xks-ca\") pod \"f106dfed-0dcd-4413-8647-1390c98dcdf8\" (UID: \"f106dfed-0dcd-4413-8647-1390c98dcdf8\") " Oct 30 13:08:17.259127 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:17.259032 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f106dfed-0dcd-4413-8647-1390c98dcdf8-kube-api-access-l8tmw\" (UniqueName: \"kubernetes.io/projected/f106dfed-0dcd-4413-8647-1390c98dcdf8-kube-api-access-l8tmw\") pod \"f106dfed-0dcd-4413-8647-1390c98dcdf8\" (UID: \"f106dfed-0dcd-4413-8647-1390c98dcdf8\") " Oct 30 13:08:17.259300 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:17.259267 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f106dfed-0dcd-4413-8647-1390c98dcdf8-maas-api-openshift-ca" pod "f106dfed-0dcd-4413-8647-1390c98dcdf8" (UID: "f106dfed-0dcd-4413-8647-1390c98dcdf8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:17.259464 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:17.259448 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f106dfed-0dcd-4413-8647-1390c98dcdf8-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:17.260930 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:17.260901 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f106dfed-0dcd-4413-8647-1390c98dcdf8-maas-api-xks-ca" pod "f106dfed-0dcd-4413-8647-1390c98dcdf8" (UID: "f106dfed-0dcd-4413-8647-1390c98dcdf8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:17.261059 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:17.261045 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f106dfed-0dcd-4413-8647-1390c98dcdf8-kube-api-access-l8tmw" pod "f106dfed-0dcd-4413-8647-1390c98dcdf8" (UID: "f106dfed-0dcd-4413-8647-1390c98dcdf8"). InnerVolumeSpecName "kube-api-access-l8tmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:17.272232 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:17.272214 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06776799-9gfcb" Oct 30 13:08:17.360271 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:17.360246 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f106dfed-0dcd-4413-8647-1390c98dcdf8-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:17.360271 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:17.360272 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8tmw\" (UniqueName: \"kubernetes.io/projected/f106dfed-0dcd-4413-8647-1390c98dcdf8-kube-api-access-l8tmw\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:17.418344 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:17.418313 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ea6e5c7-gwnpb"] Oct 30 13:08:17.419035 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:17.419013 5922 cpu_manager.go:414] "containerMap: removing container" podUID="f106dfed-0dcd-4413-8647-1390c98dcdf8" containerName="revoke-keys" Oct 30 13:08:17.419035 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:17.419035 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f106dfed-0dcd-4413-8647-1390c98dcdf8" containerName="revoke-keys" Oct 30 13:08:17.419234 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:17.419220 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="f106dfed-0dcd-4413-8647-1390c98dcdf8" containerName="revoke-keys" Oct 30 13:08:17.423960 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:17.423941 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ea6e5c7-gwnpb" Oct 30 13:08:17.444034 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:17.444006 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ea6e5c7-gwnpb"] Oct 30 13:08:17.466081 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:17.466044 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25935958-aa4d-4c13-96de-3532eaec0422-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ea6e5c7-gwnpb\" (UID: \"25935958-aa4d-4c13-96de-3532eaec0422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ea6e5c7-gwnpb" Oct 30 13:08:17.466203 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:17.466163 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq9b4\" (UniqueName: \"kubernetes.io/projected/25935958-aa4d-4c13-96de-3532eaec0422-kube-api-access-cq9b4\") pod \"maas-api-revoke-sub-4ea6e5c7-gwnpb\" (UID: \"25935958-aa4d-4c13-96de-3532eaec0422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ea6e5c7-gwnpb" Oct 30 13:08:17.466346 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:17.466289 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25935958-aa4d-4c13-96de-3532eaec0422-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ea6e5c7-gwnpb\" (UID: \"25935958-aa4d-4c13-96de-3532eaec0422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ea6e5c7-gwnpb" Oct 30 13:08:17.567541 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:17.567464 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25935958-aa4d-4c13-96de-3532eaec0422-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ea6e5c7-gwnpb\" (UID: \"25935958-aa4d-4c13-96de-3532eaec0422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ea6e5c7-gwnpb" Oct 30 13:08:17.567698 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:17.567547 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq9b4\" (UniqueName: \"kubernetes.io/projected/25935958-aa4d-4c13-96de-3532eaec0422-kube-api-access-cq9b4\") pod \"maas-api-revoke-sub-4ea6e5c7-gwnpb\" (UID: \"25935958-aa4d-4c13-96de-3532eaec0422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ea6e5c7-gwnpb" Oct 30 13:08:17.567698 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:17.567608 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25935958-aa4d-4c13-96de-3532eaec0422-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ea6e5c7-gwnpb\" (UID: \"25935958-aa4d-4c13-96de-3532eaec0422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ea6e5c7-gwnpb" Oct 30 13:08:17.568089 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:17.568069 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25935958-aa4d-4c13-96de-3532eaec0422-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ea6e5c7-gwnpb\" (UID: \"25935958-aa4d-4c13-96de-3532eaec0422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ea6e5c7-gwnpb" Oct 30 13:08:17.570083 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:17.570052 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25935958-aa4d-4c13-96de-3532eaec0422-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ea6e5c7-gwnpb\" (UID: \"25935958-aa4d-4c13-96de-3532eaec0422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ea6e5c7-gwnpb" Oct 30 13:08:17.576965 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:17.576946 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06776799-9gfcb"] Oct 30 13:08:17.578405 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:08:17.578378 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4deafea_cb6a_4c4a_b820_fcc454879ffa.slice/crio-fe382eacf93a0ad8bcaac56375c75c06a1324cecb17b5e798d34a457d3504af9 WatchSource:0}: Error finding container fe382eacf93a0ad8bcaac56375c75c06a1324cecb17b5e798d34a457d3504af9: Status 404 returned error can't find the container with id fe382eacf93a0ad8bcaac56375c75c06a1324cecb17b5e798d34a457d3504af9 Oct 30 13:08:17.595465 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:17.595438 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq9b4\" (UniqueName: \"kubernetes.io/projected/25935958-aa4d-4c13-96de-3532eaec0422-kube-api-access-cq9b4\") pod \"maas-api-revoke-sub-4ea6e5c7-gwnpb\" (UID: \"25935958-aa4d-4c13-96de-3532eaec0422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ea6e5c7-gwnpb" Oct 30 13:08:17.742623 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:17.742595 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ea6e5c7-gwnpb" Oct 30 13:08:17.900197 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:17.900064 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:18.009254 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:18.009220 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06776799-9gfcb" event={"ID":"c4deafea-cb6a-4c4a-b820-fcc454879ffa","Type":"ContainerStarted","Data":"cc1794b035ca076e37a5542c22c52ce23336d89682c9a77a5db422bdcc88eb67"} Oct 30 13:08:18.009441 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:18.009275 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06776799-9gfcb" event={"ID":"c4deafea-cb6a-4c4a-b820-fcc454879ffa","Type":"ContainerStarted","Data":"fe382eacf93a0ad8bcaac56375c75c06a1324cecb17b5e798d34a457d3504af9"} Oct 30 13:08:18.010777 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:18.010758 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30e8ab1-5ftgc" Oct 30 13:08:18.010914 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:18.010796 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30e8ab1-5ftgc" event={"ID":"f106dfed-0dcd-4413-8647-1390c98dcdf8","Type":"ContainerDied","Data":"142d9d8e992a8cea66fa5efd9a379b02db1ebcba352dc81b4c9722aa9e3f42bb"} Oct 30 13:08:18.010914 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:18.010827 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="142d9d8e992a8cea66fa5efd9a379b02db1ebcba352dc81b4c9722aa9e3f42bb" Oct 30 13:08:18.053272 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:18.053235 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ea6e5c7-gwnpb"] Oct 30 13:08:18.054302 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:08:18.054274 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25935958_aa4d_4c13_96de_3532eaec0422.slice/crio-b306ecb4b202adf00d37a792c42b508dfe1d8753d7f6cc5d414f177a696a45f0 WatchSource:0}: Error finding container b306ecb4b202adf00d37a792c42b508dfe1d8753d7f6cc5d414f177a696a45f0: Status 404 returned error can't find the container with id b306ecb4b202adf00d37a792c42b508dfe1d8753d7f6cc5d414f177a696a45f0 Oct 30 13:08:18.061922 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:18.061876 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06776799-9gfcb" podStartSLOduration=2.061862485 podStartE2EDuration="2.061862485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:18.050666332 +0000 UTC m=+954.520082078" watchObservedRunningTime="2026-10-30 13:08:18.061862485 +0000 UTC m=+954.531278265" Oct 30 13:08:18.647727 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:18.647663 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:08:19.020338 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:19.020311 5922 generic.go:356] "Generic (PLEG): container finished" podID="25935958-aa4d-4c13-96de-3532eaec0422" containerID="45b1a552fa78db38a1c997c438e902831f2b2a6412c31492c5484324b418710e" exitCode=0 Oct 30 13:08:19.020474 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:19.020395 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ea6e5c7-gwnpb" event={"ID":"25935958-aa4d-4c13-96de-3532eaec0422","Type":"ContainerDied","Data":"45b1a552fa78db38a1c997c438e902831f2b2a6412c31492c5484324b418710e"} Oct 30 13:08:19.020474 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:19.020446 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ea6e5c7-gwnpb" event={"ID":"25935958-aa4d-4c13-96de-3532eaec0422","Type":"ContainerStarted","Data":"b306ecb4b202adf00d37a792c42b508dfe1d8753d7f6cc5d414f177a696a45f0"} Oct 30 13:08:19.022013 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:19.021984 5922 generic.go:356] "Generic (PLEG): container finished" podID="c4deafea-cb6a-4c4a-b820-fcc454879ffa" containerID="cc1794b035ca076e37a5542c22c52ce23336d89682c9a77a5db422bdcc88eb67" exitCode=0 Oct 30 13:08:19.022117 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:19.022032 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06776799-9gfcb" event={"ID":"c4deafea-cb6a-4c4a-b820-fcc454879ffa","Type":"ContainerDied","Data":"cc1794b035ca076e37a5542c22c52ce23336d89682c9a77a5db422bdcc88eb67"} Oct 30 13:08:20.032626 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:20.032574 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-688053e2-gtvtz"] Oct 30 13:08:20.042606 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:20.042581 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-688053e2-gtvtz" Oct 30 13:08:20.077052 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:20.077014 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-688053e2-gtvtz"] Oct 30 13:08:20.173633 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:20.153705 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ce92897-0aa3-4c92-a50d-3a231fae4165-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-688053e2-gtvtz\" (UID: \"5ce92897-0aa3-4c92-a50d-3a231fae4165\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-688053e2-gtvtz" Oct 30 13:08:20.173633 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:20.153983 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ce92897-0aa3-4c92-a50d-3a231fae4165-maas-api-xks-ca\") pod \"maas-api-revoke-sub-688053e2-gtvtz\" (UID: \"5ce92897-0aa3-4c92-a50d-3a231fae4165\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-688053e2-gtvtz" Oct 30 13:08:20.173633 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:20.154061 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-299w2\" (UniqueName: \"kubernetes.io/projected/5ce92897-0aa3-4c92-a50d-3a231fae4165-kube-api-access-299w2\") pod \"maas-api-revoke-sub-688053e2-gtvtz\" (UID: \"5ce92897-0aa3-4c92-a50d-3a231fae4165\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-688053e2-gtvtz" Oct 30 13:08:20.303201 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:20.303075 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ce92897-0aa3-4c92-a50d-3a231fae4165-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-688053e2-gtvtz\" (UID: \"5ce92897-0aa3-4c92-a50d-3a231fae4165\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-688053e2-gtvtz" Oct 30 13:08:20.309264 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:20.309236 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ce92897-0aa3-4c92-a50d-3a231fae4165-maas-api-xks-ca\") pod \"maas-api-revoke-sub-688053e2-gtvtz\" (UID: \"5ce92897-0aa3-4c92-a50d-3a231fae4165\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-688053e2-gtvtz" Oct 30 13:08:20.344713 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:20.332753 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ce92897-0aa3-4c92-a50d-3a231fae4165-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-688053e2-gtvtz\" (UID: \"5ce92897-0aa3-4c92-a50d-3a231fae4165\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-688053e2-gtvtz" Oct 30 13:08:20.345709 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:20.345521 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-299w2\" (UniqueName: \"kubernetes.io/projected/5ce92897-0aa3-4c92-a50d-3a231fae4165-kube-api-access-299w2\") pod \"maas-api-revoke-sub-688053e2-gtvtz\" (UID: \"5ce92897-0aa3-4c92-a50d-3a231fae4165\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-688053e2-gtvtz" Oct 30 13:08:20.410232 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:20.410156 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ce92897-0aa3-4c92-a50d-3a231fae4165-maas-api-xks-ca\") pod \"maas-api-revoke-sub-688053e2-gtvtz\" (UID: \"5ce92897-0aa3-4c92-a50d-3a231fae4165\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-688053e2-gtvtz" Oct 30 13:08:20.441380 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:20.441311 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-299w2\" (UniqueName: \"kubernetes.io/projected/5ce92897-0aa3-4c92-a50d-3a231fae4165-kube-api-access-299w2\") pod \"maas-api-revoke-sub-688053e2-gtvtz\" (UID: \"5ce92897-0aa3-4c92-a50d-3a231fae4165\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-688053e2-gtvtz" Oct 30 13:08:20.446281 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:20.446109 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-688053e2-gtvtz" Oct 30 13:08:21.080648 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:21.078695 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" podUID="47517373-759d-4eeb-ab5e-6150b7e82bb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:21.220643 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:21.219208 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:21.643637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:21.638710 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" podUID="c0ea7fde-2ac5-432d-bb9b-534024186ac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 13:08:21.675195 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:21.675155 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" podUID="a85de36b-f8c8-4ade-a468-c114080cb312" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:21.772187 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:21.770883 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06776799-9gfcb" Oct 30 13:08:22.050651 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:21.976096 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" podUID="5f353928-8cb4-49a3-b999-1b90e6cb31bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:22.050651 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:21.988115 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" podUID="35619ab9-cbeb-45da-8506-3af63a273177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:22.050651 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:22.007575 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" podUID="b487a069-f519-45a3-b1d8-f32f2431797b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:22.050651 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:22.041696 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" podUID="dad5bdad-cd6f-41f1-a124-540c3524ead7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:22.069862 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:22.068749 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" podUID="14ae0aea-473f-4ca0-bcc0-f617b9839c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:22.098650 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:22.070109 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" podUID="7751779d-b082-409a-a20f-ff0ff6263808" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:22.098650 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:22.070355 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" podUID="dfd83cc8-6503-45a8-8260-c0edb8752471" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:22.102649 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:22.100050 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4deafea-cb6a-4c4a-b820-fcc454879ffa-kube-api-access-x82dt\" (UniqueName: \"kubernetes.io/projected/c4deafea-cb6a-4c4a-b820-fcc454879ffa-kube-api-access-x82dt\") pod \"c4deafea-cb6a-4c4a-b820-fcc454879ffa\" (UID: \"c4deafea-cb6a-4c4a-b820-fcc454879ffa\") " Oct 30 13:08:22.102649 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:22.100194 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4deafea-cb6a-4c4a-b820-fcc454879ffa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4deafea-cb6a-4c4a-b820-fcc454879ffa-maas-api-openshift-ca\") pod \"c4deafea-cb6a-4c4a-b820-fcc454879ffa\" (UID: \"c4deafea-cb6a-4c4a-b820-fcc454879ffa\") " Oct 30 13:08:22.102649 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:22.100225 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4deafea-cb6a-4c4a-b820-fcc454879ffa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4deafea-cb6a-4c4a-b820-fcc454879ffa-maas-api-xks-ca\") pod \"c4deafea-cb6a-4c4a-b820-fcc454879ffa\" (UID: \"c4deafea-cb6a-4c4a-b820-fcc454879ffa\") " Oct 30 13:08:22.113660 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:22.103365 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4deafea-cb6a-4c4a-b820-fcc454879ffa-maas-api-openshift-ca" pod "c4deafea-cb6a-4c4a-b820-fcc454879ffa" (UID: "c4deafea-cb6a-4c4a-b820-fcc454879ffa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:22.127666 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:22.114319 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" podUID="47517373-759d-4eeb-ab5e-6150b7e82bb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 13:08:22.127666 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:22.119728 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" Oct 30 13:08:22.210635 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:22.210024 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4deafea-cb6a-4c4a-b820-fcc454879ffa-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:22.478138 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:22.478014 5922 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" podUID="b487a069-f519-45a3-b1d8-f32f2431797b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:22.690700 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:22.684737 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" podUID="c0ea7fde-2ac5-432d-bb9b-534024186ac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:22.790105 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:22.790076 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06776799-9gfcb" Oct 30 13:08:22.863637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:22.800634 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" podUID="a85de36b-f8c8-4ade-a468-c114080cb312" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 13:08:22.863637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:22.823001 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06776799-9gfcb" event={"ID":"c4deafea-cb6a-4c4a-b820-fcc454879ffa","Type":"ContainerDied","Data":"fe382eacf93a0ad8bcaac56375c75c06a1324cecb17b5e798d34a457d3504af9"} Oct 30 13:08:22.863637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:22.823054 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe382eacf93a0ad8bcaac56375c75c06a1324cecb17b5e798d34a457d3504af9" Oct 30 13:08:22.863637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:22.823138 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" Oct 30 13:08:22.863637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:22.823208 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" Oct 30 13:08:22.934559 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:22.870393 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4deafea-cb6a-4c4a-b820-fcc454879ffa-maas-api-xks-ca" pod "c4deafea-cb6a-4c4a-b820-fcc454879ffa" (UID: "c4deafea-cb6a-4c4a-b820-fcc454879ffa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:23.005657 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.000541 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" podUID="35619ab9-cbeb-45da-8506-3af63a273177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:23.024636 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.006590 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" Oct 30 13:08:23.047679 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.024637 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" podUID="5f353928-8cb4-49a3-b999-1b90e6cb31bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:23.190299 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.045216 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4deafea-cb6a-4c4a-b820-fcc454879ffa-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:23.190866 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.106149 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" podUID="b487a069-f519-45a3-b1d8-f32f2431797b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:23.191682 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.106255 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" podUID="dad5bdad-cd6f-41f1-a124-540c3524ead7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 13:08:23.193349 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.106350 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" podUID="14ae0aea-473f-4ca0-bcc0-f617b9839c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded" Oct 30 13:08:23.245746 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.106455 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" podUID="7751779d-b082-409a-a20f-ff0ff6263808" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 13:08:23.245746 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.106536 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" podUID="dfd83cc8-6503-45a8-8260-c0edb8752471" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 13:08:23.246319 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.118027 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-688053e2-gtvtz"] Oct 30 13:08:23.246570 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.246454 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" Oct 30 13:08:23.246570 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.246532 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" Oct 30 13:08:23.246769 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.246594 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" Oct 30 13:08:23.246769 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.246671 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" Oct 30 13:08:23.248445 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.246949 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" Oct 30 13:08:23.248445 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.247022 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" Oct 30 13:08:23.252630 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.183202 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" podUID="47517373-759d-4eeb-ab5e-6150b7e82bb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:23.297633 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.292578 5922 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" podUID="35619ab9-cbeb-45da-8506-3af63a273177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded" Oct 30 13:08:23.331649 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.331510 5922 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" podUID="a85de36b-f8c8-4ade-a468-c114080cb312" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": context deadline exceeded" Oct 30 13:08:23.352871 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.352838 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4deafea-cb6a-4c4a-b820-fcc454879ffa-kube-api-access-x82dt" pod "c4deafea-cb6a-4c4a-b820-fcc454879ffa" (UID: "c4deafea-cb6a-4c4a-b820-fcc454879ffa"). InnerVolumeSpecName "kube-api-access-x82dt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:23.396636 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.363428 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4deafea-cb6a-4c4a-b820-fcc454879ffa-kube-api-access-x82dt\" (UniqueName: \"kubernetes.io/projected/c4deafea-cb6a-4c4a-b820-fcc454879ffa-kube-api-access-x82dt\") pod \"c4deafea-cb6a-4c4a-b820-fcc454879ffa\" (UID: \"c4deafea-cb6a-4c4a-b820-fcc454879ffa\") " Oct 30 13:08:23.488750 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:08:23.365143 5922 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/c4deafea-cb6a-4c4a-b820-fcc454879ffa/volumes/kubernetes.io~projected/kube-api-access-x82dt Oct 30 13:08:23.488750 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.400370 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4deafea-cb6a-4c4a-b820-fcc454879ffa-kube-api-access-x82dt" pod "c4deafea-cb6a-4c4a-b820-fcc454879ffa" (UID: "c4deafea-cb6a-4c4a-b820-fcc454879ffa"). InnerVolumeSpecName "kube-api-access-x82dt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:23.603641 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.599119 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x82dt\" (UniqueName: \"kubernetes.io/projected/c4deafea-cb6a-4c4a-b820-fcc454879ffa-kube-api-access-x82dt\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:23.697815 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.697779 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" podUID="c0ea7fde-2ac5-432d-bb9b-534024186ac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 13:08:23.751229 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:08:23.751195 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:08:23.785752 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.785702 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" Oct 30 13:08:23.826749 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.820576 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" Oct 30 13:08:23.826749 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.821542 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" Oct 30 13:08:23.826749 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.821680 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" Oct 30 13:08:23.826749 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.821745 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" Oct 30 13:08:23.826749 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.821784 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" Oct 30 13:08:23.826749 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.821842 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" Oct 30 13:08:23.826749 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.821895 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" Oct 30 13:08:23.826749 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.823033 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ea6e5c7-gwnpb" Oct 30 13:08:23.826749 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.823623 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" Oct 30 13:08:23.847301 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.847229 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" Oct 30 13:08:23.856028 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.855860 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ea6e5c7-gwnpb" Oct 30 13:08:23.856028 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.855887 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ea6e5c7-gwnpb" event={"ID":"25935958-aa4d-4c13-96de-3532eaec0422","Type":"ContainerDied","Data":"b306ecb4b202adf00d37a792c42b508dfe1d8753d7f6cc5d414f177a696a45f0"} Oct 30 13:08:23.856028 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.855953 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b306ecb4b202adf00d37a792c42b508dfe1d8753d7f6cc5d414f177a696a45f0" Oct 30 13:08:23.859209 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.859138 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-688053e2-gtvtz" event={"ID":"5ce92897-0aa3-4c92-a50d-3a231fae4165","Type":"ContainerStarted","Data":"3ddab92bcc3da058c0d71efd0a3ca7961ac8428af9866f088922b05d6bb512e3"} Oct 30 13:08:23.970777 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.970743 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25935958-aa4d-4c13-96de-3532eaec0422-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25935958-aa4d-4c13-96de-3532eaec0422-maas-api-xks-ca\") pod \"25935958-aa4d-4c13-96de-3532eaec0422\" (UID: \"25935958-aa4d-4c13-96de-3532eaec0422\") " Oct 30 13:08:23.970961 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.970905 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25935958-aa4d-4c13-96de-3532eaec0422-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25935958-aa4d-4c13-96de-3532eaec0422-maas-api-openshift-ca\") pod \"25935958-aa4d-4c13-96de-3532eaec0422\" (UID: \"25935958-aa4d-4c13-96de-3532eaec0422\") " Oct 30 13:08:23.970961 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.970940 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25935958-aa4d-4c13-96de-3532eaec0422-kube-api-access-cq9b4\" (UniqueName: \"kubernetes.io/projected/25935958-aa4d-4c13-96de-3532eaec0422-kube-api-access-cq9b4\") pod \"25935958-aa4d-4c13-96de-3532eaec0422\" (UID: \"25935958-aa4d-4c13-96de-3532eaec0422\") " Oct 30 13:08:23.973402 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.971260 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25935958-aa4d-4c13-96de-3532eaec0422-maas-api-openshift-ca" pod "25935958-aa4d-4c13-96de-3532eaec0422" (UID: "25935958-aa4d-4c13-96de-3532eaec0422"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:23.973402 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.971977 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25935958-aa4d-4c13-96de-3532eaec0422-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:23.973827 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.973796 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25935958-aa4d-4c13-96de-3532eaec0422-kube-api-access-cq9b4" pod "25935958-aa4d-4c13-96de-3532eaec0422" (UID: "25935958-aa4d-4c13-96de-3532eaec0422"). InnerVolumeSpecName "kube-api-access-cq9b4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:23.975161 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:23.975139 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25935958-aa4d-4c13-96de-3532eaec0422-maas-api-xks-ca" pod "25935958-aa4d-4c13-96de-3532eaec0422" (UID: "25935958-aa4d-4c13-96de-3532eaec0422"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:24.073853 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:24.073823 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq9b4\" (UniqueName: \"kubernetes.io/projected/25935958-aa4d-4c13-96de-3532eaec0422-kube-api-access-cq9b4\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:24.073853 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:24.073854 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25935958-aa4d-4c13-96de-3532eaec0422-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:24.869280 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:24.869200 5922 generic.go:356] "Generic (PLEG): container finished" podID="5ce92897-0aa3-4c92-a50d-3a231fae4165" containerID="a91d664160ce147cdbf0d8b2deb4c0cfd89355b3f2013e79192459f3aa4eb73b" exitCode=0 Oct 30 13:08:24.869683 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:24.869285 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-688053e2-gtvtz" event={"ID":"5ce92897-0aa3-4c92-a50d-3a231fae4165","Type":"ContainerDied","Data":"a91d664160ce147cdbf0d8b2deb4c0cfd89355b3f2013e79192459f3aa4eb73b"} Oct 30 13:08:24.959883 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:24.959851 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbfecdd2-kw7hg"] Oct 30 13:08:24.960454 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:24.960439 5922 cpu_manager.go:414] "containerMap: removing container" podUID="25935958-aa4d-4c13-96de-3532eaec0422" containerName="revoke-keys" Oct 30 13:08:24.960497 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:24.960456 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25935958-aa4d-4c13-96de-3532eaec0422" containerName="revoke-keys" Oct 30 13:08:24.960497 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:24.960482 5922 cpu_manager.go:414] "containerMap: removing container" podUID="c4deafea-cb6a-4c4a-b820-fcc454879ffa" containerName="revoke-keys" Oct 30 13:08:24.960497 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:24.960487 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4deafea-cb6a-4c4a-b820-fcc454879ffa" containerName="revoke-keys" Oct 30 13:08:24.960600 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:24.960584 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="25935958-aa4d-4c13-96de-3532eaec0422" containerName="revoke-keys" Oct 30 13:08:24.960600 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:24.960598 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4deafea-cb6a-4c4a-b820-fcc454879ffa" containerName="revoke-keys" Oct 30 13:08:24.964256 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:24.964240 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfecdd2-kw7hg" Oct 30 13:08:24.984843 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:24.984819 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbfecdd2-kw7hg"] Oct 30 13:08:25.086959 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:25.086935 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3b9423e-208d-45c0-bd24-bec761499e36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbfecdd2-kw7hg\" (UID: \"b3b9423e-208d-45c0-bd24-bec761499e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfecdd2-kw7hg" Oct 30 13:08:25.087105 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:25.086996 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3b9423e-208d-45c0-bd24-bec761499e36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbfecdd2-kw7hg\" (UID: \"b3b9423e-208d-45c0-bd24-bec761499e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfecdd2-kw7hg" Oct 30 13:08:25.087105 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:25.087091 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtpdh\" (UniqueName: \"kubernetes.io/projected/b3b9423e-208d-45c0-bd24-bec761499e36-kube-api-access-wtpdh\") pod \"maas-api-revoke-sub-cbfecdd2-kw7hg\" (UID: \"b3b9423e-208d-45c0-bd24-bec761499e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfecdd2-kw7hg" Oct 30 13:08:25.188739 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:25.188678 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3b9423e-208d-45c0-bd24-bec761499e36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbfecdd2-kw7hg\" (UID: \"b3b9423e-208d-45c0-bd24-bec761499e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfecdd2-kw7hg" Oct 30 13:08:25.188855 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:25.188734 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3b9423e-208d-45c0-bd24-bec761499e36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbfecdd2-kw7hg\" (UID: \"b3b9423e-208d-45c0-bd24-bec761499e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfecdd2-kw7hg" Oct 30 13:08:25.188855 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:25.188775 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtpdh\" (UniqueName: \"kubernetes.io/projected/b3b9423e-208d-45c0-bd24-bec761499e36-kube-api-access-wtpdh\") pod \"maas-api-revoke-sub-cbfecdd2-kw7hg\" (UID: \"b3b9423e-208d-45c0-bd24-bec761499e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfecdd2-kw7hg" Oct 30 13:08:25.189412 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:25.189390 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3b9423e-208d-45c0-bd24-bec761499e36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbfecdd2-kw7hg\" (UID: \"b3b9423e-208d-45c0-bd24-bec761499e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfecdd2-kw7hg" Oct 30 13:08:25.191070 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:25.191050 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3b9423e-208d-45c0-bd24-bec761499e36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbfecdd2-kw7hg\" (UID: \"b3b9423e-208d-45c0-bd24-bec761499e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfecdd2-kw7hg" Oct 30 13:08:25.213809 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:25.213789 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtpdh\" (UniqueName: \"kubernetes.io/projected/b3b9423e-208d-45c0-bd24-bec761499e36-kube-api-access-wtpdh\") pod \"maas-api-revoke-sub-cbfecdd2-kw7hg\" (UID: \"b3b9423e-208d-45c0-bd24-bec761499e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfecdd2-kw7hg" Oct 30 13:08:25.283906 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:25.283884 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfecdd2-kw7hg" Oct 30 13:08:25.590855 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:25.590829 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbfecdd2-kw7hg"] Oct 30 13:08:25.593193 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:08:25.593163 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3b9423e_208d_45c0_bd24_bec761499e36.slice/crio-f5907486f3097c9863058c2b983dfd93335eae9ebe30479c90131aeda1148f7c WatchSource:0}: Error finding container f5907486f3097c9863058c2b983dfd93335eae9ebe30479c90131aeda1148f7c: Status 404 returned error can't find the container with id f5907486f3097c9863058c2b983dfd93335eae9ebe30479c90131aeda1148f7c Oct 30 13:08:25.882732 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:25.882707 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfecdd2-kw7hg" event={"ID":"b3b9423e-208d-45c0-bd24-bec761499e36","Type":"ContainerStarted","Data":"f5907486f3097c9863058c2b983dfd93335eae9ebe30479c90131aeda1148f7c"} Oct 30 13:08:26.038471 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:26.038438 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-688053e2-gtvtz" Oct 30 13:08:26.048195 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:26.048169 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8462ff11-htt87"] Oct 30 13:08:26.048763 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:26.048750 5922 cpu_manager.go:414] "containerMap: removing container" podUID="5ce92897-0aa3-4c92-a50d-3a231fae4165" containerName="revoke-keys" Oct 30 13:08:26.048802 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:26.048764 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ce92897-0aa3-4c92-a50d-3a231fae4165" containerName="revoke-keys" Oct 30 13:08:26.048923 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:26.048906 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ce92897-0aa3-4c92-a50d-3a231fae4165" containerName="revoke-keys" Oct 30 13:08:26.052335 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:26.052315 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8462ff11-htt87" Oct 30 13:08:26.071714 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:26.071690 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8462ff11-htt87"] Oct 30 13:08:26.109873 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:26.109845 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:26.200651 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:26.200623 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ce92897-0aa3-4c92-a50d-3a231fae4165-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ce92897-0aa3-4c92-a50d-3a231fae4165-maas-api-xks-ca\") pod \"5ce92897-0aa3-4c92-a50d-3a231fae4165\" (UID: \"5ce92897-0aa3-4c92-a50d-3a231fae4165\") " Oct 30 13:08:26.200788 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:26.200675 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ce92897-0aa3-4c92-a50d-3a231fae4165-kube-api-access-299w2\" (UniqueName: \"kubernetes.io/projected/5ce92897-0aa3-4c92-a50d-3a231fae4165-kube-api-access-299w2\") pod \"5ce92897-0aa3-4c92-a50d-3a231fae4165\" (UID: \"5ce92897-0aa3-4c92-a50d-3a231fae4165\") " Oct 30 13:08:26.200788 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:26.200706 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ce92897-0aa3-4c92-a50d-3a231fae4165-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ce92897-0aa3-4c92-a50d-3a231fae4165-maas-api-openshift-ca\") pod \"5ce92897-0aa3-4c92-a50d-3a231fae4165\" (UID: \"5ce92897-0aa3-4c92-a50d-3a231fae4165\") " Oct 30 13:08:26.201082 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:26.201054 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/055fc516-1042-4221-aff3-b07b975bec3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8462ff11-htt87\" (UID: \"055fc516-1042-4221-aff3-b07b975bec3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8462ff11-htt87" Oct 30 13:08:26.201221 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:26.201084 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ce92897-0aa3-4c92-a50d-3a231fae4165-maas-api-openshift-ca" pod "5ce92897-0aa3-4c92-a50d-3a231fae4165" (UID: "5ce92897-0aa3-4c92-a50d-3a231fae4165"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:26.201221 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:26.201102 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlpgb\" (UniqueName: \"kubernetes.io/projected/055fc516-1042-4221-aff3-b07b975bec3d-kube-api-access-nlpgb\") pod \"maas-api-revoke-sub-8462ff11-htt87\" (UID: \"055fc516-1042-4221-aff3-b07b975bec3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8462ff11-htt87" Oct 30 13:08:26.201221 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:26.201200 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/055fc516-1042-4221-aff3-b07b975bec3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8462ff11-htt87\" (UID: \"055fc516-1042-4221-aff3-b07b975bec3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8462ff11-htt87" Oct 30 13:08:26.201435 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:26.201422 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ce92897-0aa3-4c92-a50d-3a231fae4165-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:26.202738 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:26.202715 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ce92897-0aa3-4c92-a50d-3a231fae4165-kube-api-access-299w2" pod "5ce92897-0aa3-4c92-a50d-3a231fae4165" (UID: "5ce92897-0aa3-4c92-a50d-3a231fae4165"). InnerVolumeSpecName "kube-api-access-299w2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:26.202738 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:26.202728 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ce92897-0aa3-4c92-a50d-3a231fae4165-maas-api-xks-ca" pod "5ce92897-0aa3-4c92-a50d-3a231fae4165" (UID: "5ce92897-0aa3-4c92-a50d-3a231fae4165"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:26.302314 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:26.302279 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/055fc516-1042-4221-aff3-b07b975bec3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8462ff11-htt87\" (UID: \"055fc516-1042-4221-aff3-b07b975bec3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8462ff11-htt87" Oct 30 13:08:26.302484 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:26.302470 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/055fc516-1042-4221-aff3-b07b975bec3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8462ff11-htt87\" (UID: \"055fc516-1042-4221-aff3-b07b975bec3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8462ff11-htt87" Oct 30 13:08:26.302553 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:26.302494 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlpgb\" (UniqueName: \"kubernetes.io/projected/055fc516-1042-4221-aff3-b07b975bec3d-kube-api-access-nlpgb\") pod \"maas-api-revoke-sub-8462ff11-htt87\" (UID: \"055fc516-1042-4221-aff3-b07b975bec3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8462ff11-htt87" Oct 30 13:08:26.302608 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:26.302555 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ce92897-0aa3-4c92-a50d-3a231fae4165-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:26.302608 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:26.302566 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-299w2\" (UniqueName: \"kubernetes.io/projected/5ce92897-0aa3-4c92-a50d-3a231fae4165-kube-api-access-299w2\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:26.303128 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:26.303103 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/055fc516-1042-4221-aff3-b07b975bec3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8462ff11-htt87\" (UID: \"055fc516-1042-4221-aff3-b07b975bec3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8462ff11-htt87" Oct 30 13:08:26.305175 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:26.305152 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/055fc516-1042-4221-aff3-b07b975bec3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8462ff11-htt87\" (UID: \"055fc516-1042-4221-aff3-b07b975bec3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8462ff11-htt87" Oct 30 13:08:26.327569 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:26.327547 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlpgb\" (UniqueName: \"kubernetes.io/projected/055fc516-1042-4221-aff3-b07b975bec3d-kube-api-access-nlpgb\") pod \"maas-api-revoke-sub-8462ff11-htt87\" (UID: \"055fc516-1042-4221-aff3-b07b975bec3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8462ff11-htt87" Oct 30 13:08:26.365512 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:26.365490 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8462ff11-htt87" Oct 30 13:08:26.678473 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:26.678441 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8462ff11-htt87"] Oct 30 13:08:26.679365 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:08:26.679338 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod055fc516_1042_4221_aff3_b07b975bec3d.slice/crio-ce8189ebd9cfa0760468f3b1f1d413785d7e57b23d39aefc1cfd5a42aa98390a WatchSource:0}: Error finding container ce8189ebd9cfa0760468f3b1f1d413785d7e57b23d39aefc1cfd5a42aa98390a: Status 404 returned error can't find the container with id ce8189ebd9cfa0760468f3b1f1d413785d7e57b23d39aefc1cfd5a42aa98390a Oct 30 13:08:26.891919 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:26.891885 5922 generic.go:356] "Generic (PLEG): container finished" podID="b3b9423e-208d-45c0-bd24-bec761499e36" containerID="c721f00e5b7d6b2121217cc7a7a193da8ff6c4a0f36e83cca00b2d1186b44297" exitCode=0 Oct 30 13:08:26.892299 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:26.891973 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfecdd2-kw7hg" event={"ID":"b3b9423e-208d-45c0-bd24-bec761499e36","Type":"ContainerDied","Data":"c721f00e5b7d6b2121217cc7a7a193da8ff6c4a0f36e83cca00b2d1186b44297"} Oct 30 13:08:26.893552 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:26.893525 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-688053e2-gtvtz" event={"ID":"5ce92897-0aa3-4c92-a50d-3a231fae4165","Type":"ContainerDied","Data":"3ddab92bcc3da058c0d71efd0a3ca7961ac8428af9866f088922b05d6bb512e3"} Oct 30 13:08:26.893552 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:26.893538 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-688053e2-gtvtz" Oct 30 13:08:26.893728 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:26.893565 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ddab92bcc3da058c0d71efd0a3ca7961ac8428af9866f088922b05d6bb512e3" Oct 30 13:08:26.894960 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:26.894942 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8462ff11-htt87" event={"ID":"055fc516-1042-4221-aff3-b07b975bec3d","Type":"ContainerStarted","Data":"ce8189ebd9cfa0760468f3b1f1d413785d7e57b23d39aefc1cfd5a42aa98390a"} Oct 30 13:08:27.747986 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:27.747917 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-925dbcef-gx9pd"] Oct 30 13:08:27.752265 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:27.752245 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-925dbcef-gx9pd" Oct 30 13:08:27.770683 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:27.770661 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-925dbcef-gx9pd"] Oct 30 13:08:27.903363 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:27.903335 5922 generic.go:356] "Generic (PLEG): container finished" podID="055fc516-1042-4221-aff3-b07b975bec3d" containerID="60744197d6e360df5966c56420fec4d883da73fcca17c47fd9dcad6b23348335" exitCode=0 Oct 30 13:08:27.903761 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:27.903427 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8462ff11-htt87" event={"ID":"055fc516-1042-4221-aff3-b07b975bec3d","Type":"ContainerDied","Data":"60744197d6e360df5966c56420fec4d883da73fcca17c47fd9dcad6b23348335"} Oct 30 13:08:27.920235 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:27.920198 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxgf7\" (UniqueName: \"kubernetes.io/projected/1eee749f-1f8c-4dc8-80b8-e0288d52e2b1-kube-api-access-sxgf7\") pod \"maas-api-revoke-sub-925dbcef-gx9pd\" (UID: \"1eee749f-1f8c-4dc8-80b8-e0288d52e2b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925dbcef-gx9pd" Oct 30 13:08:27.920378 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:27.920296 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eee749f-1f8c-4dc8-80b8-e0288d52e2b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-925dbcef-gx9pd\" (UID: \"1eee749f-1f8c-4dc8-80b8-e0288d52e2b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925dbcef-gx9pd" Oct 30 13:08:27.920438 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:27.920401 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eee749f-1f8c-4dc8-80b8-e0288d52e2b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-925dbcef-gx9pd\" (UID: \"1eee749f-1f8c-4dc8-80b8-e0288d52e2b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925dbcef-gx9pd" Oct 30 13:08:28.022421 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:28.022337 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eee749f-1f8c-4dc8-80b8-e0288d52e2b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-925dbcef-gx9pd\" (UID: \"1eee749f-1f8c-4dc8-80b8-e0288d52e2b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925dbcef-gx9pd" Oct 30 13:08:28.022581 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:28.022558 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxgf7\" (UniqueName: \"kubernetes.io/projected/1eee749f-1f8c-4dc8-80b8-e0288d52e2b1-kube-api-access-sxgf7\") pod \"maas-api-revoke-sub-925dbcef-gx9pd\" (UID: \"1eee749f-1f8c-4dc8-80b8-e0288d52e2b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925dbcef-gx9pd" Oct 30 13:08:28.022674 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:28.022649 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eee749f-1f8c-4dc8-80b8-e0288d52e2b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-925dbcef-gx9pd\" (UID: \"1eee749f-1f8c-4dc8-80b8-e0288d52e2b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925dbcef-gx9pd" Oct 30 13:08:28.023078 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:28.023050 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eee749f-1f8c-4dc8-80b8-e0288d52e2b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-925dbcef-gx9pd\" (UID: \"1eee749f-1f8c-4dc8-80b8-e0288d52e2b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925dbcef-gx9pd" Oct 30 13:08:28.025159 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:28.025135 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eee749f-1f8c-4dc8-80b8-e0288d52e2b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-925dbcef-gx9pd\" (UID: \"1eee749f-1f8c-4dc8-80b8-e0288d52e2b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925dbcef-gx9pd" Oct 30 13:08:28.046793 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:28.046772 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxgf7\" (UniqueName: \"kubernetes.io/projected/1eee749f-1f8c-4dc8-80b8-e0288d52e2b1-kube-api-access-sxgf7\") pod \"maas-api-revoke-sub-925dbcef-gx9pd\" (UID: \"1eee749f-1f8c-4dc8-80b8-e0288d52e2b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925dbcef-gx9pd" Oct 30 13:08:28.067032 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:28.067010 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-925dbcef-gx9pd" Oct 30 13:08:28.069526 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:28.069509 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfecdd2-kw7hg" Oct 30 13:08:28.224792 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:28.224760 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3b9423e-208d-45c0-bd24-bec761499e36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3b9423e-208d-45c0-bd24-bec761499e36-maas-api-openshift-ca\") pod \"b3b9423e-208d-45c0-bd24-bec761499e36\" (UID: \"b3b9423e-208d-45c0-bd24-bec761499e36\") " Oct 30 13:08:28.224792 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:28.224794 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3b9423e-208d-45c0-bd24-bec761499e36-kube-api-access-wtpdh\" (UniqueName: \"kubernetes.io/projected/b3b9423e-208d-45c0-bd24-bec761499e36-kube-api-access-wtpdh\") pod \"b3b9423e-208d-45c0-bd24-bec761499e36\" (UID: \"b3b9423e-208d-45c0-bd24-bec761499e36\") " Oct 30 13:08:28.225037 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:28.224973 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3b9423e-208d-45c0-bd24-bec761499e36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3b9423e-208d-45c0-bd24-bec761499e36-maas-api-xks-ca\") pod \"b3b9423e-208d-45c0-bd24-bec761499e36\" (UID: \"b3b9423e-208d-45c0-bd24-bec761499e36\") " Oct 30 13:08:28.225188 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:28.225165 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3b9423e-208d-45c0-bd24-bec761499e36-maas-api-openshift-ca" pod "b3b9423e-208d-45c0-bd24-bec761499e36" (UID: "b3b9423e-208d-45c0-bd24-bec761499e36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:28.225456 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:28.225438 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3b9423e-208d-45c0-bd24-bec761499e36-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:28.226878 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:28.226856 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3b9423e-208d-45c0-bd24-bec761499e36-kube-api-access-wtpdh" pod "b3b9423e-208d-45c0-bd24-bec761499e36" (UID: "b3b9423e-208d-45c0-bd24-bec761499e36"). InnerVolumeSpecName "kube-api-access-wtpdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:28.227115 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:28.227095 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3b9423e-208d-45c0-bd24-bec761499e36-maas-api-xks-ca" pod "b3b9423e-208d-45c0-bd24-bec761499e36" (UID: "b3b9423e-208d-45c0-bd24-bec761499e36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:28.326964 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:28.326903 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3b9423e-208d-45c0-bd24-bec761499e36-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:28.326964 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:28.326927 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtpdh\" (UniqueName: \"kubernetes.io/projected/b3b9423e-208d-45c0-bd24-bec761499e36-kube-api-access-wtpdh\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:28.338105 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:28.338075 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:28.393694 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:28.393663 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-925dbcef-gx9pd"] Oct 30 13:08:28.912190 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:28.912145 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfecdd2-kw7hg" event={"ID":"b3b9423e-208d-45c0-bd24-bec761499e36","Type":"ContainerDied","Data":"f5907486f3097c9863058c2b983dfd93335eae9ebe30479c90131aeda1148f7c"} Oct 30 13:08:28.912600 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:28.912209 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5907486f3097c9863058c2b983dfd93335eae9ebe30479c90131aeda1148f7c" Oct 30 13:08:28.912600 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:28.912163 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfecdd2-kw7hg" Oct 30 13:08:28.913878 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:28.913851 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-925dbcef-gx9pd" event={"ID":"1eee749f-1f8c-4dc8-80b8-e0288d52e2b1","Type":"ContainerStarted","Data":"82e57b9fe02b47147dfbf945e45868f8faeeaf2150b54b69a09b616df7b24112"} Oct 30 13:08:28.914000 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:28.913901 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-925dbcef-gx9pd" event={"ID":"1eee749f-1f8c-4dc8-80b8-e0288d52e2b1","Type":"ContainerStarted","Data":"aa2dd0c924a037763a2c0ab640638ece066d64dd7797d055197157eb0813fc20"} Oct 30 13:08:28.958288 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:28.958233 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-925dbcef-gx9pd" podStartSLOduration=1.958214911 podStartE2EDuration="1.958214911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:28.954415272 +0000 UTC m=+965.423831058" watchObservedRunningTime="2026-10-30 13:08:28.958214911 +0000 UTC m=+965.427630657" Oct 30 13:08:29.090416 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.090390 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8462ff11-htt87" Oct 30 13:08:29.222854 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.222783 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5101680f-4lrqb"] Oct 30 13:08:29.223400 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.223385 5922 cpu_manager.go:414] "containerMap: removing container" podUID="b3b9423e-208d-45c0-bd24-bec761499e36" containerName="revoke-keys" Oct 30 13:08:29.223449 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.223401 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3b9423e-208d-45c0-bd24-bec761499e36" containerName="revoke-keys" Oct 30 13:08:29.223449 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.223430 5922 cpu_manager.go:414] "containerMap: removing container" podUID="055fc516-1042-4221-aff3-b07b975bec3d" containerName="revoke-keys" Oct 30 13:08:29.223449 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.223435 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="055fc516-1042-4221-aff3-b07b975bec3d" containerName="revoke-keys" Oct 30 13:08:29.223538 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.223520 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="055fc516-1042-4221-aff3-b07b975bec3d" containerName="revoke-keys" Oct 30 13:08:29.223538 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.223534 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3b9423e-208d-45c0-bd24-bec761499e36" containerName="revoke-keys" Oct 30 13:08:29.227212 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.227194 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5101680f-4lrqb" Oct 30 13:08:29.238106 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.238086 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/055fc516-1042-4221-aff3-b07b975bec3d-kube-api-access-nlpgb\" (UniqueName: \"kubernetes.io/projected/055fc516-1042-4221-aff3-b07b975bec3d-kube-api-access-nlpgb\") pod \"055fc516-1042-4221-aff3-b07b975bec3d\" (UID: \"055fc516-1042-4221-aff3-b07b975bec3d\") " Oct 30 13:08:29.238212 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.238200 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/055fc516-1042-4221-aff3-b07b975bec3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/055fc516-1042-4221-aff3-b07b975bec3d-maas-api-openshift-ca\") pod \"055fc516-1042-4221-aff3-b07b975bec3d\" (UID: \"055fc516-1042-4221-aff3-b07b975bec3d\") " Oct 30 13:08:29.238282 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.238245 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/055fc516-1042-4221-aff3-b07b975bec3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/055fc516-1042-4221-aff3-b07b975bec3d-maas-api-xks-ca\") pod \"055fc516-1042-4221-aff3-b07b975bec3d\" (UID: \"055fc516-1042-4221-aff3-b07b975bec3d\") " Oct 30 13:08:29.238667 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.238598 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/055fc516-1042-4221-aff3-b07b975bec3d-maas-api-openshift-ca" pod "055fc516-1042-4221-aff3-b07b975bec3d" (UID: "055fc516-1042-4221-aff3-b07b975bec3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:29.240188 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.240168 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/055fc516-1042-4221-aff3-b07b975bec3d-kube-api-access-nlpgb" pod "055fc516-1042-4221-aff3-b07b975bec3d" (UID: "055fc516-1042-4221-aff3-b07b975bec3d"). InnerVolumeSpecName "kube-api-access-nlpgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:29.240274 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.240226 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/055fc516-1042-4221-aff3-b07b975bec3d-maas-api-xks-ca" pod "055fc516-1042-4221-aff3-b07b975bec3d" (UID: "055fc516-1042-4221-aff3-b07b975bec3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:29.248492 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.248469 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5101680f-4lrqb"] Oct 30 13:08:29.340089 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.340060 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/348fae6c-e0b1-47de-a4c5-c2db9e1cd49e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5101680f-4lrqb\" (UID: \"348fae6c-e0b1-47de-a4c5-c2db9e1cd49e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5101680f-4lrqb" Oct 30 13:08:29.340270 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.340170 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/348fae6c-e0b1-47de-a4c5-c2db9e1cd49e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5101680f-4lrqb\" (UID: \"348fae6c-e0b1-47de-a4c5-c2db9e1cd49e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5101680f-4lrqb" Oct 30 13:08:29.340339 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.340278 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgb85\" (UniqueName: \"kubernetes.io/projected/348fae6c-e0b1-47de-a4c5-c2db9e1cd49e-kube-api-access-xgb85\") pod \"maas-api-revoke-sub-5101680f-4lrqb\" (UID: \"348fae6c-e0b1-47de-a4c5-c2db9e1cd49e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5101680f-4lrqb" Oct 30 13:08:29.340437 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.340422 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/055fc516-1042-4221-aff3-b07b975bec3d-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:29.340491 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.340442 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/055fc516-1042-4221-aff3-b07b975bec3d-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:29.340491 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.340458 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlpgb\" (UniqueName: \"kubernetes.io/projected/055fc516-1042-4221-aff3-b07b975bec3d-kube-api-access-nlpgb\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:29.442054 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.442023 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/348fae6c-e0b1-47de-a4c5-c2db9e1cd49e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5101680f-4lrqb\" (UID: \"348fae6c-e0b1-47de-a4c5-c2db9e1cd49e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5101680f-4lrqb" Oct 30 13:08:29.442199 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.442082 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/348fae6c-e0b1-47de-a4c5-c2db9e1cd49e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5101680f-4lrqb\" (UID: \"348fae6c-e0b1-47de-a4c5-c2db9e1cd49e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5101680f-4lrqb" Oct 30 13:08:29.442199 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.442128 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgb85\" (UniqueName: \"kubernetes.io/projected/348fae6c-e0b1-47de-a4c5-c2db9e1cd49e-kube-api-access-xgb85\") pod \"maas-api-revoke-sub-5101680f-4lrqb\" (UID: \"348fae6c-e0b1-47de-a4c5-c2db9e1cd49e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5101680f-4lrqb" Oct 30 13:08:29.442751 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.442728 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/348fae6c-e0b1-47de-a4c5-c2db9e1cd49e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5101680f-4lrqb\" (UID: \"348fae6c-e0b1-47de-a4c5-c2db9e1cd49e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5101680f-4lrqb" Oct 30 13:08:29.444265 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.444246 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/348fae6c-e0b1-47de-a4c5-c2db9e1cd49e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5101680f-4lrqb\" (UID: \"348fae6c-e0b1-47de-a4c5-c2db9e1cd49e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5101680f-4lrqb" Oct 30 13:08:29.466383 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.466362 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgb85\" (UniqueName: \"kubernetes.io/projected/348fae6c-e0b1-47de-a4c5-c2db9e1cd49e-kube-api-access-xgb85\") pod \"maas-api-revoke-sub-5101680f-4lrqb\" (UID: \"348fae6c-e0b1-47de-a4c5-c2db9e1cd49e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5101680f-4lrqb" Oct 30 13:08:29.541944 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.541895 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5101680f-4lrqb" Oct 30 13:08:29.705899 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.705873 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d66ae31-rhpxq"] Oct 30 13:08:29.711125 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.711107 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d66ae31-rhpxq" Oct 30 13:08:29.735109 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.735084 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d66ae31-rhpxq"] Oct 30 13:08:29.846208 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.846176 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5101680f-4lrqb"] Oct 30 13:08:29.846981 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:08:29.846951 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod348fae6c_e0b1_47de_a4c5_c2db9e1cd49e.slice/crio-2a4e03da7265530def0115b2e2f7b2f3a8bd0244a264d3e1f1421c07ab396420 WatchSource:0}: Error finding container 2a4e03da7265530def0115b2e2f7b2f3a8bd0244a264d3e1f1421c07ab396420: Status 404 returned error can't find the container with id 2a4e03da7265530def0115b2e2f7b2f3a8bd0244a264d3e1f1421c07ab396420 Oct 30 13:08:29.849048 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.849009 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9872n\" (UniqueName: \"kubernetes.io/projected/d1e8049c-5832-4f22-b078-e29a51557bfb-kube-api-access-9872n\") pod \"maas-api-revoke-sub-8d66ae31-rhpxq\" (UID: \"d1e8049c-5832-4f22-b078-e29a51557bfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d66ae31-rhpxq" Oct 30 13:08:29.849204 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.849188 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1e8049c-5832-4f22-b078-e29a51557bfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d66ae31-rhpxq\" (UID: \"d1e8049c-5832-4f22-b078-e29a51557bfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d66ae31-rhpxq" Oct 30 13:08:29.849252 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.849243 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1e8049c-5832-4f22-b078-e29a51557bfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d66ae31-rhpxq\" (UID: \"d1e8049c-5832-4f22-b078-e29a51557bfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d66ae31-rhpxq" Oct 30 13:08:29.922412 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.922389 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8462ff11-htt87" Oct 30 13:08:29.922412 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.922388 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8462ff11-htt87" event={"ID":"055fc516-1042-4221-aff3-b07b975bec3d","Type":"ContainerDied","Data":"ce8189ebd9cfa0760468f3b1f1d413785d7e57b23d39aefc1cfd5a42aa98390a"} Oct 30 13:08:29.922815 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.922450 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce8189ebd9cfa0760468f3b1f1d413785d7e57b23d39aefc1cfd5a42aa98390a" Oct 30 13:08:29.923846 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.923824 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5101680f-4lrqb" event={"ID":"348fae6c-e0b1-47de-a4c5-c2db9e1cd49e","Type":"ContainerStarted","Data":"2a4e03da7265530def0115b2e2f7b2f3a8bd0244a264d3e1f1421c07ab396420"} Oct 30 13:08:29.925859 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.925838 5922 generic.go:356] "Generic (PLEG): container finished" podID="1eee749f-1f8c-4dc8-80b8-e0288d52e2b1" containerID="82e57b9fe02b47147dfbf945e45868f8faeeaf2150b54b69a09b616df7b24112" exitCode=0 Oct 30 13:08:29.925955 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.925916 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-925dbcef-gx9pd" event={"ID":"1eee749f-1f8c-4dc8-80b8-e0288d52e2b1","Type":"ContainerDied","Data":"82e57b9fe02b47147dfbf945e45868f8faeeaf2150b54b69a09b616df7b24112"} Oct 30 13:08:29.950518 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.950492 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9872n\" (UniqueName: \"kubernetes.io/projected/d1e8049c-5832-4f22-b078-e29a51557bfb-kube-api-access-9872n\") pod \"maas-api-revoke-sub-8d66ae31-rhpxq\" (UID: \"d1e8049c-5832-4f22-b078-e29a51557bfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d66ae31-rhpxq" Oct 30 13:08:29.950647 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.950604 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1e8049c-5832-4f22-b078-e29a51557bfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d66ae31-rhpxq\" (UID: \"d1e8049c-5832-4f22-b078-e29a51557bfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d66ae31-rhpxq" Oct 30 13:08:29.950696 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.950679 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1e8049c-5832-4f22-b078-e29a51557bfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d66ae31-rhpxq\" (UID: \"d1e8049c-5832-4f22-b078-e29a51557bfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d66ae31-rhpxq" Oct 30 13:08:29.951244 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.951222 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1e8049c-5832-4f22-b078-e29a51557bfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d66ae31-rhpxq\" (UID: \"d1e8049c-5832-4f22-b078-e29a51557bfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d66ae31-rhpxq" Oct 30 13:08:29.952857 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.952838 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1e8049c-5832-4f22-b078-e29a51557bfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d66ae31-rhpxq\" (UID: \"d1e8049c-5832-4f22-b078-e29a51557bfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d66ae31-rhpxq" Oct 30 13:08:29.974792 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:29.974766 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9872n\" (UniqueName: \"kubernetes.io/projected/d1e8049c-5832-4f22-b078-e29a51557bfb-kube-api-access-9872n\") pod \"maas-api-revoke-sub-8d66ae31-rhpxq\" (UID: \"d1e8049c-5832-4f22-b078-e29a51557bfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d66ae31-rhpxq" Oct 30 13:08:30.024450 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:30.024426 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d66ae31-rhpxq" Oct 30 13:08:30.229801 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:30.229772 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:30.345384 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:30.345290 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d66ae31-rhpxq"] Oct 30 13:08:30.346306 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:08:30.346279 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1e8049c_5832_4f22_b078_e29a51557bfb.slice/crio-b28a976e1e95e5791f117f3b627b926abdc284e7afabbe42c035868c86bc387e WatchSource:0}: Error finding container b28a976e1e95e5791f117f3b627b926abdc284e7afabbe42c035868c86bc387e: Status 404 returned error can't find the container with id b28a976e1e95e5791f117f3b627b926abdc284e7afabbe42c035868c86bc387e Oct 30 13:08:30.935506 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:30.935425 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d66ae31-rhpxq" event={"ID":"d1e8049c-5832-4f22-b078-e29a51557bfb","Type":"ContainerStarted","Data":"d297df8dcf181a776ef5f89ee39585a0dd2cb3c85129face9e4737fce5406c5c"} Oct 30 13:08:30.935506 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:30.935489 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d66ae31-rhpxq" event={"ID":"d1e8049c-5832-4f22-b078-e29a51557bfb","Type":"ContainerStarted","Data":"b28a976e1e95e5791f117f3b627b926abdc284e7afabbe42c035868c86bc387e"} Oct 30 13:08:30.937084 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:30.937060 5922 generic.go:356] "Generic (PLEG): container finished" podID="348fae6c-e0b1-47de-a4c5-c2db9e1cd49e" containerID="ac633b60600bdbe934660e775f143f26d146dd66870e397db9fbcef5861a1223" exitCode=0 Oct 30 13:08:30.937186 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:30.937113 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5101680f-4lrqb" event={"ID":"348fae6c-e0b1-47de-a4c5-c2db9e1cd49e","Type":"ContainerDied","Data":"ac633b60600bdbe934660e775f143f26d146dd66870e397db9fbcef5861a1223"} Oct 30 13:08:30.978360 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:30.978315 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d66ae31-rhpxq" podStartSLOduration=1.978300846 podStartE2EDuration="1.978300846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:30.975271671 +0000 UTC m=+967.444687442" watchObservedRunningTime="2026-10-30 13:08:30.978300846 +0000 UTC m=+967.447716590" Oct 30 13:08:31.104866 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:31.104837 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-925dbcef-gx9pd" Oct 30 13:08:31.165911 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:31.165881 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1eee749f-1f8c-4dc8-80b8-e0288d52e2b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eee749f-1f8c-4dc8-80b8-e0288d52e2b1-maas-api-xks-ca\") pod \"1eee749f-1f8c-4dc8-80b8-e0288d52e2b1\" (UID: \"1eee749f-1f8c-4dc8-80b8-e0288d52e2b1\") " Oct 30 13:08:31.166044 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:31.166021 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eee749f-1f8c-4dc8-80b8-e0288d52e2b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eee749f-1f8c-4dc8-80b8-e0288d52e2b1-maas-api-openshift-ca\") pod \"1eee749f-1f8c-4dc8-80b8-e0288d52e2b1\" (UID: \"1eee749f-1f8c-4dc8-80b8-e0288d52e2b1\") " Oct 30 13:08:31.166122 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:31.166100 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eee749f-1f8c-4dc8-80b8-e0288d52e2b1-kube-api-access-sxgf7\" (UniqueName: \"kubernetes.io/projected/1eee749f-1f8c-4dc8-80b8-e0288d52e2b1-kube-api-access-sxgf7\") pod \"1eee749f-1f8c-4dc8-80b8-e0288d52e2b1\" (UID: \"1eee749f-1f8c-4dc8-80b8-e0288d52e2b1\") " Oct 30 13:08:31.166777 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:31.166750 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eee749f-1f8c-4dc8-80b8-e0288d52e2b1-maas-api-openshift-ca" pod "1eee749f-1f8c-4dc8-80b8-e0288d52e2b1" (UID: "1eee749f-1f8c-4dc8-80b8-e0288d52e2b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:31.168407 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:31.168383 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eee749f-1f8c-4dc8-80b8-e0288d52e2b1-maas-api-xks-ca" pod "1eee749f-1f8c-4dc8-80b8-e0288d52e2b1" (UID: "1eee749f-1f8c-4dc8-80b8-e0288d52e2b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:31.168822 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:31.168794 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eee749f-1f8c-4dc8-80b8-e0288d52e2b1-kube-api-access-sxgf7" pod "1eee749f-1f8c-4dc8-80b8-e0288d52e2b1" (UID: "1eee749f-1f8c-4dc8-80b8-e0288d52e2b1"). InnerVolumeSpecName "kube-api-access-sxgf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:31.268170 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:31.268143 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxgf7\" (UniqueName: \"kubernetes.io/projected/1eee749f-1f8c-4dc8-80b8-e0288d52e2b1-kube-api-access-sxgf7\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:31.268275 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:31.268176 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eee749f-1f8c-4dc8-80b8-e0288d52e2b1-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:31.268275 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:31.268191 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eee749f-1f8c-4dc8-80b8-e0288d52e2b1-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:31.947359 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:31.947329 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-925dbcef-gx9pd" Oct 30 13:08:31.947850 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:31.947325 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-925dbcef-gx9pd" event={"ID":"1eee749f-1f8c-4dc8-80b8-e0288d52e2b1","Type":"ContainerDied","Data":"aa2dd0c924a037763a2c0ab640638ece066d64dd7797d055197157eb0813fc20"} Oct 30 13:08:31.947850 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:31.947464 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa2dd0c924a037763a2c0ab640638ece066d64dd7797d055197157eb0813fc20" Oct 30 13:08:31.949165 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:31.949097 5922 generic.go:356] "Generic (PLEG): container finished" podID="d1e8049c-5832-4f22-b078-e29a51557bfb" containerID="d297df8dcf181a776ef5f89ee39585a0dd2cb3c85129face9e4737fce5406c5c" exitCode=0 Oct 30 13:08:31.949307 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:31.949176 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d66ae31-rhpxq" event={"ID":"d1e8049c-5832-4f22-b078-e29a51557bfb","Type":"ContainerDied","Data":"d297df8dcf181a776ef5f89ee39585a0dd2cb3c85129face9e4737fce5406c5c"} Oct 30 13:08:32.010920 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:32.010885 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:32.149100 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:32.149069 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5101680f-4lrqb" Oct 30 13:08:32.175759 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:32.175730 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/348fae6c-e0b1-47de-a4c5-c2db9e1cd49e-kube-api-access-xgb85\" (UniqueName: \"kubernetes.io/projected/348fae6c-e0b1-47de-a4c5-c2db9e1cd49e-kube-api-access-xgb85\") pod \"348fae6c-e0b1-47de-a4c5-c2db9e1cd49e\" (UID: \"348fae6c-e0b1-47de-a4c5-c2db9e1cd49e\") " Oct 30 13:08:32.175937 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:32.175875 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/348fae6c-e0b1-47de-a4c5-c2db9e1cd49e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/348fae6c-e0b1-47de-a4c5-c2db9e1cd49e-maas-api-xks-ca\") pod \"348fae6c-e0b1-47de-a4c5-c2db9e1cd49e\" (UID: \"348fae6c-e0b1-47de-a4c5-c2db9e1cd49e\") " Oct 30 13:08:32.176006 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:32.175984 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/348fae6c-e0b1-47de-a4c5-c2db9e1cd49e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/348fae6c-e0b1-47de-a4c5-c2db9e1cd49e-maas-api-openshift-ca\") pod \"348fae6c-e0b1-47de-a4c5-c2db9e1cd49e\" (UID: \"348fae6c-e0b1-47de-a4c5-c2db9e1cd49e\") " Oct 30 13:08:32.176332 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:32.176302 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/348fae6c-e0b1-47de-a4c5-c2db9e1cd49e-maas-api-openshift-ca" pod "348fae6c-e0b1-47de-a4c5-c2db9e1cd49e" (UID: "348fae6c-e0b1-47de-a4c5-c2db9e1cd49e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:32.176439 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:32.176416 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/348fae6c-e0b1-47de-a4c5-c2db9e1cd49e-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:32.177842 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:32.177821 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/348fae6c-e0b1-47de-a4c5-c2db9e1cd49e-maas-api-xks-ca" pod "348fae6c-e0b1-47de-a4c5-c2db9e1cd49e" (UID: "348fae6c-e0b1-47de-a4c5-c2db9e1cd49e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:32.177912 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:32.177890 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/348fae6c-e0b1-47de-a4c5-c2db9e1cd49e-kube-api-access-xgb85" pod "348fae6c-e0b1-47de-a4c5-c2db9e1cd49e" (UID: "348fae6c-e0b1-47de-a4c5-c2db9e1cd49e"). InnerVolumeSpecName "kube-api-access-xgb85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:32.277539 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:32.277459 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgb85\" (UniqueName: \"kubernetes.io/projected/348fae6c-e0b1-47de-a4c5-c2db9e1cd49e-kube-api-access-xgb85\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:32.277539 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:32.277489 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/348fae6c-e0b1-47de-a4c5-c2db9e1cd49e-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:32.515377 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:32.515342 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08bacf2d-st7nj"] Oct 30 13:08:32.516048 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:32.516026 5922 cpu_manager.go:414] "containerMap: removing container" podUID="348fae6c-e0b1-47de-a4c5-c2db9e1cd49e" containerName="revoke-keys" Oct 30 13:08:32.516048 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:32.516048 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="348fae6c-e0b1-47de-a4c5-c2db9e1cd49e" containerName="revoke-keys" Oct 30 13:08:32.516308 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:32.516111 5922 cpu_manager.go:414] "containerMap: removing container" podUID="1eee749f-1f8c-4dc8-80b8-e0288d52e2b1" containerName="revoke-keys" Oct 30 13:08:32.516308 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:32.516120 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eee749f-1f8c-4dc8-80b8-e0288d52e2b1" containerName="revoke-keys" Oct 30 13:08:32.516308 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:32.516274 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eee749f-1f8c-4dc8-80b8-e0288d52e2b1" containerName="revoke-keys" Oct 30 13:08:32.516308 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:32.516295 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="348fae6c-e0b1-47de-a4c5-c2db9e1cd49e" containerName="revoke-keys" Oct 30 13:08:32.520197 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:32.520178 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08bacf2d-st7nj" Oct 30 13:08:32.544110 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:32.544044 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08bacf2d-st7nj"] Oct 30 13:08:32.581442 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:32.581409 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/207be5eb-d83a-477b-af9a-550a8abe5e73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08bacf2d-st7nj\" (UID: \"207be5eb-d83a-477b-af9a-550a8abe5e73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08bacf2d-st7nj" Oct 30 13:08:32.581660 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:32.581639 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/207be5eb-d83a-477b-af9a-550a8abe5e73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08bacf2d-st7nj\" (UID: \"207be5eb-d83a-477b-af9a-550a8abe5e73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08bacf2d-st7nj" Oct 30 13:08:32.581867 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:32.581842 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcz92\" (UniqueName: \"kubernetes.io/projected/207be5eb-d83a-477b-af9a-550a8abe5e73-kube-api-access-gcz92\") pod \"maas-api-revoke-sub-08bacf2d-st7nj\" (UID: \"207be5eb-d83a-477b-af9a-550a8abe5e73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08bacf2d-st7nj" Oct 30 13:08:32.682480 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:32.682452 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/207be5eb-d83a-477b-af9a-550a8abe5e73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08bacf2d-st7nj\" (UID: \"207be5eb-d83a-477b-af9a-550a8abe5e73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08bacf2d-st7nj" Oct 30 13:08:32.682574 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:32.682559 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcz92\" (UniqueName: \"kubernetes.io/projected/207be5eb-d83a-477b-af9a-550a8abe5e73-kube-api-access-gcz92\") pod \"maas-api-revoke-sub-08bacf2d-st7nj\" (UID: \"207be5eb-d83a-477b-af9a-550a8abe5e73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08bacf2d-st7nj" Oct 30 13:08:32.682650 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:32.682629 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/207be5eb-d83a-477b-af9a-550a8abe5e73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08bacf2d-st7nj\" (UID: \"207be5eb-d83a-477b-af9a-550a8abe5e73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08bacf2d-st7nj" Oct 30 13:08:32.683097 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:32.683077 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/207be5eb-d83a-477b-af9a-550a8abe5e73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08bacf2d-st7nj\" (UID: \"207be5eb-d83a-477b-af9a-550a8abe5e73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08bacf2d-st7nj" Oct 30 13:08:32.685067 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:32.685047 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/207be5eb-d83a-477b-af9a-550a8abe5e73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08bacf2d-st7nj\" (UID: \"207be5eb-d83a-477b-af9a-550a8abe5e73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08bacf2d-st7nj" Oct 30 13:08:32.701953 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:32.701931 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:32.706860 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:32.706843 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcz92\" (UniqueName: \"kubernetes.io/projected/207be5eb-d83a-477b-af9a-550a8abe5e73-kube-api-access-gcz92\") pod \"maas-api-revoke-sub-08bacf2d-st7nj\" (UID: \"207be5eb-d83a-477b-af9a-550a8abe5e73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08bacf2d-st7nj" Oct 30 13:08:32.834330 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:32.834253 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08bacf2d-st7nj" Oct 30 13:08:32.960441 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:32.960417 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5101680f-4lrqb" Oct 30 13:08:32.960441 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:32.960408 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5101680f-4lrqb" event={"ID":"348fae6c-e0b1-47de-a4c5-c2db9e1cd49e","Type":"ContainerDied","Data":"2a4e03da7265530def0115b2e2f7b2f3a8bd0244a264d3e1f1421c07ab396420"} Oct 30 13:08:32.961011 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:32.960470 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a4e03da7265530def0115b2e2f7b2f3a8bd0244a264d3e1f1421c07ab396420" Oct 30 13:08:33.087865 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:33.087803 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d66ae31-rhpxq" Oct 30 13:08:33.140987 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:08:33.140961 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207be5eb_d83a_477b_af9a_550a8abe5e73.slice/crio-1b47d88a86a78a67cd0d8bce726a768838de4bab20e046d74ca89de3c2da4a7a WatchSource:0}: Error finding container 1b47d88a86a78a67cd0d8bce726a768838de4bab20e046d74ca89de3c2da4a7a: Status 404 returned error can't find the container with id 1b47d88a86a78a67cd0d8bce726a768838de4bab20e046d74ca89de3c2da4a7a Oct 30 13:08:33.148628 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:33.148589 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08bacf2d-st7nj"] Oct 30 13:08:33.188591 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:33.188567 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1e8049c-5832-4f22-b078-e29a51557bfb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1e8049c-5832-4f22-b078-e29a51557bfb-maas-api-openshift-ca\") pod \"d1e8049c-5832-4f22-b078-e29a51557bfb\" (UID: \"d1e8049c-5832-4f22-b078-e29a51557bfb\") " Oct 30 13:08:33.188787 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:33.188769 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1e8049c-5832-4f22-b078-e29a51557bfb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1e8049c-5832-4f22-b078-e29a51557bfb-maas-api-xks-ca\") pod \"d1e8049c-5832-4f22-b078-e29a51557bfb\" (UID: \"d1e8049c-5832-4f22-b078-e29a51557bfb\") " Oct 30 13:08:33.188856 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:33.188807 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1e8049c-5832-4f22-b078-e29a51557bfb-kube-api-access-9872n\" (UniqueName: \"kubernetes.io/projected/d1e8049c-5832-4f22-b078-e29a51557bfb-kube-api-access-9872n\") pod \"d1e8049c-5832-4f22-b078-e29a51557bfb\" (UID: \"d1e8049c-5832-4f22-b078-e29a51557bfb\") " Oct 30 13:08:33.188910 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:33.188886 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1e8049c-5832-4f22-b078-e29a51557bfb-maas-api-openshift-ca" pod "d1e8049c-5832-4f22-b078-e29a51557bfb" (UID: "d1e8049c-5832-4f22-b078-e29a51557bfb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:33.189254 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:33.189236 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1e8049c-5832-4f22-b078-e29a51557bfb-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:33.190637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:33.190604 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1e8049c-5832-4f22-b078-e29a51557bfb-maas-api-xks-ca" pod "d1e8049c-5832-4f22-b078-e29a51557bfb" (UID: "d1e8049c-5832-4f22-b078-e29a51557bfb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:33.190907 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:33.190889 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1e8049c-5832-4f22-b078-e29a51557bfb-kube-api-access-9872n" pod "d1e8049c-5832-4f22-b078-e29a51557bfb" (UID: "d1e8049c-5832-4f22-b078-e29a51557bfb"). InnerVolumeSpecName "kube-api-access-9872n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:33.290680 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:33.290656 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1e8049c-5832-4f22-b078-e29a51557bfb-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:33.290680 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:33.290679 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9872n\" (UniqueName: \"kubernetes.io/projected/d1e8049c-5832-4f22-b078-e29a51557bfb-kube-api-access-9872n\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:33.770261 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:33.770228 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:33.969933 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:33.969897 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08bacf2d-st7nj" event={"ID":"207be5eb-d83a-477b-af9a-550a8abe5e73","Type":"ContainerStarted","Data":"65af167f59e75105a08570ed9d8cb2925c74a29ec11a666fe77f819610f8b6c9"} Oct 30 13:08:33.970338 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:33.969965 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08bacf2d-st7nj" event={"ID":"207be5eb-d83a-477b-af9a-550a8abe5e73","Type":"ContainerStarted","Data":"1b47d88a86a78a67cd0d8bce726a768838de4bab20e046d74ca89de3c2da4a7a"} Oct 30 13:08:33.971532 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:33.971512 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d66ae31-rhpxq" Oct 30 13:08:33.971642 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:33.971552 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d66ae31-rhpxq" event={"ID":"d1e8049c-5832-4f22-b078-e29a51557bfb","Type":"ContainerDied","Data":"b28a976e1e95e5791f117f3b627b926abdc284e7afabbe42c035868c86bc387e"} Oct 30 13:08:33.971642 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:33.971593 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b28a976e1e95e5791f117f3b627b926abdc284e7afabbe42c035868c86bc387e" Oct 30 13:08:34.014562 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:34.014518 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08bacf2d-st7nj" podStartSLOduration=2.014506156 podStartE2EDuration="2.014506156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:34.012831667 +0000 UTC m=+970.482247412" watchObservedRunningTime="2026-10-30 13:08:34.014506156 +0000 UTC m=+970.483921900" Oct 30 13:08:34.980739 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:34.980705 5922 generic.go:356] "Generic (PLEG): container finished" podID="207be5eb-d83a-477b-af9a-550a8abe5e73" containerID="65af167f59e75105a08570ed9d8cb2925c74a29ec11a666fe77f819610f8b6c9" exitCode=0 Oct 30 13:08:34.981182 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:34.980787 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08bacf2d-st7nj" event={"ID":"207be5eb-d83a-477b-af9a-550a8abe5e73","Type":"ContainerDied","Data":"65af167f59e75105a08570ed9d8cb2925c74a29ec11a666fe77f819610f8b6c9"} Oct 30 13:08:35.644882 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:35.644846 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:36.134404 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:36.134380 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08bacf2d-st7nj" Oct 30 13:08:36.226349 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:36.226320 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/207be5eb-d83a-477b-af9a-550a8abe5e73-kube-api-access-gcz92\" (UniqueName: \"kubernetes.io/projected/207be5eb-d83a-477b-af9a-550a8abe5e73-kube-api-access-gcz92\") pod \"207be5eb-d83a-477b-af9a-550a8abe5e73\" (UID: \"207be5eb-d83a-477b-af9a-550a8abe5e73\") " Oct 30 13:08:36.226516 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:36.226358 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/207be5eb-d83a-477b-af9a-550a8abe5e73-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/207be5eb-d83a-477b-af9a-550a8abe5e73-maas-api-xks-ca\") pod \"207be5eb-d83a-477b-af9a-550a8abe5e73\" (UID: \"207be5eb-d83a-477b-af9a-550a8abe5e73\") " Oct 30 13:08:36.226516 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:36.226387 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/207be5eb-d83a-477b-af9a-550a8abe5e73-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/207be5eb-d83a-477b-af9a-550a8abe5e73-maas-api-openshift-ca\") pod \"207be5eb-d83a-477b-af9a-550a8abe5e73\" (UID: \"207be5eb-d83a-477b-af9a-550a8abe5e73\") " Oct 30 13:08:36.226825 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:36.226792 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/207be5eb-d83a-477b-af9a-550a8abe5e73-maas-api-openshift-ca" pod "207be5eb-d83a-477b-af9a-550a8abe5e73" (UID: "207be5eb-d83a-477b-af9a-550a8abe5e73"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:36.228386 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:36.228367 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/207be5eb-d83a-477b-af9a-550a8abe5e73-maas-api-xks-ca" pod "207be5eb-d83a-477b-af9a-550a8abe5e73" (UID: "207be5eb-d83a-477b-af9a-550a8abe5e73"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:36.228461 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:36.228441 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/207be5eb-d83a-477b-af9a-550a8abe5e73-kube-api-access-gcz92" pod "207be5eb-d83a-477b-af9a-550a8abe5e73" (UID: "207be5eb-d83a-477b-af9a-550a8abe5e73"). InnerVolumeSpecName "kube-api-access-gcz92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:36.328341 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:36.328272 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcz92\" (UniqueName: \"kubernetes.io/projected/207be5eb-d83a-477b-af9a-550a8abe5e73-kube-api-access-gcz92\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:36.328341 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:36.328299 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/207be5eb-d83a-477b-af9a-550a8abe5e73-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:36.328341 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:36.328315 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/207be5eb-d83a-477b-af9a-550a8abe5e73-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:36.997059 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:36.997026 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08bacf2d-st7nj" Oct 30 13:08:36.997255 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:36.997065 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08bacf2d-st7nj" event={"ID":"207be5eb-d83a-477b-af9a-550a8abe5e73","Type":"ContainerDied","Data":"1b47d88a86a78a67cd0d8bce726a768838de4bab20e046d74ca89de3c2da4a7a"} Oct 30 13:08:36.997255 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:36.997112 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b47d88a86a78a67cd0d8bce726a768838de4bab20e046d74ca89de3c2da4a7a" Oct 30 13:08:37.535910 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:37.535881 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:38.039718 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:38.039676 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66fb3c85-7v8nn"] Oct 30 13:08:38.040268 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:38.040254 5922 cpu_manager.go:414] "containerMap: removing container" podUID="207be5eb-d83a-477b-af9a-550a8abe5e73" containerName="revoke-keys" Oct 30 13:08:38.040316 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:38.040269 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="207be5eb-d83a-477b-af9a-550a8abe5e73" containerName="revoke-keys" Oct 30 13:08:38.040316 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:38.040304 5922 cpu_manager.go:414] "containerMap: removing container" podUID="d1e8049c-5832-4f22-b078-e29a51557bfb" containerName="revoke-keys" Oct 30 13:08:38.040316 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:38.040309 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1e8049c-5832-4f22-b078-e29a51557bfb" containerName="revoke-keys" Oct 30 13:08:38.040415 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:38.040406 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1e8049c-5832-4f22-b078-e29a51557bfb" containerName="revoke-keys" Oct 30 13:08:38.040449 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:38.040421 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="207be5eb-d83a-477b-af9a-550a8abe5e73" containerName="revoke-keys" Oct 30 13:08:38.044273 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:38.044257 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66fb3c85-7v8nn" Oct 30 13:08:38.064106 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:38.064077 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66fb3c85-7v8nn"] Oct 30 13:08:38.069493 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:38.069459 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:08:38.069607 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:38.069492 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:08:38.069607 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:38.069479 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:08:38.071646 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:38.071592 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:08:38.151796 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:38.151757 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e420d739-6af6-408d-bb35-b5af95de2c5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66fb3c85-7v8nn\" (UID: \"e420d739-6af6-408d-bb35-b5af95de2c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66fb3c85-7v8nn" Oct 30 13:08:38.151796 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:38.151800 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h44s\" (UniqueName: \"kubernetes.io/projected/e420d739-6af6-408d-bb35-b5af95de2c5f-kube-api-access-4h44s\") pod \"maas-api-revoke-sub-66fb3c85-7v8nn\" (UID: \"e420d739-6af6-408d-bb35-b5af95de2c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66fb3c85-7v8nn" Oct 30 13:08:38.151962 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:38.151898 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e420d739-6af6-408d-bb35-b5af95de2c5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66fb3c85-7v8nn\" (UID: \"e420d739-6af6-408d-bb35-b5af95de2c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66fb3c85-7v8nn" Oct 30 13:08:38.252595 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:38.252557 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e420d739-6af6-408d-bb35-b5af95de2c5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66fb3c85-7v8nn\" (UID: \"e420d739-6af6-408d-bb35-b5af95de2c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66fb3c85-7v8nn" Oct 30 13:08:38.252595 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:38.252590 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h44s\" (UniqueName: \"kubernetes.io/projected/e420d739-6af6-408d-bb35-b5af95de2c5f-kube-api-access-4h44s\") pod \"maas-api-revoke-sub-66fb3c85-7v8nn\" (UID: \"e420d739-6af6-408d-bb35-b5af95de2c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66fb3c85-7v8nn" Oct 30 13:08:38.252855 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:38.252668 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e420d739-6af6-408d-bb35-b5af95de2c5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66fb3c85-7v8nn\" (UID: \"e420d739-6af6-408d-bb35-b5af95de2c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66fb3c85-7v8nn" Oct 30 13:08:38.253410 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:38.253387 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e420d739-6af6-408d-bb35-b5af95de2c5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66fb3c85-7v8nn\" (UID: \"e420d739-6af6-408d-bb35-b5af95de2c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66fb3c85-7v8nn" Oct 30 13:08:38.254930 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:38.254904 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e420d739-6af6-408d-bb35-b5af95de2c5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66fb3c85-7v8nn\" (UID: \"e420d739-6af6-408d-bb35-b5af95de2c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66fb3c85-7v8nn" Oct 30 13:08:38.277672 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:38.277647 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h44s\" (UniqueName: \"kubernetes.io/projected/e420d739-6af6-408d-bb35-b5af95de2c5f-kube-api-access-4h44s\") pod \"maas-api-revoke-sub-66fb3c85-7v8nn\" (UID: \"e420d739-6af6-408d-bb35-b5af95de2c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66fb3c85-7v8nn" Oct 30 13:08:38.358586 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:38.358521 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66fb3c85-7v8nn" Oct 30 13:08:38.670306 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:38.670271 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66fb3c85-7v8nn"] Oct 30 13:08:38.949607 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:38.949579 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be022317-9k5dl"] Oct 30 13:08:38.956973 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:38.956954 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be022317-9k5dl" Oct 30 13:08:38.975377 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:38.975348 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be022317-9k5dl"] Oct 30 13:08:39.016644 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:39.016583 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66fb3c85-7v8nn" event={"ID":"e420d739-6af6-408d-bb35-b5af95de2c5f","Type":"ContainerStarted","Data":"f16b093f50b8156f881db45962bb451bf6f51865439580e0c15efb06c760ccf6"} Oct 30 13:08:39.016757 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:39.016675 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66fb3c85-7v8nn" event={"ID":"e420d739-6af6-408d-bb35-b5af95de2c5f","Type":"ContainerStarted","Data":"de184c70e1a50a68c1330c7ec1f378ae07ab7b3f403bddffe6cfa8dbc279a425"} Oct 30 13:08:39.060762 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:39.060722 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66fb3c85-7v8nn" podStartSLOduration=1.060710088 podStartE2EDuration="1.060710088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:39.05713464 +0000 UTC m=+975.526550385" watchObservedRunningTime="2026-10-30 13:08:39.060710088 +0000 UTC m=+975.530125833" Oct 30 13:08:39.062457 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:39.062435 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ddaea9f-d694-4294-bbc3-3d718ae8abd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be022317-9k5dl\" (UID: \"2ddaea9f-d694-4294-bbc3-3d718ae8abd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be022317-9k5dl" Oct 30 13:08:39.062670 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:39.062642 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2stgk\" (UniqueName: \"kubernetes.io/projected/2ddaea9f-d694-4294-bbc3-3d718ae8abd1-kube-api-access-2stgk\") pod \"maas-api-revoke-sub-be022317-9k5dl\" (UID: \"2ddaea9f-d694-4294-bbc3-3d718ae8abd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be022317-9k5dl" Oct 30 13:08:39.062762 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:39.062745 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ddaea9f-d694-4294-bbc3-3d718ae8abd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be022317-9k5dl\" (UID: \"2ddaea9f-d694-4294-bbc3-3d718ae8abd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be022317-9k5dl" Oct 30 13:08:39.163689 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:39.163661 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ddaea9f-d694-4294-bbc3-3d718ae8abd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be022317-9k5dl\" (UID: \"2ddaea9f-d694-4294-bbc3-3d718ae8abd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be022317-9k5dl" Oct 30 13:08:39.163803 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:39.163789 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ddaea9f-d694-4294-bbc3-3d718ae8abd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be022317-9k5dl\" (UID: \"2ddaea9f-d694-4294-bbc3-3d718ae8abd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be022317-9k5dl" Oct 30 13:08:39.163892 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:39.163862 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2stgk\" (UniqueName: \"kubernetes.io/projected/2ddaea9f-d694-4294-bbc3-3d718ae8abd1-kube-api-access-2stgk\") pod \"maas-api-revoke-sub-be022317-9k5dl\" (UID: \"2ddaea9f-d694-4294-bbc3-3d718ae8abd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be022317-9k5dl" Oct 30 13:08:39.164406 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:39.164376 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ddaea9f-d694-4294-bbc3-3d718ae8abd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be022317-9k5dl\" (UID: \"2ddaea9f-d694-4294-bbc3-3d718ae8abd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be022317-9k5dl" Oct 30 13:08:39.166078 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:39.166057 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ddaea9f-d694-4294-bbc3-3d718ae8abd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be022317-9k5dl\" (UID: \"2ddaea9f-d694-4294-bbc3-3d718ae8abd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be022317-9k5dl" Oct 30 13:08:39.190055 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:39.190034 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2stgk\" (UniqueName: \"kubernetes.io/projected/2ddaea9f-d694-4294-bbc3-3d718ae8abd1-kube-api-access-2stgk\") pod \"maas-api-revoke-sub-be022317-9k5dl\" (UID: \"2ddaea9f-d694-4294-bbc3-3d718ae8abd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be022317-9k5dl" Oct 30 13:08:39.272591 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:39.272534 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be022317-9k5dl" Oct 30 13:08:39.408606 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:39.408570 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:39.582780 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:39.582755 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be022317-9k5dl"] Oct 30 13:08:39.655575 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:08:39.655542 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ddaea9f_d694_4294_bbc3_3d718ae8abd1.slice/crio-96221aba334b02e8da11c33d2aef87bb4f4affba3c9b3367102844134592eb08 WatchSource:0}: Error finding container 96221aba334b02e8da11c33d2aef87bb4f4affba3c9b3367102844134592eb08: Status 404 returned error can't find the container with id 96221aba334b02e8da11c33d2aef87bb4f4affba3c9b3367102844134592eb08 Oct 30 13:08:40.028170 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:40.028127 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be022317-9k5dl" event={"ID":"2ddaea9f-d694-4294-bbc3-3d718ae8abd1","Type":"ContainerStarted","Data":"25aea5083d07e431a1d25029628b8f0bef171d531588bfe128d98ce740c95d3d"} Oct 30 13:08:40.028659 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:40.028204 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be022317-9k5dl" event={"ID":"2ddaea9f-d694-4294-bbc3-3d718ae8abd1","Type":"ContainerStarted","Data":"96221aba334b02e8da11c33d2aef87bb4f4affba3c9b3367102844134592eb08"} Oct 30 13:08:40.031574 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:40.031547 5922 generic.go:356] "Generic (PLEG): container finished" podID="e420d739-6af6-408d-bb35-b5af95de2c5f" containerID="f16b093f50b8156f881db45962bb451bf6f51865439580e0c15efb06c760ccf6" exitCode=0 Oct 30 13:08:40.031774 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:40.031743 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66fb3c85-7v8nn" event={"ID":"e420d739-6af6-408d-bb35-b5af95de2c5f","Type":"ContainerDied","Data":"f16b093f50b8156f881db45962bb451bf6f51865439580e0c15efb06c760ccf6"} Oct 30 13:08:40.111676 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:40.102209 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be022317-9k5dl" podStartSLOduration=2.102183996 podStartE2EDuration="2.102183996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:40.081544429 +0000 UTC m=+976.550960175" watchObservedRunningTime="2026-10-30 13:08:40.102183996 +0000 UTC m=+976.571599753" Oct 30 13:08:41.101114 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:41.099348 5922 generic.go:356] "Generic (PLEG): container finished" podID="2ddaea9f-d694-4294-bbc3-3d718ae8abd1" containerID="25aea5083d07e431a1d25029628b8f0bef171d531588bfe128d98ce740c95d3d" exitCode=0 Oct 30 13:08:41.101114 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:41.100134 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be022317-9k5dl" event={"ID":"2ddaea9f-d694-4294-bbc3-3d718ae8abd1","Type":"ContainerDied","Data":"25aea5083d07e431a1d25029628b8f0bef171d531588bfe128d98ce740c95d3d"} Oct 30 13:08:41.669637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:41.668743 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" podUID="47517373-759d-4eeb-ab5e-6150b7e82bb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:41.743741 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:41.743696 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" podUID="dad5bdad-cd6f-41f1-a124-540c3524ead7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:41.798742 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:41.798695 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" podUID="a85de36b-f8c8-4ade-a468-c114080cb312" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:41.800170 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:41.800140 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" podUID="14ae0aea-473f-4ca0-bcc0-f617b9839c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:41.800707 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:41.800681 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" podUID="7751779d-b082-409a-a20f-ff0ff6263808" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:41.826635 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:41.825801 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" podUID="dfd83cc8-6503-45a8-8260-c0edb8752471" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 13:08:41.851633 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:41.849690 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" podUID="c0ea7fde-2ac5-432d-bb9b-534024186ac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:41.851633 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:41.850535 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" podUID="5f353928-8cb4-49a3-b999-1b90e6cb31bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:41.868836 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:41.864829 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" podUID="35619ab9-cbeb-45da-8506-3af63a273177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:42.551144 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:42.534097 5922 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" podUID="b487a069-f519-45a3-b1d8-f32f2431797b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:42.689661 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:42.688749 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" podUID="b487a069-f519-45a3-b1d8-f32f2431797b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:42.714636 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:42.691407 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" podUID="47517373-759d-4eeb-ab5e-6150b7e82bb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:42.715148 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:42.715113 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" Oct 30 13:08:42.814058 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:42.812762 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" podUID="7751779d-b082-409a-a20f-ff0ff6263808" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:42.899638 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:42.871865 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" podUID="dad5bdad-cd6f-41f1-a124-540c3524ead7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 13:08:42.974194 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:42.974121 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" podUID="14ae0aea-473f-4ca0-bcc0-f617b9839c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded" Oct 30 13:08:42.989637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:42.981734 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" podUID="a85de36b-f8c8-4ade-a468-c114080cb312" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:42.989637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:42.982061 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" podUID="5f353928-8cb4-49a3-b999-1b90e6cb31bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 13:08:43.015860 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:42.996245 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" podUID="35619ab9-cbeb-45da-8506-3af63a273177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:43.015860 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:42.996501 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" Oct 30 13:08:43.015860 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:42.996628 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" Oct 30 13:08:43.015860 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:42.996787 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" Oct 30 13:08:43.015860 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:42.996956 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" Oct 30 13:08:43.015860 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:42.999348 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" podUID="c0ea7fde-2ac5-432d-bb9b-534024186ac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 13:08:43.079094 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:43.059241 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" podUID="dfd83cc8-6503-45a8-8260-c0edb8752471" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 13:08:43.079094 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:43.066569 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" Oct 30 13:08:43.079094 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:43.074560 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" Oct 30 13:08:43.079094 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:43.078250 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" Oct 30 13:08:43.079094 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:43.078432 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" Oct 30 13:08:43.197192 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:43.197144 5922 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" podUID="a85de36b-f8c8-4ade-a468-c114080cb312" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:43.393051 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:43.390853 5922 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" podUID="35619ab9-cbeb-45da-8506-3af63a273177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:43.396186 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:43.396160 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66fb3c85-7v8nn" Oct 30 13:08:43.595471 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:43.595431 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e420d739-6af6-408d-bb35-b5af95de2c5f-kube-api-access-4h44s\" (UniqueName: \"kubernetes.io/projected/e420d739-6af6-408d-bb35-b5af95de2c5f-kube-api-access-4h44s\") pod \"e420d739-6af6-408d-bb35-b5af95de2c5f\" (UID: \"e420d739-6af6-408d-bb35-b5af95de2c5f\") " Oct 30 13:08:43.700695 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:43.697692 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" podUID="b487a069-f519-45a3-b1d8-f32f2431797b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:43.740637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:43.703372 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" podUID="47517373-759d-4eeb-ab5e-6150b7e82bb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 13:08:43.740637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:43.739686 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e420d739-6af6-408d-bb35-b5af95de2c5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e420d739-6af6-408d-bb35-b5af95de2c5f-maas-api-xks-ca\") pod \"e420d739-6af6-408d-bb35-b5af95de2c5f\" (UID: \"e420d739-6af6-408d-bb35-b5af95de2c5f\") " Oct 30 13:08:43.751813 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:43.741082 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e420d739-6af6-408d-bb35-b5af95de2c5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e420d739-6af6-408d-bb35-b5af95de2c5f-maas-api-openshift-ca\") pod \"e420d739-6af6-408d-bb35-b5af95de2c5f\" (UID: \"e420d739-6af6-408d-bb35-b5af95de2c5f\") " Oct 30 13:08:43.789967 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:43.789932 5922 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:08:43.790196 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:43.790168 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" podUID="d0132701-3b77-4cce-9989-a0d03ac479f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:43.813699 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:43.796931 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e420d739-6af6-408d-bb35-b5af95de2c5f-maas-api-openshift-ca" pod "e420d739-6af6-408d-bb35-b5af95de2c5f" (UID: "e420d739-6af6-408d-bb35-b5af95de2c5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:43.813699 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:43.798478 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e420d739-6af6-408d-bb35-b5af95de2c5f-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:43.813699 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:43.813075 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" podUID="7751779d-b082-409a-a20f-ff0ff6263808" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 13:08:43.850638 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:43.840875 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e420d739-6af6-408d-bb35-b5af95de2c5f-maas-api-xks-ca" pod "e420d739-6af6-408d-bb35-b5af95de2c5f" (UID: "e420d739-6af6-408d-bb35-b5af95de2c5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:43.930627 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:43.930566 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e420d739-6af6-408d-bb35-b5af95de2c5f-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:44.190856 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:43.993929 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" podUID="5f353928-8cb4-49a3-b999-1b90e6cb31bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 13:08:44.191382 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:43.993958 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" podUID="dad5bdad-cd6f-41f1-a124-540c3524ead7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 13:08:44.224067 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:43.993982 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" podUID="a85de36b-f8c8-4ade-a468-c114080cb312" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 13:08:44.224592 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:44.067498 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" podUID="c0ea7fde-2ac5-432d-bb9b-534024186ac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:44.224993 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:44.067532 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" podUID="35619ab9-cbeb-45da-8506-3af63a273177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 13:08:44.225352 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:44.105160 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" podUID="dfd83cc8-6503-45a8-8260-c0edb8752471" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:44.244034 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:44.105194 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" podUID="14ae0aea-473f-4ca0-bcc0-f617b9839c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded" Oct 30 13:08:44.244683 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:44.145485 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e420d739-6af6-408d-bb35-b5af95de2c5f-kube-api-access-4h44s" pod "e420d739-6af6-408d-bb35-b5af95de2c5f" (UID: "e420d739-6af6-408d-bb35-b5af95de2c5f"). InnerVolumeSpecName "kube-api-access-4h44s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:44.261035 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:44.261008 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" Oct 30 13:08:44.429719 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:44.262156 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:44.430008 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:44.271118 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66fb3c85-7v8nn" Oct 30 13:08:44.430747 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:44.405226 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h44s\" (UniqueName: \"kubernetes.io/projected/e420d739-6af6-408d-bb35-b5af95de2c5f-kube-api-access-4h44s\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:44.430939 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:44.430915 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66fb3c85-7v8nn" event={"ID":"e420d739-6af6-408d-bb35-b5af95de2c5f","Type":"ContainerDied","Data":"de184c70e1a50a68c1330c7ec1f378ae07ab7b3f403bddffe6cfa8dbc279a425"} Oct 30 13:08:44.431051 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:44.431039 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de184c70e1a50a68c1330c7ec1f378ae07ab7b3f403bddffe6cfa8dbc279a425" Oct 30 13:08:44.449825 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:44.449789 5922 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:08:44.450077 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:44.450037 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" podUID="aee561c5-3aa7-4021-9d3e-05e7636d19be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:44.474636 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:44.454925 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" Oct 30 13:08:44.708302 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:44.708257 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" podUID="47517373-759d-4eeb-ab5e-6150b7e82bb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 13:08:44.761229 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:44.761077 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" podUID="b487a069-f519-45a3-b1d8-f32f2431797b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:44.817668 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:44.813917 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" podUID="7751779d-b082-409a-a20f-ff0ff6263808" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:44.866519 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:44.866481 5922 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:08:45.030699 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:44.983963 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" podUID="1830fbd9-7c51-40d0-b6df-49a723185784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:45.198716 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.197005 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" podUID="5f353928-8cb4-49a3-b999-1b90e6cb31bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:45.327739 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.322966 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" Oct 30 13:08:45.331991 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.331656 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" podUID="c0ea7fde-2ac5-432d-bb9b-534024186ac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 13:08:45.403648 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.402408 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" podUID="35619ab9-cbeb-45da-8506-3af63a273177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:45.403648 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.402736 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm"] Oct 30 13:08:45.403986 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.403969 5922 cpu_manager.go:414] "containerMap: removing container" podUID="e420d739-6af6-408d-bb35-b5af95de2c5f" containerName="revoke-keys" Oct 30 13:08:45.404117 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.404104 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e420d739-6af6-408d-bb35-b5af95de2c5f" containerName="revoke-keys" Oct 30 13:08:45.404454 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.404441 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="e420d739-6af6-408d-bb35-b5af95de2c5f" containerName="revoke-keys" Oct 30 13:08:45.446772 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.440408 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm"] Oct 30 13:08:45.476100 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.476073 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" Oct 30 13:08:45.476768 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.458844 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" Oct 30 13:08:45.477508 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.460192 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" podUID="14ae0aea-473f-4ca0-bcc0-f617b9839c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded" Oct 30 13:08:45.609353 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.609274 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:45.629744 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.629703 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be022317-9k5dl" event={"ID":"2ddaea9f-d694-4294-bbc3-3d718ae8abd1","Type":"ContainerDied","Data":"96221aba334b02e8da11c33d2aef87bb4f4affba3c9b3367102844134592eb08"} Oct 30 13:08:45.629978 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.629961 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96221aba334b02e8da11c33d2aef87bb4f4affba3c9b3367102844134592eb08" Oct 30 13:08:45.654892 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.651376 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" Oct 30 13:08:45.654892 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.652715 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" Oct 30 13:08:45.736642 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.702239 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-42ee9d\"/\"distinct-w0-42ee9d-kserve-self-signed-certs\"" Oct 30 13:08:45.736642 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.729423 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be022317-9k5dl" Oct 30 13:08:45.736642 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.731566 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" Oct 30 13:08:45.736642 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.731665 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" Oct 30 13:08:45.736642 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.731717 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" Oct 30 13:08:45.742886 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.742853 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56c47c05-d7wdp"] Oct 30 13:08:45.744028 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.744004 5922 cpu_manager.go:414] "containerMap: removing container" podUID="2ddaea9f-d694-4294-bbc3-3d718ae8abd1" containerName="revoke-keys" Oct 30 13:08:45.744165 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.744153 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ddaea9f-d694-4294-bbc3-3d718ae8abd1" containerName="revoke-keys" Oct 30 13:08:45.744478 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.744464 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ddaea9f-d694-4294-bbc3-3d718ae8abd1" containerName="revoke-keys" Oct 30 13:08:45.749302 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.749281 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" Oct 30 13:08:45.749457 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.749445 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" Oct 30 13:08:45.749583 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.749572 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56c47c05-d7wdp"] Oct 30 13:08:45.749715 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.749356 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c47c05-d7wdp" Oct 30 13:08:45.784661 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.784626 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a296df26-3a3b-40ef-a3ab-1866ecf7dcbc-tls-certs\") pod \"distinct-w0-42ee9d-kserve-f46fffc8d-r92sm\" (UID: \"a296df26-3a3b-40ef-a3ab-1866ecf7dcbc\") " pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" Oct 30 13:08:45.784829 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.784706 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbzjm\" (UniqueName: \"kubernetes.io/projected/a296df26-3a3b-40ef-a3ab-1866ecf7dcbc-kube-api-access-lbzjm\") pod \"distinct-w0-42ee9d-kserve-f46fffc8d-r92sm\" (UID: \"a296df26-3a3b-40ef-a3ab-1866ecf7dcbc\") " pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" Oct 30 13:08:45.784829 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.784741 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a296df26-3a3b-40ef-a3ab-1866ecf7dcbc-dshm\") pod \"distinct-w0-42ee9d-kserve-f46fffc8d-r92sm\" (UID: \"a296df26-3a3b-40ef-a3ab-1866ecf7dcbc\") " pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" Oct 30 13:08:45.784927 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.784894 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a296df26-3a3b-40ef-a3ab-1866ecf7dcbc-home\") pod \"distinct-w0-42ee9d-kserve-f46fffc8d-r92sm\" (UID: \"a296df26-3a3b-40ef-a3ab-1866ecf7dcbc\") " pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" Oct 30 13:08:45.784969 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.784953 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a296df26-3a3b-40ef-a3ab-1866ecf7dcbc-tmp-dir\") pod \"distinct-w0-42ee9d-kserve-f46fffc8d-r92sm\" (UID: \"a296df26-3a3b-40ef-a3ab-1866ecf7dcbc\") " pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" Oct 30 13:08:45.785073 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.785058 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a296df26-3a3b-40ef-a3ab-1866ecf7dcbc-model-cache\") pod \"distinct-w0-42ee9d-kserve-f46fffc8d-r92sm\" (UID: \"a296df26-3a3b-40ef-a3ab-1866ecf7dcbc\") " pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" Oct 30 13:08:45.886705 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.886627 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ddaea9f-d694-4294-bbc3-3d718ae8abd1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ddaea9f-d694-4294-bbc3-3d718ae8abd1-maas-api-openshift-ca\") pod \"2ddaea9f-d694-4294-bbc3-3d718ae8abd1\" (UID: \"2ddaea9f-d694-4294-bbc3-3d718ae8abd1\") " Oct 30 13:08:45.886864 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.886708 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ddaea9f-d694-4294-bbc3-3d718ae8abd1-kube-api-access-2stgk\" (UniqueName: \"kubernetes.io/projected/2ddaea9f-d694-4294-bbc3-3d718ae8abd1-kube-api-access-2stgk\") pod \"2ddaea9f-d694-4294-bbc3-3d718ae8abd1\" (UID: \"2ddaea9f-d694-4294-bbc3-3d718ae8abd1\") " Oct 30 13:08:45.886864 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.886778 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ddaea9f-d694-4294-bbc3-3d718ae8abd1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ddaea9f-d694-4294-bbc3-3d718ae8abd1-maas-api-xks-ca\") pod \"2ddaea9f-d694-4294-bbc3-3d718ae8abd1\" (UID: \"2ddaea9f-d694-4294-bbc3-3d718ae8abd1\") " Oct 30 13:08:45.886997 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.886928 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ddaea9f-d694-4294-bbc3-3d718ae8abd1-maas-api-openshift-ca" pod "2ddaea9f-d694-4294-bbc3-3d718ae8abd1" (UID: "2ddaea9f-d694-4294-bbc3-3d718ae8abd1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:45.887188 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.887164 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqqlt\" (UniqueName: \"kubernetes.io/projected/c69ddfca-b346-4241-b6f6-5a08c31db00b-kube-api-access-cqqlt\") pod \"maas-api-revoke-sub-56c47c05-d7wdp\" (UID: \"c69ddfca-b346-4241-b6f6-5a08c31db00b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c47c05-d7wdp" Oct 30 13:08:45.887259 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.887240 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a296df26-3a3b-40ef-a3ab-1866ecf7dcbc-tls-certs\") pod \"distinct-w0-42ee9d-kserve-f46fffc8d-r92sm\" (UID: \"a296df26-3a3b-40ef-a3ab-1866ecf7dcbc\") " pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" Oct 30 13:08:45.887313 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.887302 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbzjm\" (UniqueName: \"kubernetes.io/projected/a296df26-3a3b-40ef-a3ab-1866ecf7dcbc-kube-api-access-lbzjm\") pod \"distinct-w0-42ee9d-kserve-f46fffc8d-r92sm\" (UID: \"a296df26-3a3b-40ef-a3ab-1866ecf7dcbc\") " pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" Oct 30 13:08:45.887367 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.887341 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a296df26-3a3b-40ef-a3ab-1866ecf7dcbc-dshm\") pod \"distinct-w0-42ee9d-kserve-f46fffc8d-r92sm\" (UID: \"a296df26-3a3b-40ef-a3ab-1866ecf7dcbc\") " pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" Oct 30 13:08:45.887465 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.887445 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c69ddfca-b346-4241-b6f6-5a08c31db00b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56c47c05-d7wdp\" (UID: \"c69ddfca-b346-4241-b6f6-5a08c31db00b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c47c05-d7wdp" Oct 30 13:08:45.887519 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.887484 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a296df26-3a3b-40ef-a3ab-1866ecf7dcbc-home\") pod \"distinct-w0-42ee9d-kserve-f46fffc8d-r92sm\" (UID: \"a296df26-3a3b-40ef-a3ab-1866ecf7dcbc\") " pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" Oct 30 13:08:45.887576 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.887522 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a296df26-3a3b-40ef-a3ab-1866ecf7dcbc-tmp-dir\") pod \"distinct-w0-42ee9d-kserve-f46fffc8d-r92sm\" (UID: \"a296df26-3a3b-40ef-a3ab-1866ecf7dcbc\") " pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" Oct 30 13:08:45.887701 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.887680 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a296df26-3a3b-40ef-a3ab-1866ecf7dcbc-model-cache\") pod \"distinct-w0-42ee9d-kserve-f46fffc8d-r92sm\" (UID: \"a296df26-3a3b-40ef-a3ab-1866ecf7dcbc\") " pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" Oct 30 13:08:45.887763 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.887731 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c69ddfca-b346-4241-b6f6-5a08c31db00b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56c47c05-d7wdp\" (UID: \"c69ddfca-b346-4241-b6f6-5a08c31db00b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c47c05-d7wdp" Oct 30 13:08:45.888100 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.888076 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ddaea9f-d694-4294-bbc3-3d718ae8abd1-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:45.888209 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.888152 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a296df26-3a3b-40ef-a3ab-1866ecf7dcbc-home\") pod \"distinct-w0-42ee9d-kserve-f46fffc8d-r92sm\" (UID: \"a296df26-3a3b-40ef-a3ab-1866ecf7dcbc\") " pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" Oct 30 13:08:45.888716 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.888689 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a296df26-3a3b-40ef-a3ab-1866ecf7dcbc-tmp-dir\") pod \"distinct-w0-42ee9d-kserve-f46fffc8d-r92sm\" (UID: \"a296df26-3a3b-40ef-a3ab-1866ecf7dcbc\") " pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" Oct 30 13:08:45.889571 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.889542 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a296df26-3a3b-40ef-a3ab-1866ecf7dcbc-model-cache\") pod \"distinct-w0-42ee9d-kserve-f46fffc8d-r92sm\" (UID: \"a296df26-3a3b-40ef-a3ab-1866ecf7dcbc\") " pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" Oct 30 13:08:45.891358 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.890827 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a296df26-3a3b-40ef-a3ab-1866ecf7dcbc-dshm\") pod \"distinct-w0-42ee9d-kserve-f46fffc8d-r92sm\" (UID: \"a296df26-3a3b-40ef-a3ab-1866ecf7dcbc\") " pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" Oct 30 13:08:45.891358 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.891316 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a296df26-3a3b-40ef-a3ab-1866ecf7dcbc-tls-certs\") pod \"distinct-w0-42ee9d-kserve-f46fffc8d-r92sm\" (UID: \"a296df26-3a3b-40ef-a3ab-1866ecf7dcbc\") " pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" Oct 30 13:08:45.899306 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.899278 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ddaea9f-d694-4294-bbc3-3d718ae8abd1-maas-api-xks-ca" pod "2ddaea9f-d694-4294-bbc3-3d718ae8abd1" (UID: "2ddaea9f-d694-4294-bbc3-3d718ae8abd1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:45.899409 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.899378 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ddaea9f-d694-4294-bbc3-3d718ae8abd1-kube-api-access-2stgk" pod "2ddaea9f-d694-4294-bbc3-3d718ae8abd1" (UID: "2ddaea9f-d694-4294-bbc3-3d718ae8abd1"). InnerVolumeSpecName "kube-api-access-2stgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:45.914297 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.914267 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbzjm\" (UniqueName: \"kubernetes.io/projected/a296df26-3a3b-40ef-a3ab-1866ecf7dcbc-kube-api-access-lbzjm\") pod \"distinct-w0-42ee9d-kserve-f46fffc8d-r92sm\" (UID: \"a296df26-3a3b-40ef-a3ab-1866ecf7dcbc\") " pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" Oct 30 13:08:45.989516 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.989472 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c69ddfca-b346-4241-b6f6-5a08c31db00b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56c47c05-d7wdp\" (UID: \"c69ddfca-b346-4241-b6f6-5a08c31db00b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c47c05-d7wdp" Oct 30 13:08:45.989689 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.989673 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqqlt\" (UniqueName: \"kubernetes.io/projected/c69ddfca-b346-4241-b6f6-5a08c31db00b-kube-api-access-cqqlt\") pod \"maas-api-revoke-sub-56c47c05-d7wdp\" (UID: \"c69ddfca-b346-4241-b6f6-5a08c31db00b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c47c05-d7wdp" Oct 30 13:08:45.989801 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.989786 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c69ddfca-b346-4241-b6f6-5a08c31db00b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56c47c05-d7wdp\" (UID: \"c69ddfca-b346-4241-b6f6-5a08c31db00b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c47c05-d7wdp" Oct 30 13:08:45.990135 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.990108 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ddaea9f-d694-4294-bbc3-3d718ae8abd1-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:45.990283 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.990137 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2stgk\" (UniqueName: \"kubernetes.io/projected/2ddaea9f-d694-4294-bbc3-3d718ae8abd1-kube-api-access-2stgk\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:45.990283 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.990185 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c69ddfca-b346-4241-b6f6-5a08c31db00b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56c47c05-d7wdp\" (UID: \"c69ddfca-b346-4241-b6f6-5a08c31db00b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c47c05-d7wdp" Oct 30 13:08:45.992295 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.992267 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" Oct 30 13:08:45.992574 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:45.992555 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c69ddfca-b346-4241-b6f6-5a08c31db00b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56c47c05-d7wdp\" (UID: \"c69ddfca-b346-4241-b6f6-5a08c31db00b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c47c05-d7wdp" Oct 30 13:08:46.016211 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:46.016183 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqqlt\" (UniqueName: \"kubernetes.io/projected/c69ddfca-b346-4241-b6f6-5a08c31db00b-kube-api-access-cqqlt\") pod \"maas-api-revoke-sub-56c47c05-d7wdp\" (UID: \"c69ddfca-b346-4241-b6f6-5a08c31db00b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c47c05-d7wdp" Oct 30 13:08:46.067935 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:46.067905 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c47c05-d7wdp" Oct 30 13:08:46.325012 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:46.324981 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm"] Oct 30 13:08:46.325438 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:08:46.325416 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda296df26_3a3b_40ef_a3ab_1866ecf7dcbc.slice/crio-8e44954240c2e15513fc436a61e8d7649d0148dd59f598e0cac55fd670e090c9 WatchSource:0}: Error finding container 8e44954240c2e15513fc436a61e8d7649d0148dd59f598e0cac55fd670e090c9: Status 404 returned error can't find the container with id 8e44954240c2e15513fc436a61e8d7649d0148dd59f598e0cac55fd670e090c9 Oct 30 13:08:46.382466 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:08:46.382441 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc69ddfca_b346_4241_b6f6_5a08c31db00b.slice/crio-f5d3f03f45e17da999f87581f28a25ba46920e01f8d4fa82360f92e9d1a77905 WatchSource:0}: Error finding container f5d3f03f45e17da999f87581f28a25ba46920e01f8d4fa82360f92e9d1a77905: Status 404 returned error can't find the container with id f5d3f03f45e17da999f87581f28a25ba46920e01f8d4fa82360f92e9d1a77905 Oct 30 13:08:46.382927 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:46.382903 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56c47c05-d7wdp"] Oct 30 13:08:46.639179 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:46.639143 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" event={"ID":"a296df26-3a3b-40ef-a3ab-1866ecf7dcbc","Type":"ContainerStarted","Data":"ec236ae48eaf54b6aa004cec816f861bef7139ccc86a08636af684361fb347c7"} Oct 30 13:08:46.639336 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:46.639188 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" event={"ID":"a296df26-3a3b-40ef-a3ab-1866ecf7dcbc","Type":"ContainerStarted","Data":"8e44954240c2e15513fc436a61e8d7649d0148dd59f598e0cac55fd670e090c9"} Oct 30 13:08:46.640592 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:46.640566 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c47c05-d7wdp" event={"ID":"c69ddfca-b346-4241-b6f6-5a08c31db00b","Type":"ContainerStarted","Data":"f5d3f03f45e17da999f87581f28a25ba46920e01f8d4fa82360f92e9d1a77905"} Oct 30 13:08:46.640737 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:46.640647 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be022317-9k5dl" Oct 30 13:08:46.685468 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:46.685426 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" podStartSLOduration=3.68541237 podStartE2EDuration="3.68541237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:46.68178269 +0000 UTC m=+983.151198437" watchObservedRunningTime="2026-10-30 13:08:46.68541237 +0000 UTC m=+983.154828155" Oct 30 13:08:46.992860 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:46.992828 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" Oct 30 13:08:47.659675 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:47.659575 5922 generic.go:356] "Generic (PLEG): container finished" podID="c69ddfca-b346-4241-b6f6-5a08c31db00b" containerID="3637756e4f5e53072f89a762b3d88e136366aeb954218b78199f7a271b5a7d25" exitCode=0 Oct 30 13:08:47.659675 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:47.659658 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c47c05-d7wdp" event={"ID":"c69ddfca-b346-4241-b6f6-5a08c31db00b","Type":"ContainerDied","Data":"3637756e4f5e53072f89a762b3d88e136366aeb954218b78199f7a271b5a7d25"} Oct 30 13:08:48.824333 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:48.824307 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c47c05-d7wdp" Oct 30 13:08:48.931224 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:48.931149 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c69ddfca-b346-4241-b6f6-5a08c31db00b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c69ddfca-b346-4241-b6f6-5a08c31db00b-maas-api-xks-ca\") pod \"c69ddfca-b346-4241-b6f6-5a08c31db00b\" (UID: \"c69ddfca-b346-4241-b6f6-5a08c31db00b\") " Oct 30 13:08:48.931364 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:48.931313 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c69ddfca-b346-4241-b6f6-5a08c31db00b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c69ddfca-b346-4241-b6f6-5a08c31db00b-maas-api-openshift-ca\") pod \"c69ddfca-b346-4241-b6f6-5a08c31db00b\" (UID: \"c69ddfca-b346-4241-b6f6-5a08c31db00b\") " Oct 30 13:08:48.931468 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:48.931452 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c69ddfca-b346-4241-b6f6-5a08c31db00b-kube-api-access-cqqlt\" (UniqueName: \"kubernetes.io/projected/c69ddfca-b346-4241-b6f6-5a08c31db00b-kube-api-access-cqqlt\") pod \"c69ddfca-b346-4241-b6f6-5a08c31db00b\" (UID: \"c69ddfca-b346-4241-b6f6-5a08c31db00b\") " Oct 30 13:08:48.931738 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:48.931708 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c69ddfca-b346-4241-b6f6-5a08c31db00b-maas-api-openshift-ca" pod "c69ddfca-b346-4241-b6f6-5a08c31db00b" (UID: "c69ddfca-b346-4241-b6f6-5a08c31db00b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:48.932073 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:48.932048 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c69ddfca-b346-4241-b6f6-5a08c31db00b-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:48.933376 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:48.933346 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c69ddfca-b346-4241-b6f6-5a08c31db00b-maas-api-xks-ca" pod "c69ddfca-b346-4241-b6f6-5a08c31db00b" (UID: "c69ddfca-b346-4241-b6f6-5a08c31db00b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:48.934002 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:48.933969 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c69ddfca-b346-4241-b6f6-5a08c31db00b-kube-api-access-cqqlt" pod "c69ddfca-b346-4241-b6f6-5a08c31db00b" (UID: "c69ddfca-b346-4241-b6f6-5a08c31db00b"). InnerVolumeSpecName "kube-api-access-cqqlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:49.033792 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:49.033765 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c69ddfca-b346-4241-b6f6-5a08c31db00b-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:49.033792 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:49.033791 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqqlt\" (UniqueName: \"kubernetes.io/projected/c69ddfca-b346-4241-b6f6-5a08c31db00b-kube-api-access-cqqlt\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:49.129127 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:49.129094 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71e7d561-h96hh"] Oct 30 13:08:49.129808 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:49.129790 5922 cpu_manager.go:414] "containerMap: removing container" podUID="c69ddfca-b346-4241-b6f6-5a08c31db00b" containerName="revoke-keys" Oct 30 13:08:49.129912 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:49.129809 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c69ddfca-b346-4241-b6f6-5a08c31db00b" containerName="revoke-keys" Oct 30 13:08:49.130008 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:49.129994 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="c69ddfca-b346-4241-b6f6-5a08c31db00b" containerName="revoke-keys" Oct 30 13:08:49.133922 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:49.133896 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e7d561-h96hh" Oct 30 13:08:49.157454 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:49.157420 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71e7d561-h96hh"] Oct 30 13:08:49.235926 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:49.235891 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16ba72c8-5f1d-492a-8140-ee4171654b96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71e7d561-h96hh\" (UID: \"16ba72c8-5f1d-492a-8140-ee4171654b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e7d561-h96hh" Oct 30 13:08:49.236090 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:49.235968 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqnfn\" (UniqueName: \"kubernetes.io/projected/16ba72c8-5f1d-492a-8140-ee4171654b96-kube-api-access-rqnfn\") pod \"maas-api-revoke-sub-71e7d561-h96hh\" (UID: \"16ba72c8-5f1d-492a-8140-ee4171654b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e7d561-h96hh" Oct 30 13:08:49.236156 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:49.236120 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16ba72c8-5f1d-492a-8140-ee4171654b96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71e7d561-h96hh\" (UID: \"16ba72c8-5f1d-492a-8140-ee4171654b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e7d561-h96hh" Oct 30 13:08:49.337817 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:49.337786 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16ba72c8-5f1d-492a-8140-ee4171654b96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71e7d561-h96hh\" (UID: \"16ba72c8-5f1d-492a-8140-ee4171654b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e7d561-h96hh" Oct 30 13:08:49.337988 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:49.337973 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16ba72c8-5f1d-492a-8140-ee4171654b96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71e7d561-h96hh\" (UID: \"16ba72c8-5f1d-492a-8140-ee4171654b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e7d561-h96hh" Oct 30 13:08:49.338113 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:49.338016 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqnfn\" (UniqueName: \"kubernetes.io/projected/16ba72c8-5f1d-492a-8140-ee4171654b96-kube-api-access-rqnfn\") pod \"maas-api-revoke-sub-71e7d561-h96hh\" (UID: \"16ba72c8-5f1d-492a-8140-ee4171654b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e7d561-h96hh" Oct 30 13:08:49.338663 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:49.338638 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16ba72c8-5f1d-492a-8140-ee4171654b96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71e7d561-h96hh\" (UID: \"16ba72c8-5f1d-492a-8140-ee4171654b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e7d561-h96hh" Oct 30 13:08:49.339852 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:49.339834 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16ba72c8-5f1d-492a-8140-ee4171654b96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71e7d561-h96hh\" (UID: \"16ba72c8-5f1d-492a-8140-ee4171654b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e7d561-h96hh" Oct 30 13:08:49.363392 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:49.363366 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqnfn\" (UniqueName: \"kubernetes.io/projected/16ba72c8-5f1d-492a-8140-ee4171654b96-kube-api-access-rqnfn\") pod \"maas-api-revoke-sub-71e7d561-h96hh\" (UID: \"16ba72c8-5f1d-492a-8140-ee4171654b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e7d561-h96hh" Oct 30 13:08:49.464219 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:49.464189 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e7d561-h96hh" Oct 30 13:08:49.526242 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:49.521016 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-555d7dd0-vtkp6"] Oct 30 13:08:49.527852 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:49.527825 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555d7dd0-vtkp6" Oct 30 13:08:49.551851 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:49.551815 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-555d7dd0-vtkp6"] Oct 30 13:08:49.651914 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:49.651874 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlbt9\" (UniqueName: \"kubernetes.io/projected/fefc8aa7-6d48-40b2-8d64-44a1ad767c1a-kube-api-access-tlbt9\") pod \"maas-api-revoke-sub-555d7dd0-vtkp6\" (UID: \"fefc8aa7-6d48-40b2-8d64-44a1ad767c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555d7dd0-vtkp6" Oct 30 13:08:49.651914 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:49.651911 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fefc8aa7-6d48-40b2-8d64-44a1ad767c1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-555d7dd0-vtkp6\" (UID: \"fefc8aa7-6d48-40b2-8d64-44a1ad767c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555d7dd0-vtkp6" Oct 30 13:08:49.652144 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:49.651931 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fefc8aa7-6d48-40b2-8d64-44a1ad767c1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-555d7dd0-vtkp6\" (UID: \"fefc8aa7-6d48-40b2-8d64-44a1ad767c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555d7dd0-vtkp6" Oct 30 13:08:49.678146 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:49.678120 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c47c05-d7wdp" Oct 30 13:08:49.678146 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:49.678133 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c47c05-d7wdp" event={"ID":"c69ddfca-b346-4241-b6f6-5a08c31db00b","Type":"ContainerDied","Data":"f5d3f03f45e17da999f87581f28a25ba46920e01f8d4fa82360f92e9d1a77905"} Oct 30 13:08:49.678314 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:49.678179 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5d3f03f45e17da999f87581f28a25ba46920e01f8d4fa82360f92e9d1a77905" Oct 30 13:08:49.753813 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:49.753781 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlbt9\" (UniqueName: \"kubernetes.io/projected/fefc8aa7-6d48-40b2-8d64-44a1ad767c1a-kube-api-access-tlbt9\") pod \"maas-api-revoke-sub-555d7dd0-vtkp6\" (UID: \"fefc8aa7-6d48-40b2-8d64-44a1ad767c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555d7dd0-vtkp6" Oct 30 13:08:49.753954 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:49.753818 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fefc8aa7-6d48-40b2-8d64-44a1ad767c1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-555d7dd0-vtkp6\" (UID: \"fefc8aa7-6d48-40b2-8d64-44a1ad767c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555d7dd0-vtkp6" Oct 30 13:08:49.753954 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:49.753837 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fefc8aa7-6d48-40b2-8d64-44a1ad767c1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-555d7dd0-vtkp6\" (UID: \"fefc8aa7-6d48-40b2-8d64-44a1ad767c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555d7dd0-vtkp6" Oct 30 13:08:49.754387 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:49.754365 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fefc8aa7-6d48-40b2-8d64-44a1ad767c1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-555d7dd0-vtkp6\" (UID: \"fefc8aa7-6d48-40b2-8d64-44a1ad767c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555d7dd0-vtkp6" Oct 30 13:08:49.756011 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:49.755993 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fefc8aa7-6d48-40b2-8d64-44a1ad767c1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-555d7dd0-vtkp6\" (UID: \"fefc8aa7-6d48-40b2-8d64-44a1ad767c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555d7dd0-vtkp6" Oct 30 13:08:49.775474 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:49.775448 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71e7d561-h96hh"] Oct 30 13:08:49.775575 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:08:49.775503 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16ba72c8_5f1d_492a_8140_ee4171654b96.slice/crio-687a5ba385755fc01be5f20ece6f7276d1cc243423aee0df7da5a4aef5858731 WatchSource:0}: Error finding container 687a5ba385755fc01be5f20ece6f7276d1cc243423aee0df7da5a4aef5858731: Status 404 returned error can't find the container with id 687a5ba385755fc01be5f20ece6f7276d1cc243423aee0df7da5a4aef5858731 Oct 30 13:08:49.778649 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:49.778629 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlbt9\" (UniqueName: \"kubernetes.io/projected/fefc8aa7-6d48-40b2-8d64-44a1ad767c1a-kube-api-access-tlbt9\") pod \"maas-api-revoke-sub-555d7dd0-vtkp6\" (UID: \"fefc8aa7-6d48-40b2-8d64-44a1ad767c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555d7dd0-vtkp6" Oct 30 13:08:49.863210 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:49.863187 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555d7dd0-vtkp6" Oct 30 13:08:49.905291 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:49.905254 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:50.176284 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:50.176257 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-555d7dd0-vtkp6"] Oct 30 13:08:50.176397 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:08:50.176351 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfefc8aa7_6d48_40b2_8d64_44a1ad767c1a.slice/crio-a99bc1e36593dd0e1cffd8b00e02f235743c053729db6e5b458a211e5a0ada45 WatchSource:0}: Error finding container a99bc1e36593dd0e1cffd8b00e02f235743c053729db6e5b458a211e5a0ada45: Status 404 returned error can't find the container with id a99bc1e36593dd0e1cffd8b00e02f235743c053729db6e5b458a211e5a0ada45 Oct 30 13:08:50.689057 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:50.689024 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555d7dd0-vtkp6" event={"ID":"fefc8aa7-6d48-40b2-8d64-44a1ad767c1a","Type":"ContainerStarted","Data":"36c01587692285970bbb3344ea9205c08a95e6fee15a2d58c822cb5376f092a9"} Oct 30 13:08:50.689183 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:50.689077 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555d7dd0-vtkp6" event={"ID":"fefc8aa7-6d48-40b2-8d64-44a1ad767c1a","Type":"ContainerStarted","Data":"a99bc1e36593dd0e1cffd8b00e02f235743c053729db6e5b458a211e5a0ada45"} Oct 30 13:08:50.690884 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:50.690857 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e7d561-h96hh" event={"ID":"16ba72c8-5f1d-492a-8140-ee4171654b96","Type":"ContainerStarted","Data":"5e300b0795401d3e348e75077417cac184169b31b32e4605e2dc933525ae03b1"} Oct 30 13:08:50.690988 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:50.690906 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e7d561-h96hh" event={"ID":"16ba72c8-5f1d-492a-8140-ee4171654b96","Type":"ContainerStarted","Data":"687a5ba385755fc01be5f20ece6f7276d1cc243423aee0df7da5a4aef5858731"} Oct 30 13:08:50.735327 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:50.735238 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555d7dd0-vtkp6" podStartSLOduration=1.735223971 podStartE2EDuration="1.735223971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:50.730847175 +0000 UTC m=+987.200262931" watchObservedRunningTime="2026-10-30 13:08:50.735223971 +0000 UTC m=+987.204639717" Oct 30 13:08:51.588142 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:51.588094 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:51.703824 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:51.703791 5922 generic.go:356] "Generic (PLEG): container finished" podID="16ba72c8-5f1d-492a-8140-ee4171654b96" containerID="5e300b0795401d3e348e75077417cac184169b31b32e4605e2dc933525ae03b1" exitCode=0 Oct 30 13:08:51.703973 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:51.703873 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e7d561-h96hh" event={"ID":"16ba72c8-5f1d-492a-8140-ee4171654b96","Type":"ContainerDied","Data":"5e300b0795401d3e348e75077417cac184169b31b32e4605e2dc933525ae03b1"} Oct 30 13:08:51.705760 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:51.705737 5922 generic.go:356] "Generic (PLEG): container finished" podID="fefc8aa7-6d48-40b2-8d64-44a1ad767c1a" containerID="36c01587692285970bbb3344ea9205c08a95e6fee15a2d58c822cb5376f092a9" exitCode=0 Oct 30 13:08:51.705885 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:51.705838 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555d7dd0-vtkp6" event={"ID":"fefc8aa7-6d48-40b2-8d64-44a1ad767c1a","Type":"ContainerDied","Data":"36c01587692285970bbb3344ea9205c08a95e6fee15a2d58c822cb5376f092a9"} Oct 30 13:08:51.884378 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:51.884344 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e7d561-h96hh" Oct 30 13:08:51.987854 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:51.987814 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16ba72c8-5f1d-492a-8140-ee4171654b96-kube-api-access-rqnfn\" (UniqueName: \"kubernetes.io/projected/16ba72c8-5f1d-492a-8140-ee4171654b96-kube-api-access-rqnfn\") pod \"16ba72c8-5f1d-492a-8140-ee4171654b96\" (UID: \"16ba72c8-5f1d-492a-8140-ee4171654b96\") " Oct 30 13:08:51.988001 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:51.987950 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16ba72c8-5f1d-492a-8140-ee4171654b96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16ba72c8-5f1d-492a-8140-ee4171654b96-maas-api-openshift-ca\") pod \"16ba72c8-5f1d-492a-8140-ee4171654b96\" (UID: \"16ba72c8-5f1d-492a-8140-ee4171654b96\") " Oct 30 13:08:51.988079 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:51.988062 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16ba72c8-5f1d-492a-8140-ee4171654b96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16ba72c8-5f1d-492a-8140-ee4171654b96-maas-api-xks-ca\") pod \"16ba72c8-5f1d-492a-8140-ee4171654b96\" (UID: \"16ba72c8-5f1d-492a-8140-ee4171654b96\") " Oct 30 13:08:51.988341 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:51.988313 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16ba72c8-5f1d-492a-8140-ee4171654b96-maas-api-openshift-ca" pod "16ba72c8-5f1d-492a-8140-ee4171654b96" (UID: "16ba72c8-5f1d-492a-8140-ee4171654b96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:51.988651 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:51.988635 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16ba72c8-5f1d-492a-8140-ee4171654b96-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:51.989869 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:51.989838 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16ba72c8-5f1d-492a-8140-ee4171654b96-kube-api-access-rqnfn" pod "16ba72c8-5f1d-492a-8140-ee4171654b96" (UID: "16ba72c8-5f1d-492a-8140-ee4171654b96"). InnerVolumeSpecName "kube-api-access-rqnfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:51.989983 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:51.989966 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16ba72c8-5f1d-492a-8140-ee4171654b96-maas-api-xks-ca" pod "16ba72c8-5f1d-492a-8140-ee4171654b96" (UID: "16ba72c8-5f1d-492a-8140-ee4171654b96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:52.089653 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:52.089599 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqnfn\" (UniqueName: \"kubernetes.io/projected/16ba72c8-5f1d-492a-8140-ee4171654b96-kube-api-access-rqnfn\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:52.089653 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:52.089654 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16ba72c8-5f1d-492a-8140-ee4171654b96-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:52.715410 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:52.715385 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e7d561-h96hh" Oct 30 13:08:52.715899 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:52.715418 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e7d561-h96hh" event={"ID":"16ba72c8-5f1d-492a-8140-ee4171654b96","Type":"ContainerDied","Data":"687a5ba385755fc01be5f20ece6f7276d1cc243423aee0df7da5a4aef5858731"} Oct 30 13:08:52.715899 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:52.715474 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="687a5ba385755fc01be5f20ece6f7276d1cc243423aee0df7da5a4aef5858731" Oct 30 13:08:52.859877 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:52.859852 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555d7dd0-vtkp6" Oct 30 13:08:53.000452 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:53.000382 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fefc8aa7-6d48-40b2-8d64-44a1ad767c1a-kube-api-access-tlbt9\" (UniqueName: \"kubernetes.io/projected/fefc8aa7-6d48-40b2-8d64-44a1ad767c1a-kube-api-access-tlbt9\") pod \"fefc8aa7-6d48-40b2-8d64-44a1ad767c1a\" (UID: \"fefc8aa7-6d48-40b2-8d64-44a1ad767c1a\") " Oct 30 13:08:53.000452 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:53.000418 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fefc8aa7-6d48-40b2-8d64-44a1ad767c1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fefc8aa7-6d48-40b2-8d64-44a1ad767c1a-maas-api-xks-ca\") pod \"fefc8aa7-6d48-40b2-8d64-44a1ad767c1a\" (UID: \"fefc8aa7-6d48-40b2-8d64-44a1ad767c1a\") " Oct 30 13:08:53.000727 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:53.000632 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fefc8aa7-6d48-40b2-8d64-44a1ad767c1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fefc8aa7-6d48-40b2-8d64-44a1ad767c1a-maas-api-openshift-ca\") pod \"fefc8aa7-6d48-40b2-8d64-44a1ad767c1a\" (UID: \"fefc8aa7-6d48-40b2-8d64-44a1ad767c1a\") " Oct 30 13:08:53.000976 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:53.000947 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fefc8aa7-6d48-40b2-8d64-44a1ad767c1a-maas-api-openshift-ca" pod "fefc8aa7-6d48-40b2-8d64-44a1ad767c1a" (UID: "fefc8aa7-6d48-40b2-8d64-44a1ad767c1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:53.002550 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:53.002526 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fefc8aa7-6d48-40b2-8d64-44a1ad767c1a-maas-api-xks-ca" pod "fefc8aa7-6d48-40b2-8d64-44a1ad767c1a" (UID: "fefc8aa7-6d48-40b2-8d64-44a1ad767c1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:53.002550 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:53.002539 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fefc8aa7-6d48-40b2-8d64-44a1ad767c1a-kube-api-access-tlbt9" pod "fefc8aa7-6d48-40b2-8d64-44a1ad767c1a" (UID: "fefc8aa7-6d48-40b2-8d64-44a1ad767c1a"). InnerVolumeSpecName "kube-api-access-tlbt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:53.102733 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:53.102709 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlbt9\" (UniqueName: \"kubernetes.io/projected/fefc8aa7-6d48-40b2-8d64-44a1ad767c1a-kube-api-access-tlbt9\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:53.102733 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:53.102733 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fefc8aa7-6d48-40b2-8d64-44a1ad767c1a-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:53.102854 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:53.102746 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fefc8aa7-6d48-40b2-8d64-44a1ad767c1a-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:53.401146 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:53.401062 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:53.553851 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:08:53.553818 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:08:53.639545 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:53.639506 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:53.724416 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:53.724391 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555d7dd0-vtkp6" Oct 30 13:08:53.724785 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:53.724427 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555d7dd0-vtkp6" event={"ID":"fefc8aa7-6d48-40b2-8d64-44a1ad767c1a","Type":"ContainerDied","Data":"a99bc1e36593dd0e1cffd8b00e02f235743c053729db6e5b458a211e5a0ada45"} Oct 30 13:08:53.724785 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:53.724471 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a99bc1e36593dd0e1cffd8b00e02f235743c053729db6e5b458a211e5a0ada45" Oct 30 13:08:55.179676 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:55.179640 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:55.855650 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:55.855582 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:55.968373 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:55.968329 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:55.988752 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:55.988721 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e3e4f23-5wnb5"] Oct 30 13:08:55.989648 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:55.989630 5922 cpu_manager.go:414] "containerMap: removing container" podUID="fefc8aa7-6d48-40b2-8d64-44a1ad767c1a" containerName="revoke-keys" Oct 30 13:08:55.989705 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:55.989652 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fefc8aa7-6d48-40b2-8d64-44a1ad767c1a" containerName="revoke-keys" Oct 30 13:08:55.989705 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:55.989692 5922 cpu_manager.go:414] "containerMap: removing container" podUID="16ba72c8-5f1d-492a-8140-ee4171654b96" containerName="revoke-keys" Oct 30 13:08:55.989705 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:55.989701 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16ba72c8-5f1d-492a-8140-ee4171654b96" containerName="revoke-keys" Oct 30 13:08:55.989871 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:55.989860 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="fefc8aa7-6d48-40b2-8d64-44a1ad767c1a" containerName="revoke-keys" Oct 30 13:08:55.989902 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:55.989881 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="16ba72c8-5f1d-492a-8140-ee4171654b96" containerName="revoke-keys" Oct 30 13:08:55.994538 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:55.994516 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" Oct 30 13:08:55.994681 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:55.994629 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e3e4f23-5wnb5" Oct 30 13:08:56.006272 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:56.006252 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" Oct 30 13:08:56.015737 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:56.015714 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e3e4f23-5wnb5"] Oct 30 13:08:56.021217 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:56.021198 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:08:56.021362 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:56.021340 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:08:56.021518 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:56.021503 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:08:56.022149 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:56.022126 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:08:56.142317 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:56.142245 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4931dc41-6e44-4365-ad2d-f0c6ae9a44d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e3e4f23-5wnb5\" (UID: \"4931dc41-6e44-4365-ad2d-f0c6ae9a44d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e3e4f23-5wnb5" Oct 30 13:08:56.142317 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:56.142273 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwf5p\" (UniqueName: \"kubernetes.io/projected/4931dc41-6e44-4365-ad2d-f0c6ae9a44d5-kube-api-access-dwf5p\") pod \"maas-api-revoke-sub-2e3e4f23-5wnb5\" (UID: \"4931dc41-6e44-4365-ad2d-f0c6ae9a44d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e3e4f23-5wnb5" Oct 30 13:08:56.142473 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:56.142407 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4931dc41-6e44-4365-ad2d-f0c6ae9a44d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e3e4f23-5wnb5\" (UID: \"4931dc41-6e44-4365-ad2d-f0c6ae9a44d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e3e4f23-5wnb5" Oct 30 13:08:56.244155 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:56.244125 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4931dc41-6e44-4365-ad2d-f0c6ae9a44d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e3e4f23-5wnb5\" (UID: \"4931dc41-6e44-4365-ad2d-f0c6ae9a44d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e3e4f23-5wnb5" Oct 30 13:08:56.244529 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:56.244168 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwf5p\" (UniqueName: \"kubernetes.io/projected/4931dc41-6e44-4365-ad2d-f0c6ae9a44d5-kube-api-access-dwf5p\") pod \"maas-api-revoke-sub-2e3e4f23-5wnb5\" (UID: \"4931dc41-6e44-4365-ad2d-f0c6ae9a44d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e3e4f23-5wnb5" Oct 30 13:08:56.244529 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:56.244257 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4931dc41-6e44-4365-ad2d-f0c6ae9a44d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e3e4f23-5wnb5\" (UID: \"4931dc41-6e44-4365-ad2d-f0c6ae9a44d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e3e4f23-5wnb5" Oct 30 13:08:56.244984 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:56.244963 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4931dc41-6e44-4365-ad2d-f0c6ae9a44d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e3e4f23-5wnb5\" (UID: \"4931dc41-6e44-4365-ad2d-f0c6ae9a44d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e3e4f23-5wnb5" Oct 30 13:08:56.246514 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:56.246487 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4931dc41-6e44-4365-ad2d-f0c6ae9a44d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e3e4f23-5wnb5\" (UID: \"4931dc41-6e44-4365-ad2d-f0c6ae9a44d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e3e4f23-5wnb5" Oct 30 13:08:56.269333 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:56.269307 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwf5p\" (UniqueName: \"kubernetes.io/projected/4931dc41-6e44-4365-ad2d-f0c6ae9a44d5-kube-api-access-dwf5p\") pod \"maas-api-revoke-sub-2e3e4f23-5wnb5\" (UID: \"4931dc41-6e44-4365-ad2d-f0c6ae9a44d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e3e4f23-5wnb5" Oct 30 13:08:56.310431 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:56.310409 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e3e4f23-5wnb5" Oct 30 13:08:56.616407 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:56.616376 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e3e4f23-5wnb5"] Oct 30 13:08:56.616649 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:08:56.616597 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4931dc41_6e44_4365_ad2d_f0c6ae9a44d5.slice/crio-cd8f64d33379f8b1abbae548dd9f8feae5dafca5391d57304a839646948eead9 WatchSource:0}: Error finding container cd8f64d33379f8b1abbae548dd9f8feae5dafca5391d57304a839646948eead9: Status 404 returned error can't find the container with id cd8f64d33379f8b1abbae548dd9f8feae5dafca5391d57304a839646948eead9 Oct 30 13:08:56.752334 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:56.752308 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e3e4f23-5wnb5" event={"ID":"4931dc41-6e44-4365-ad2d-f0c6ae9a44d5","Type":"ContainerStarted","Data":"cd8f64d33379f8b1abbae548dd9f8feae5dafca5391d57304a839646948eead9"} Oct 30 13:08:56.764218 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:56.764196 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" Oct 30 13:08:57.093898 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:57.093864 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:57.761246 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:57.761218 5922 generic.go:356] "Generic (PLEG): container finished" podID="4931dc41-6e44-4365-ad2d-f0c6ae9a44d5" containerID="6086d058ca6acbc00bb7475ab07d68a2eccfda3d4256ce486c01678cf03a9195" exitCode=0 Oct 30 13:08:57.761601 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:57.761302 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e3e4f23-5wnb5" event={"ID":"4931dc41-6e44-4365-ad2d-f0c6ae9a44d5","Type":"ContainerDied","Data":"6086d058ca6acbc00bb7475ab07d68a2eccfda3d4256ce486c01678cf03a9195"} Oct 30 13:08:58.775844 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:58.775809 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:58.963390 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:58.963362 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e3e4f23-5wnb5" Oct 30 13:08:59.080863 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:59.080798 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4931dc41-6e44-4365-ad2d-f0c6ae9a44d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4931dc41-6e44-4365-ad2d-f0c6ae9a44d5-maas-api-openshift-ca\") pod \"4931dc41-6e44-4365-ad2d-f0c6ae9a44d5\" (UID: \"4931dc41-6e44-4365-ad2d-f0c6ae9a44d5\") " Oct 30 13:08:59.080987 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:59.080939 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4931dc41-6e44-4365-ad2d-f0c6ae9a44d5-kube-api-access-dwf5p\" (UniqueName: \"kubernetes.io/projected/4931dc41-6e44-4365-ad2d-f0c6ae9a44d5-kube-api-access-dwf5p\") pod \"4931dc41-6e44-4365-ad2d-f0c6ae9a44d5\" (UID: \"4931dc41-6e44-4365-ad2d-f0c6ae9a44d5\") " Oct 30 13:08:59.080987 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:59.080974 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4931dc41-6e44-4365-ad2d-f0c6ae9a44d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4931dc41-6e44-4365-ad2d-f0c6ae9a44d5-maas-api-xks-ca\") pod \"4931dc41-6e44-4365-ad2d-f0c6ae9a44d5\" (UID: \"4931dc41-6e44-4365-ad2d-f0c6ae9a44d5\") " Oct 30 13:08:59.081143 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:59.081113 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4931dc41-6e44-4365-ad2d-f0c6ae9a44d5-maas-api-openshift-ca" pod "4931dc41-6e44-4365-ad2d-f0c6ae9a44d5" (UID: "4931dc41-6e44-4365-ad2d-f0c6ae9a44d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:59.081414 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:59.081399 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4931dc41-6e44-4365-ad2d-f0c6ae9a44d5-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:59.083078 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:59.083054 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4931dc41-6e44-4365-ad2d-f0c6ae9a44d5-kube-api-access-dwf5p" pod "4931dc41-6e44-4365-ad2d-f0c6ae9a44d5" (UID: "4931dc41-6e44-4365-ad2d-f0c6ae9a44d5"). InnerVolumeSpecName "kube-api-access-dwf5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:59.083172 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:59.083079 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4931dc41-6e44-4365-ad2d-f0c6ae9a44d5-maas-api-xks-ca" pod "4931dc41-6e44-4365-ad2d-f0c6ae9a44d5" (UID: "4931dc41-6e44-4365-ad2d-f0c6ae9a44d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:59.183045 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:59.183018 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwf5p\" (UniqueName: \"kubernetes.io/projected/4931dc41-6e44-4365-ad2d-f0c6ae9a44d5-kube-api-access-dwf5p\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:59.183045 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:59.183042 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4931dc41-6e44-4365-ad2d-f0c6ae9a44d5-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:08:59.780025 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:59.779998 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e3e4f23-5wnb5" Oct 30 13:08:59.780437 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:59.780033 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e3e4f23-5wnb5" event={"ID":"4931dc41-6e44-4365-ad2d-f0c6ae9a44d5","Type":"ContainerDied","Data":"cd8f64d33379f8b1abbae548dd9f8feae5dafca5391d57304a839646948eead9"} Oct 30 13:08:59.780437 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:08:59.780089 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd8f64d33379f8b1abbae548dd9f8feae5dafca5391d57304a839646948eead9" Oct 30 13:09:00.525393 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:00.525354 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:02.428040 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:02.428003 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:02.917518 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:02.917488 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8651-afdbd5"] Oct 30 13:09:02.918140 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:02.918126 5922 cpu_manager.go:414] "containerMap: removing container" podUID="4931dc41-6e44-4365-ad2d-f0c6ae9a44d5" containerName="revoke-keys" Oct 30 13:09:02.918189 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:02.918142 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4931dc41-6e44-4365-ad2d-f0c6ae9a44d5" containerName="revoke-keys" Oct 30 13:09:02.918265 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:02.918256 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="4931dc41-6e44-4365-ad2d-f0c6ae9a44d5" containerName="revoke-keys" Oct 30 13:09:02.923233 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:02.923218 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8651-afdbd5" Oct 30 13:09:02.938530 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:02.938503 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8651-afdbd5"] Oct 30 13:09:02.982498 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:02.982474 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4gdbp\"" Oct 30 13:09:03.031212 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:03.031187 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxgcn\" (UniqueName: \"kubernetes.io/projected/77e5eb80-34e5-4748-ab2c-e6fa8393bea3-kube-api-access-vxgcn\") pod \"test-curl-8651-afdbd5\" (UID: \"77e5eb80-34e5-4748-ab2c-e6fa8393bea3\") " pod="opendatahub/test-curl-8651-afdbd5" Oct 30 13:09:03.132962 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:03.132932 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxgcn\" (UniqueName: \"kubernetes.io/projected/77e5eb80-34e5-4748-ab2c-e6fa8393bea3-kube-api-access-vxgcn\") pod \"test-curl-8651-afdbd5\" (UID: \"77e5eb80-34e5-4748-ab2c-e6fa8393bea3\") " pod="opendatahub/test-curl-8651-afdbd5" Oct 30 13:09:03.157111 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:03.157087 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxgcn\" (UniqueName: \"kubernetes.io/projected/77e5eb80-34e5-4748-ab2c-e6fa8393bea3-kube-api-access-vxgcn\") pod \"test-curl-8651-afdbd5\" (UID: \"77e5eb80-34e5-4748-ab2c-e6fa8393bea3\") " pod="opendatahub/test-curl-8651-afdbd5" Oct 30 13:09:03.238125 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:03.237897 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8651-afdbd5" Oct 30 13:09:03.341258 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:03.340451 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:09:03.556224 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:09:03.556179 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77e5eb80_34e5_4748_ab2c_e6fa8393bea3.slice/crio-f481f7463963e92244e92c4f21fcc5daeaf9fb3b403da1f2a6511456d0480f9e WatchSource:0}: Error finding container f481f7463963e92244e92c4f21fcc5daeaf9fb3b403da1f2a6511456d0480f9e: Status 404 returned error can't find the container with id f481f7463963e92244e92c4f21fcc5daeaf9fb3b403da1f2a6511456d0480f9e Oct 30 13:09:03.558518 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:03.558501 5922 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:09:03.558594 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:03.558524 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8651-afdbd5"] Oct 30 13:09:03.814969 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:03.814871 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8651-afdbd5" event={"ID":"77e5eb80-34e5-4748-ab2c-e6fa8393bea3","Type":"ContainerStarted","Data":"f481f7463963e92244e92c4f21fcc5daeaf9fb3b403da1f2a6511456d0480f9e"} Oct 30 13:09:04.202955 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:04.202913 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:04.848385 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:04.848162 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8651-afdbd5" event={"ID":"77e5eb80-34e5-4748-ab2c-e6fa8393bea3","Type":"ContainerStarted","Data":"4a11a6c02cfa55cf421eaee6e3864e0370a12785e6d18012a4a76e55c1e138aa"} Oct 30 13:09:04.904858 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:04.904792 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8651-afdbd5" podStartSLOduration=2.197828201 podStartE2EDuration="2.904773135s" totalImagesPullingTime="706.944934ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:04.89980014 +0000 UTC m=+1001.369215916" watchObservedRunningTime="2026-10-30 13:09:04.904773135 +0000 UTC m=+1001.374188885" Oct 30 13:09:05.534635 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:05.522825 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q"] Oct 30 13:09:05.837693 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:05.818731 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" Oct 30 13:09:05.943641 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:05.903594 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-workload-socket\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q\" (UID: \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" Oct 30 13:09:05.943641 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:05.942859 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-credential-socket\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q\" (UID: \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" Oct 30 13:09:05.943641 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:05.942941 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-workload-certs\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q\" (UID: \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" Oct 30 13:09:05.943641 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:05.943122 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbk57\" (UniqueName: \"kubernetes.io/projected/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-kube-api-access-sbk57\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q\" (UID: \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" Oct 30 13:09:05.943641 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:05.943160 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-istio-data\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q\" (UID: \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" Oct 30 13:09:05.943641 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:05.943214 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-istiod-ca-cert\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q\" (UID: \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" Oct 30 13:09:05.943641 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:05.943408 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-istio-token\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q\" (UID: \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" Oct 30 13:09:05.943641 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:05.943452 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-istio-envoy\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q\" (UID: \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" Oct 30 13:09:05.943641 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:05.943509 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-istio-podinfo\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q\" (UID: \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" Oct 30 13:09:06.136540 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.135107 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-istio-podinfo\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q\" (UID: \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" Oct 30 13:09:06.136540 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.135291 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-workload-socket\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q\" (UID: \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" Oct 30 13:09:06.136540 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.135473 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-credential-socket\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q\" (UID: \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" Oct 30 13:09:06.136540 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.135548 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-workload-certs\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q\" (UID: \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" Oct 30 13:09:06.136540 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.135599 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbk57\" (UniqueName: \"kubernetes.io/projected/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-kube-api-access-sbk57\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q\" (UID: \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" Oct 30 13:09:06.136540 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.135658 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-istio-data\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q\" (UID: \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" Oct 30 13:09:06.136540 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.135711 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-istiod-ca-cert\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q\" (UID: \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" Oct 30 13:09:06.136540 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.135758 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-istio-token\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q\" (UID: \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" Oct 30 13:09:06.136540 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.135800 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-istio-envoy\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q\" (UID: \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" Oct 30 13:09:06.144101 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.144064 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-workload-certs\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q\" (UID: \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" Oct 30 13:09:06.194637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.148775 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-istio-data\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q\" (UID: \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" Oct 30 13:09:06.194637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.149473 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-istiod-ca-cert\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q\" (UID: \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" Oct 30 13:09:06.194637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.150049 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-workload-socket\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q\" (UID: \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" Oct 30 13:09:06.194637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.189732 5922 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" podUID="7751779d-b082-409a-a20f-ff0ff6263808" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded" Oct 30 13:09:06.194637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.190142 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-credential-socket\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q\" (UID: \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" Oct 30 13:09:06.283403 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.283343 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-istio-envoy\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q\" (UID: \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" Oct 30 13:09:06.390891 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.230912 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" podUID="35619ab9-cbeb-45da-8506-3af63a273177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:06.392957 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.392924 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-istio-podinfo\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q\" (UID: \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" Oct 30 13:09:06.411733 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.393662 5922 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" podUID="47517373-759d-4eeb-ab5e-6150b7e82bb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:06.411991 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.393881 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbk57\" (UniqueName: \"kubernetes.io/projected/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-kube-api-access-sbk57\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q\" (UID: \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" Oct 30 13:09:06.412118 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.411422 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q"] Oct 30 13:09:06.412319 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.412303 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4"] Oct 30 13:09:06.457757 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.454758 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" podUID="14ae0aea-473f-4ca0-bcc0-f617b9839c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:06.486638 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.272633 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" podUID="dfd83cc8-6503-45a8-8260-c0edb8752471" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:06.486638 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.481310 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" podUID="dad5bdad-cd6f-41f1-a124-540c3524ead7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:06.486638 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.483267 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" podUID="a85de36b-f8c8-4ade-a468-c114080cb312" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:06.490725 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.489660 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" podUID="c0ea7fde-2ac5-432d-bb9b-534024186ac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:06.495960 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.495930 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-istio-token\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q\" (UID: \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" Oct 30 13:09:06.498830 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.498798 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4"] Oct 30 13:09:06.498955 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.498920 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" Oct 30 13:09:06.499731 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.499708 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk"] Oct 30 13:09:06.547873 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.547842 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" Oct 30 13:09:06.578637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.554825 5922 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" podUID="14ae0aea-473f-4ca0-bcc0-f617b9839c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:06.578637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.576048 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk"] Oct 30 13:09:06.584870 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.584840 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" Oct 30 13:09:06.672690 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.669276 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03433bd3-844d-42ca-b3b2-158f1a1f678d-istiod-ca-cert\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk\" (UID: \"03433bd3-844d-42ca-b3b2-158f1a1f678d\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" Oct 30 13:09:06.672690 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.669337 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03433bd3-844d-42ca-b3b2-158f1a1f678d-istio-data\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk\" (UID: \"03433bd3-844d-42ca-b3b2-158f1a1f678d\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" Oct 30 13:09:06.672690 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.669383 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e546672-c25a-4243-a2aa-b2f23dbcae95-workload-socket\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4\" (UID: \"4e546672-c25a-4243-a2aa-b2f23dbcae95\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" Oct 30 13:09:06.672690 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.669407 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e546672-c25a-4243-a2aa-b2f23dbcae95-istio-token\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4\" (UID: \"4e546672-c25a-4243-a2aa-b2f23dbcae95\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" Oct 30 13:09:06.672690 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.669435 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft2mc\" (UniqueName: \"kubernetes.io/projected/03433bd3-844d-42ca-b3b2-158f1a1f678d-kube-api-access-ft2mc\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk\" (UID: \"03433bd3-844d-42ca-b3b2-158f1a1f678d\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" Oct 30 13:09:06.672690 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.669481 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03433bd3-844d-42ca-b3b2-158f1a1f678d-istio-podinfo\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk\" (UID: \"03433bd3-844d-42ca-b3b2-158f1a1f678d\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" Oct 30 13:09:06.672690 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.669520 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7r9q\" (UniqueName: \"kubernetes.io/projected/4e546672-c25a-4243-a2aa-b2f23dbcae95-kube-api-access-l7r9q\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4\" (UID: \"4e546672-c25a-4243-a2aa-b2f23dbcae95\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" Oct 30 13:09:06.672690 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.669596 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03433bd3-844d-42ca-b3b2-158f1a1f678d-workload-certs\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk\" (UID: \"03433bd3-844d-42ca-b3b2-158f1a1f678d\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" Oct 30 13:09:06.672690 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.669655 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03433bd3-844d-42ca-b3b2-158f1a1f678d-istio-token\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk\" (UID: \"03433bd3-844d-42ca-b3b2-158f1a1f678d\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" Oct 30 13:09:06.672690 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.669678 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e546672-c25a-4243-a2aa-b2f23dbcae95-istio-podinfo\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4\" (UID: \"4e546672-c25a-4243-a2aa-b2f23dbcae95\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" Oct 30 13:09:06.672690 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.669718 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03433bd3-844d-42ca-b3b2-158f1a1f678d-credential-socket\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk\" (UID: \"03433bd3-844d-42ca-b3b2-158f1a1f678d\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" Oct 30 13:09:06.672690 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.669762 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e546672-c25a-4243-a2aa-b2f23dbcae95-istio-envoy\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4\" (UID: \"4e546672-c25a-4243-a2aa-b2f23dbcae95\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" Oct 30 13:09:06.672690 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.669800 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e546672-c25a-4243-a2aa-b2f23dbcae95-credential-socket\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4\" (UID: \"4e546672-c25a-4243-a2aa-b2f23dbcae95\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" Oct 30 13:09:06.672690 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.669835 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03433bd3-844d-42ca-b3b2-158f1a1f678d-istio-envoy\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk\" (UID: \"03433bd3-844d-42ca-b3b2-158f1a1f678d\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" Oct 30 13:09:06.672690 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.669874 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e546672-c25a-4243-a2aa-b2f23dbcae95-istio-data\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4\" (UID: \"4e546672-c25a-4243-a2aa-b2f23dbcae95\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" Oct 30 13:09:06.672690 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.669973 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e546672-c25a-4243-a2aa-b2f23dbcae95-workload-certs\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4\" (UID: \"4e546672-c25a-4243-a2aa-b2f23dbcae95\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" Oct 30 13:09:06.672690 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.670015 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e546672-c25a-4243-a2aa-b2f23dbcae95-istiod-ca-cert\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4\" (UID: \"4e546672-c25a-4243-a2aa-b2f23dbcae95\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" Oct 30 13:09:06.672690 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.670043 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03433bd3-844d-42ca-b3b2-158f1a1f678d-workload-socket\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk\" (UID: \"03433bd3-844d-42ca-b3b2-158f1a1f678d\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" Oct 30 13:09:06.693434 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.693394 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" podUID="5f353928-8cb4-49a3-b999-1b90e6cb31bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 13:09:06.695227 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.695175 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" podUID="47517373-759d-4eeb-ab5e-6150b7e82bb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:06.787651 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.772551 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e546672-c25a-4243-a2aa-b2f23dbcae95-istio-envoy\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4\" (UID: \"4e546672-c25a-4243-a2aa-b2f23dbcae95\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" Oct 30 13:09:06.787651 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.772637 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e546672-c25a-4243-a2aa-b2f23dbcae95-credential-socket\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4\" (UID: \"4e546672-c25a-4243-a2aa-b2f23dbcae95\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" Oct 30 13:09:06.787651 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.772678 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03433bd3-844d-42ca-b3b2-158f1a1f678d-istio-envoy\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk\" (UID: \"03433bd3-844d-42ca-b3b2-158f1a1f678d\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" Oct 30 13:09:06.787651 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.772716 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e546672-c25a-4243-a2aa-b2f23dbcae95-istio-data\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4\" (UID: \"4e546672-c25a-4243-a2aa-b2f23dbcae95\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" Oct 30 13:09:06.787651 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.772805 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e546672-c25a-4243-a2aa-b2f23dbcae95-workload-certs\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4\" (UID: \"4e546672-c25a-4243-a2aa-b2f23dbcae95\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" Oct 30 13:09:06.787651 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.772848 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e546672-c25a-4243-a2aa-b2f23dbcae95-istiod-ca-cert\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4\" (UID: \"4e546672-c25a-4243-a2aa-b2f23dbcae95\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" Oct 30 13:09:06.787651 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.772877 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03433bd3-844d-42ca-b3b2-158f1a1f678d-workload-socket\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk\" (UID: \"03433bd3-844d-42ca-b3b2-158f1a1f678d\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" Oct 30 13:09:06.787651 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.772929 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03433bd3-844d-42ca-b3b2-158f1a1f678d-istiod-ca-cert\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk\" (UID: \"03433bd3-844d-42ca-b3b2-158f1a1f678d\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" Oct 30 13:09:06.787651 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.772969 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03433bd3-844d-42ca-b3b2-158f1a1f678d-istio-data\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk\" (UID: \"03433bd3-844d-42ca-b3b2-158f1a1f678d\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" Oct 30 13:09:06.787651 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.773020 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e546672-c25a-4243-a2aa-b2f23dbcae95-workload-socket\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4\" (UID: \"4e546672-c25a-4243-a2aa-b2f23dbcae95\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" Oct 30 13:09:06.787651 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.773044 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e546672-c25a-4243-a2aa-b2f23dbcae95-istio-token\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4\" (UID: \"4e546672-c25a-4243-a2aa-b2f23dbcae95\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" Oct 30 13:09:06.787651 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.773070 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft2mc\" (UniqueName: \"kubernetes.io/projected/03433bd3-844d-42ca-b3b2-158f1a1f678d-kube-api-access-ft2mc\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk\" (UID: \"03433bd3-844d-42ca-b3b2-158f1a1f678d\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" Oct 30 13:09:06.787651 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.773126 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03433bd3-844d-42ca-b3b2-158f1a1f678d-istio-podinfo\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk\" (UID: \"03433bd3-844d-42ca-b3b2-158f1a1f678d\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" Oct 30 13:09:06.787651 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.773179 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7r9q\" (UniqueName: \"kubernetes.io/projected/4e546672-c25a-4243-a2aa-b2f23dbcae95-kube-api-access-l7r9q\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4\" (UID: \"4e546672-c25a-4243-a2aa-b2f23dbcae95\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" Oct 30 13:09:06.787651 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.773244 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03433bd3-844d-42ca-b3b2-158f1a1f678d-workload-certs\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk\" (UID: \"03433bd3-844d-42ca-b3b2-158f1a1f678d\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" Oct 30 13:09:06.787651 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.773286 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03433bd3-844d-42ca-b3b2-158f1a1f678d-istio-token\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk\" (UID: \"03433bd3-844d-42ca-b3b2-158f1a1f678d\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" Oct 30 13:09:06.787651 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.773308 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e546672-c25a-4243-a2aa-b2f23dbcae95-istio-podinfo\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4\" (UID: \"4e546672-c25a-4243-a2aa-b2f23dbcae95\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" Oct 30 13:09:06.787651 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.773355 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03433bd3-844d-42ca-b3b2-158f1a1f678d-credential-socket\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk\" (UID: \"03433bd3-844d-42ca-b3b2-158f1a1f678d\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" Oct 30 13:09:06.787651 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.774337 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e546672-c25a-4243-a2aa-b2f23dbcae95-workload-socket\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4\" (UID: \"4e546672-c25a-4243-a2aa-b2f23dbcae95\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" Oct 30 13:09:06.787651 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.777517 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e546672-c25a-4243-a2aa-b2f23dbcae95-istiod-ca-cert\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4\" (UID: \"4e546672-c25a-4243-a2aa-b2f23dbcae95\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" Oct 30 13:09:06.787651 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.778943 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03433bd3-844d-42ca-b3b2-158f1a1f678d-istio-data\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk\" (UID: \"03433bd3-844d-42ca-b3b2-158f1a1f678d\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" Oct 30 13:09:06.787651 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.779896 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03433bd3-844d-42ca-b3b2-158f1a1f678d-istiod-ca-cert\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk\" (UID: \"03433bd3-844d-42ca-b3b2-158f1a1f678d\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" Oct 30 13:09:06.797378 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.795724 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" podUID="b487a069-f519-45a3-b1d8-f32f2431797b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 13:09:06.797378 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.796755 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e546672-c25a-4243-a2aa-b2f23dbcae95-istio-data\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4\" (UID: \"4e546672-c25a-4243-a2aa-b2f23dbcae95\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" Oct 30 13:09:06.797378 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.797311 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e546672-c25a-4243-a2aa-b2f23dbcae95-credential-socket\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4\" (UID: \"4e546672-c25a-4243-a2aa-b2f23dbcae95\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" Oct 30 13:09:06.797834 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.797791 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e546672-c25a-4243-a2aa-b2f23dbcae95-workload-certs\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4\" (UID: \"4e546672-c25a-4243-a2aa-b2f23dbcae95\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" Oct 30 13:09:06.816235 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.815438 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" podUID="a296df26-3a3b-40ef-a3ab-1866ecf7dcbc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.110:8000/ready\": context deadline exceeded" Oct 30 13:09:06.830662 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.821807 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03433bd3-844d-42ca-b3b2-158f1a1f678d-credential-socket\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk\" (UID: \"03433bd3-844d-42ca-b3b2-158f1a1f678d\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" Oct 30 13:09:06.830662 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.821884 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" podUID="7751779d-b082-409a-a20f-ff0ff6263808" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 13:09:06.830662 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.822405 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03433bd3-844d-42ca-b3b2-158f1a1f678d-workload-certs\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk\" (UID: \"03433bd3-844d-42ca-b3b2-158f1a1f678d\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" Oct 30 13:09:06.845715 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.822604 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03433bd3-844d-42ca-b3b2-158f1a1f678d-workload-socket\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk\" (UID: \"03433bd3-844d-42ca-b3b2-158f1a1f678d\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" Oct 30 13:09:06.922858 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.899882 5922 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" podUID="c0ea7fde-2ac5-432d-bb9b-534024186ac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:06.933636 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.923692 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft2mc\" (UniqueName: \"kubernetes.io/projected/03433bd3-844d-42ca-b3b2-158f1a1f678d-kube-api-access-ft2mc\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk\" (UID: \"03433bd3-844d-42ca-b3b2-158f1a1f678d\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" Oct 30 13:09:06.933636 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.924411 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03433bd3-844d-42ca-b3b2-158f1a1f678d-istio-token\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk\" (UID: \"03433bd3-844d-42ca-b3b2-158f1a1f678d\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" Oct 30 13:09:06.933636 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.925396 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03433bd3-844d-42ca-b3b2-158f1a1f678d-istio-podinfo\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk\" (UID: \"03433bd3-844d-42ca-b3b2-158f1a1f678d\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" Oct 30 13:09:06.976638 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.955956 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03433bd3-844d-42ca-b3b2-158f1a1f678d-istio-envoy\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk\" (UID: \"03433bd3-844d-42ca-b3b2-158f1a1f678d\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" Oct 30 13:09:06.976638 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.956571 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7r9q\" (UniqueName: \"kubernetes.io/projected/4e546672-c25a-4243-a2aa-b2f23dbcae95-kube-api-access-l7r9q\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4\" (UID: \"4e546672-c25a-4243-a2aa-b2f23dbcae95\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" Oct 30 13:09:06.976638 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.956844 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" Oct 30 13:09:06.976638 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.957578 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e546672-c25a-4243-a2aa-b2f23dbcae95-istio-podinfo\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4\" (UID: \"4e546672-c25a-4243-a2aa-b2f23dbcae95\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" Oct 30 13:09:06.976638 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.957907 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e546672-c25a-4243-a2aa-b2f23dbcae95-istio-envoy\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4\" (UID: \"4e546672-c25a-4243-a2aa-b2f23dbcae95\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" Oct 30 13:09:06.990651 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.984104 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e546672-c25a-4243-a2aa-b2f23dbcae95-istio-token\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4\" (UID: \"4e546672-c25a-4243-a2aa-b2f23dbcae95\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" Oct 30 13:09:06.999417 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:06.998708 5922 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" podUID="a296df26-3a3b-40ef-a3ab-1866ecf7dcbc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.110:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:07.194169 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:07.186706 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" Oct 30 13:09:07.402772 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:07.402725 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" podUID="35619ab9-cbeb-45da-8506-3af63a273177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:07.403739 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:07.403713 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" Oct 30 13:09:07.460334 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:07.458559 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" podUID="14ae0aea-473f-4ca0-bcc0-f617b9839c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:07.460334 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:07.459294 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" Oct 30 13:09:07.496092 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:07.492778 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" podUID="a85de36b-f8c8-4ade-a468-c114080cb312" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:07.496092 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:07.493448 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" Oct 30 13:09:07.496092 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:07.494263 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" podUID="c0ea7fde-2ac5-432d-bb9b-534024186ac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:07.496092 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:07.494566 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" Oct 30 13:09:07.496092 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:07.495342 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" podUID="dad5bdad-cd6f-41f1-a124-540c3524ead7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:07.496092 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:07.495698 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" Oct 30 13:09:07.504936 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:07.504897 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" podUID="dfd83cc8-6503-45a8-8260-c0edb8752471" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 13:09:07.505655 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:07.505343 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" Oct 30 13:09:07.702655 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:07.702153 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" podUID="47517373-759d-4eeb-ab5e-6150b7e82bb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:07.702655 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:07.702555 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" podUID="5f353928-8cb4-49a3-b999-1b90e6cb31bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 13:09:07.763223 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:07.763134 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" Oct 30 13:09:07.763504 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:07.763484 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" Oct 30 13:09:07.803772 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:07.803726 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" podUID="b487a069-f519-45a3-b1d8-f32f2431797b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 13:09:07.804436 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:07.804411 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" Oct 30 13:09:07.956760 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:07.868596 5922 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" podUID="dad5bdad-cd6f-41f1-a124-540c3524ead7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": context deadline exceeded" Oct 30 13:09:07.956760 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:07.869343 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" podUID="7751779d-b082-409a-a20f-ff0ff6263808" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 13:09:07.956760 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:07.870820 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" podUID="a296df26-3a3b-40ef-a3ab-1866ecf7dcbc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.110:8000/ready\": context deadline exceeded" Oct 30 13:09:07.956760 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:07.872680 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" Oct 30 13:09:07.956760 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:07.872739 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" Oct 30 13:09:08.439729 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:08.425331 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" podUID="35619ab9-cbeb-45da-8506-3af63a273177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:08.490763 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:08.485704 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" podUID="14ae0aea-473f-4ca0-bcc0-f617b9839c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:08.580710 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:08.495690 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" podUID="dad5bdad-cd6f-41f1-a124-540c3524ead7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:08.581911 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:08.495744 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" podUID="c0ea7fde-2ac5-432d-bb9b-534024186ac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:08.582343 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:08.495766 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" podUID="a85de36b-f8c8-4ade-a468-c114080cb312" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 13:09:08.582754 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:08.507674 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" podUID="dfd83cc8-6503-45a8-8260-c0edb8752471" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:08.702460 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:08.702421 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" podUID="47517373-759d-4eeb-ab5e-6150b7e82bb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:08.703465 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:08.703433 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" podUID="5f353928-8cb4-49a3-b999-1b90e6cb31bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:09.172745 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:09.167915 5922 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" podUID="5f353928-8cb4-49a3-b999-1b90e6cb31bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded" Oct 30 13:09:09.172745 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:09.171810 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" podUID="7751779d-b082-409a-a20f-ff0ff6263808" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:09.172745 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:09.172135 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" podUID="a296df26-3a3b-40ef-a3ab-1866ecf7dcbc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.110:8000/ready\": context deadline exceeded" Oct 30 13:09:09.236965 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:09.236877 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" Oct 30 13:09:09.271065 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:09.269599 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" Oct 30 13:09:09.279758 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:09.271420 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" Oct 30 13:09:09.294600 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:09.293895 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" Oct 30 13:09:09.297818 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:09.296247 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" Oct 30 13:09:09.299766 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:09.298441 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" Oct 30 13:09:09.342017 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:09.341928 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" Oct 30 13:09:09.342818 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:09.342801 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" Oct 30 13:09:09.388180 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:09.385788 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:09.394999 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:09.394170 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" Oct 30 13:09:09.394999 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:09.394780 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:09:09.394999 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:09.394936 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:09:09.395267 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:09.395048 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:09:09.405167 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:09.403653 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" podUID="b487a069-f519-45a3-b1d8-f32f2431797b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:09.424659 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:09.423163 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" Oct 30 13:09:09.440255 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:09.440231 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" Oct 30 13:09:09.457568 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:09.457435 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q"] Oct 30 13:09:09.462936 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:09:09.462914 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03433bd3_844d_42ca_b3b2_158f1a1f678d.slice/crio-6ff39cbce3d44ddb3bbe5ca17d0b57a4736063e85ea674cda16a958bea4beb41 WatchSource:0}: Error finding container 6ff39cbce3d44ddb3bbe5ca17d0b57a4736063e85ea674cda16a958bea4beb41: Status 404 returned error can't find the container with id 6ff39cbce3d44ddb3bbe5ca17d0b57a4736063e85ea674cda16a958bea4beb41 Oct 30 13:09:09.471409 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:09.471387 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8651-afdbd5"] Oct 30 13:09:09.471734 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:09.471705 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8651-afdbd5" podUID="77e5eb80-34e5-4748-ab2c-e6fa8393bea3" containerName="test-curl-8651-afdbd5" containerID="cri-o://4a11a6c02cfa55cf421eaee6e3864e0370a12785e6d18012a4a76e55c1e138aa" gracePeriod=2 Oct 30 13:09:09.474646 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:09.474580 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:09:09.474842 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:09.474815 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:09:09.474993 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:09.474973 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:09:09.478110 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:09.478085 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8651-afdbd5"] Oct 30 13:09:09.493114 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:09.493064 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk"] Oct 30 13:09:09.543772 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:09.543745 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4"] Oct 30 13:09:09.548156 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:09.548123 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:09:09.548443 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:09.548416 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:09:09.548650 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:09.548626 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:09:10.229491 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:10.229459 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:10.264215 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:10.264185 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" event={"ID":"03433bd3-844d-42ca-b3b2-158f1a1f678d","Type":"ContainerStarted","Data":"2ed3debba84d1f6b67506d240bcdc230131bf906720e8debef4e93c8f2112afe"} Oct 30 13:09:10.264380 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:10.264253 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" event={"ID":"03433bd3-844d-42ca-b3b2-158f1a1f678d","Type":"ContainerStarted","Data":"6ff39cbce3d44ddb3bbe5ca17d0b57a4736063e85ea674cda16a958bea4beb41"} Oct 30 13:09:10.266061 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:10.266034 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" event={"ID":"983e3a6f-7d5b-49d1-ab1f-560b1f59c935","Type":"ContainerStarted","Data":"7f729fa6f134d11c0dacf3d27d042882c0fd0df9dbfd29853a12b457ecd06dff"} Oct 30 13:09:10.266189 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:10.266085 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" event={"ID":"983e3a6f-7d5b-49d1-ab1f-560b1f59c935","Type":"ContainerStarted","Data":"91b29d19a1ceeae8d187a7d2570adeb8bcfbe7ef47b5770d917b766b95afaff8"} Oct 30 13:09:10.267798 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:10.267779 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" event={"ID":"4e546672-c25a-4243-a2aa-b2f23dbcae95","Type":"ContainerStarted","Data":"8f8eada104aa41b86740bb6cd2e794ed937407fbb2817ee38261b9e543b43004"} Oct 30 13:09:10.267894 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:10.267820 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" event={"ID":"4e546672-c25a-4243-a2aa-b2f23dbcae95","Type":"ContainerStarted","Data":"e1b8217f5115a8740b970ea6adf37dcaadc010041cb2e652bb863f3256364da1"} Oct 30 13:09:10.311987 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:10.311919 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" podStartSLOduration=5.3118986679999995 podStartE2EDuration="5.311898668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:10.310147958 +0000 UTC m=+1006.779563768" watchObservedRunningTime="2026-10-30 13:09:10.311898668 +0000 UTC m=+1006.781314426" Oct 30 13:09:10.401631 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:10.400715 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" podStartSLOduration=5.382605929 podStartE2EDuration="5.382605929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:10.358230579 +0000 UTC m=+1006.827646424" watchObservedRunningTime="2026-10-30 13:09:10.382605929 +0000 UTC m=+1006.852021674" Oct 30 13:09:10.449390 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:10.449318 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" podStartSLOduration=5.449298551 podStartE2EDuration="5.449298551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:10.412135602 +0000 UTC m=+1006.881551367" watchObservedRunningTime="2026-10-30 13:09:10.449298551 +0000 UTC m=+1006.918714297" Oct 30 13:09:10.595783 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:10.595676 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" Oct 30 13:09:10.697728 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:10.662876 5922 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 13:09:10.697728 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:10.662946 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" podUID="983e3a6f-7d5b-49d1-ab1f-560b1f59c935" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 13:09:10.962634 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:10.959756 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" Oct 30 13:09:10.974853 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:10.974814 5922 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 13:09:10.975153 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:10.975105 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" podUID="03433bd3-844d-42ca-b3b2-158f1a1f678d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 13:09:11.189637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:11.188879 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" Oct 30 13:09:11.189637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:11.189055 5922 patch_prober.go:29] interesting pod/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 13:09:11.189637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:11.189103 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" podUID="4e546672-c25a-4243-a2aa-b2f23dbcae95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 13:09:11.233694 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:11.223421 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:11.597639 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:11.597329 5922 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 13:09:11.597639 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:11.597555 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" podUID="983e3a6f-7d5b-49d1-ab1f-560b1f59c935" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 13:09:11.686769 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:11.683390 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:11.962632 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:11.958705 5922 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 13:09:11.962632 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:11.958756 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" podUID="03433bd3-844d-42ca-b3b2-158f1a1f678d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 13:09:12.091638 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:12.086056 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:12.192750 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:12.192714 5922 patch_prober.go:29] interesting pod/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 13:09:12.193054 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:12.193020 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" podUID="4e546672-c25a-4243-a2aa-b2f23dbcae95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 13:09:12.296652 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:12.292958 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:12.551921 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:12.550546 5922 generic.go:356] "Generic (PLEG): container finished" podID="77e5eb80-34e5-4748-ab2c-e6fa8393bea3" containerID="4a11a6c02cfa55cf421eaee6e3864e0370a12785e6d18012a4a76e55c1e138aa" exitCode=137 Oct 30 13:09:12.597630 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:12.596738 5922 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 13:09:12.597630 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:12.596808 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" podUID="983e3a6f-7d5b-49d1-ab1f-560b1f59c935" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 13:09:12.625634 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:12.625459 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8651-afdbd5" Oct 30 13:09:12.660561 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:12.660536 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77e5eb80-34e5-4748-ab2c-e6fa8393bea3-kube-api-access-vxgcn\" (UniqueName: \"kubernetes.io/projected/77e5eb80-34e5-4748-ab2c-e6fa8393bea3-kube-api-access-vxgcn\") pod \"77e5eb80-34e5-4748-ab2c-e6fa8393bea3\" (UID: \"77e5eb80-34e5-4748-ab2c-e6fa8393bea3\") " Oct 30 13:09:12.670787 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:12.670753 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77e5eb80-34e5-4748-ab2c-e6fa8393bea3-kube-api-access-vxgcn" pod "77e5eb80-34e5-4748-ab2c-e6fa8393bea3" (UID: "77e5eb80-34e5-4748-ab2c-e6fa8393bea3"). InnerVolumeSpecName "kube-api-access-vxgcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:09:12.767634 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:12.766693 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxgcn\" (UniqueName: \"kubernetes.io/projected/77e5eb80-34e5-4748-ab2c-e6fa8393bea3-kube-api-access-vxgcn\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:09:12.958288 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:12.958242 5922 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 13:09:12.958496 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:12.958315 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" podUID="03433bd3-844d-42ca-b3b2-158f1a1f678d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 13:09:12.980690 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:12.980654 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:13.151386 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:13.151045 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:13.188647 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:13.188158 5922 patch_prober.go:29] interesting pod/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 13:09:13.188647 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:13.188220 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" podUID="4e546672-c25a-4243-a2aa-b2f23dbcae95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 13:09:13.560262 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:13.560236 5922 scope.go:122] "RemoveContainer" containerID="4a11a6c02cfa55cf421eaee6e3864e0370a12785e6d18012a4a76e55c1e138aa" Oct 30 13:09:13.560262 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:13.560262 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8651-afdbd5" Oct 30 13:09:13.591999 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:13.591959 5922 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 13:09:13.592273 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:13.592241 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" podUID="983e3a6f-7d5b-49d1-ab1f-560b1f59c935" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 13:09:13.749382 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:13.749340 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77e5eb80-34e5-4748-ab2c-e6fa8393bea3" path="/var/lib/kubelet/pods/77e5eb80-34e5-4748-ab2c-e6fa8393bea3/volumes" Oct 30 13:09:13.957689 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:13.957656 5922 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 13:09:13.957859 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:13.957712 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" podUID="03433bd3-844d-42ca-b3b2-158f1a1f678d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 13:09:14.193636 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:14.193075 5922 patch_prober.go:29] interesting pod/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 13:09:14.193636 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:14.193156 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" podUID="4e546672-c25a-4243-a2aa-b2f23dbcae95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 13:09:14.534407 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:14.534366 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:14.623435 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:14.623400 5922 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 13:09:14.628813 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:14.628778 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" podUID="983e3a6f-7d5b-49d1-ab1f-560b1f59c935" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 13:09:14.958211 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:14.958165 5922 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 13:09:14.958831 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:14.958781 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" podUID="03433bd3-844d-42ca-b3b2-158f1a1f678d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 13:09:15.194115 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:15.194082 5922 patch_prober.go:29] interesting pod/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 13:09:15.194380 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:15.194351 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" podUID="4e546672-c25a-4243-a2aa-b2f23dbcae95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 13:09:15.595637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:15.594934 5922 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 13:09:15.595637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:15.594998 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" podUID="983e3a6f-7d5b-49d1-ab1f-560b1f59c935" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 13:09:15.957791 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:15.957757 5922 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 13:09:15.957931 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:15.957810 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" podUID="03433bd3-844d-42ca-b3b2-158f1a1f678d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 13:09:16.191273 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:16.191233 5922 patch_prober.go:29] interesting pod/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 13:09:16.191871 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:16.191839 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" podUID="4e546672-c25a-4243-a2aa-b2f23dbcae95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 13:09:16.484827 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:16.484794 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:16.587230 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:16.587190 5922 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 13:09:16.587401 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:16.587257 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" podUID="983e3a6f-7d5b-49d1-ab1f-560b1f59c935" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 13:09:16.610833 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:16.610795 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:16.898740 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:16.898657 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93b6dd98-6nxm4"] Oct 30 13:09:16.899981 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:16.899952 5922 cpu_manager.go:414] "containerMap: removing container" podUID="77e5eb80-34e5-4748-ab2c-e6fa8393bea3" containerName="test-curl-8651-afdbd5" Oct 30 13:09:16.900141 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:16.900124 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77e5eb80-34e5-4748-ab2c-e6fa8393bea3" containerName="test-curl-8651-afdbd5" Oct 30 13:09:16.900453 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:16.900433 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="77e5eb80-34e5-4748-ab2c-e6fa8393bea3" containerName="test-curl-8651-afdbd5" Oct 30 13:09:16.907553 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:16.907534 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b6dd98-6nxm4" Oct 30 13:09:16.923641 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:16.917369 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93b6dd98-6nxm4"] Oct 30 13:09:16.940312 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:16.940085 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:09:16.940312 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:16.940184 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:09:16.940312 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:16.940213 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:09:16.940312 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:16.940280 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:09:16.957145 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:16.957108 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbj9k\" (UniqueName: \"kubernetes.io/projected/47c3d395-2a73-4415-901c-a007b2d167ff-kube-api-access-zbj9k\") pod \"maas-api-revoke-sub-93b6dd98-6nxm4\" (UID: \"47c3d395-2a73-4415-901c-a007b2d167ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b6dd98-6nxm4" Oct 30 13:09:16.957308 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:16.957251 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47c3d395-2a73-4415-901c-a007b2d167ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93b6dd98-6nxm4\" (UID: \"47c3d395-2a73-4415-901c-a007b2d167ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b6dd98-6nxm4" Oct 30 13:09:16.957504 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:16.957480 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47c3d395-2a73-4415-901c-a007b2d167ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93b6dd98-6nxm4\" (UID: \"47c3d395-2a73-4415-901c-a007b2d167ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b6dd98-6nxm4" Oct 30 13:09:17.058452 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:17.058415 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47c3d395-2a73-4415-901c-a007b2d167ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93b6dd98-6nxm4\" (UID: \"47c3d395-2a73-4415-901c-a007b2d167ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b6dd98-6nxm4" Oct 30 13:09:17.058668 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:17.058527 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbj9k\" (UniqueName: \"kubernetes.io/projected/47c3d395-2a73-4415-901c-a007b2d167ff-kube-api-access-zbj9k\") pod \"maas-api-revoke-sub-93b6dd98-6nxm4\" (UID: \"47c3d395-2a73-4415-901c-a007b2d167ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b6dd98-6nxm4" Oct 30 13:09:17.058668 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:17.058582 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47c3d395-2a73-4415-901c-a007b2d167ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93b6dd98-6nxm4\" (UID: \"47c3d395-2a73-4415-901c-a007b2d167ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b6dd98-6nxm4" Oct 30 13:09:17.059273 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:17.059250 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47c3d395-2a73-4415-901c-a007b2d167ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93b6dd98-6nxm4\" (UID: \"47c3d395-2a73-4415-901c-a007b2d167ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b6dd98-6nxm4" Oct 30 13:09:17.061010 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:17.060983 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47c3d395-2a73-4415-901c-a007b2d167ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93b6dd98-6nxm4\" (UID: \"47c3d395-2a73-4415-901c-a007b2d167ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b6dd98-6nxm4" Oct 30 13:09:17.090549 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:17.090516 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbj9k\" (UniqueName: \"kubernetes.io/projected/47c3d395-2a73-4415-901c-a007b2d167ff-kube-api-access-zbj9k\") pod \"maas-api-revoke-sub-93b6dd98-6nxm4\" (UID: \"47c3d395-2a73-4415-901c-a007b2d167ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b6dd98-6nxm4" Oct 30 13:09:17.122594 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:17.122559 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:17.126681 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:17.126657 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" Oct 30 13:09:17.127220 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:17.127201 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" Oct 30 13:09:17.127951 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:17.127934 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" Oct 30 13:09:17.195056 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:17.195027 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" Oct 30 13:09:17.195868 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:17.195842 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" Oct 30 13:09:17.196539 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:17.196517 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" Oct 30 13:09:17.241629 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:17.241578 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b6dd98-6nxm4" Oct 30 13:09:17.647532 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:17.647506 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93b6dd98-6nxm4"] Oct 30 13:09:17.654138 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:17.654109 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" Oct 30 13:09:17.655175 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:17.655148 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" Oct 30 13:09:17.656076 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:17.656057 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" Oct 30 13:09:17.672044 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:17.672023 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b6dd98-6nxm4" event={"ID":"47c3d395-2a73-4415-901c-a007b2d167ff","Type":"ContainerStarted","Data":"78291806020fb496b6a7f4d5866ee4bb7c349c39bbf4159de1a9a7cc2872a1a3"} Oct 30 13:09:18.640049 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:18.639928 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:18.681516 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:18.681488 5922 generic.go:356] "Generic (PLEG): container finished" podID="47c3d395-2a73-4415-901c-a007b2d167ff" containerID="5b8f3b32480a71efef16c562ccabc8572deca9250a9d340aeb959b20b3412a87" exitCode=0 Oct 30 13:09:18.681683 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:18.681565 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b6dd98-6nxm4" event={"ID":"47c3d395-2a73-4415-901c-a007b2d167ff","Type":"ContainerDied","Data":"5b8f3b32480a71efef16c562ccabc8572deca9250a9d340aeb959b20b3412a87"} Oct 30 13:09:18.769420 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:18.769336 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:18.813887 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:18.813860 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-751d84fb-666b55f879-5g5l5"] Oct 30 13:09:18.818553 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:18.818536 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-751d84fb-666b55f879-5g5l5" Oct 30 13:09:18.836974 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:18.836953 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-751d84fb\"" Oct 30 13:09:18.837406 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:18.837389 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-751d84fb-dockercfg-fcbf5\"" Oct 30 13:09:18.838871 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:18.838849 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-751d84fb-666b55f879-5g5l5"] Oct 30 13:09:18.871385 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:18.871354 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-751d84fb-9696f945b-4spf8"] Oct 30 13:09:18.875823 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:18.875800 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-751d84fb-9696f945b-4spf8" Oct 30 13:09:18.888157 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:18.888132 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th2kk\" (UniqueName: \"kubernetes.io/projected/e5eecf33-3127-48f3-af3d-428dc1ca61da-kube-api-access-th2kk\") pod \"payload-processing-e2e-ait-751d84fb-9696f945b-4spf8\" (UID: \"e5eecf33-3127-48f3-af3d-428dc1ca61da\") " pod="openshift-ingress/payload-processing-e2e-ait-751d84fb-9696f945b-4spf8" Oct 30 13:09:18.888284 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:18.888267 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5eecf33-3127-48f3-af3d-428dc1ca61da-plugins-config-volume\") pod \"payload-processing-e2e-ait-751d84fb-9696f945b-4spf8\" (UID: \"e5eecf33-3127-48f3-af3d-428dc1ca61da\") " pod="openshift-ingress/payload-processing-e2e-ait-751d84fb-9696f945b-4spf8" Oct 30 13:09:18.888348 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:18.888330 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9fa2d25-84a2-4041-8cba-9988db6ef2ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-751d84fb-666b55f879-5g5l5\" (UID: \"e9fa2d25-84a2-4041-8cba-9988db6ef2ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-751d84fb-666b55f879-5g5l5" Oct 30 13:09:18.888408 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:18.888395 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8cck\" (UniqueName: \"kubernetes.io/projected/e9fa2d25-84a2-4041-8cba-9988db6ef2ed-kube-api-access-m8cck\") pod \"payload-pre-processing-e2e-ait-751d84fb-666b55f879-5g5l5\" (UID: \"e9fa2d25-84a2-4041-8cba-9988db6ef2ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-751d84fb-666b55f879-5g5l5" Oct 30 13:09:18.900477 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:18.900454 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-751d84fb-9696f945b-4spf8"] Oct 30 13:09:18.989424 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:18.989389 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th2kk\" (UniqueName: \"kubernetes.io/projected/e5eecf33-3127-48f3-af3d-428dc1ca61da-kube-api-access-th2kk\") pod \"payload-processing-e2e-ait-751d84fb-9696f945b-4spf8\" (UID: \"e5eecf33-3127-48f3-af3d-428dc1ca61da\") " pod="openshift-ingress/payload-processing-e2e-ait-751d84fb-9696f945b-4spf8" Oct 30 13:09:18.989593 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:18.989488 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5eecf33-3127-48f3-af3d-428dc1ca61da-plugins-config-volume\") pod \"payload-processing-e2e-ait-751d84fb-9696f945b-4spf8\" (UID: \"e5eecf33-3127-48f3-af3d-428dc1ca61da\") " pod="openshift-ingress/payload-processing-e2e-ait-751d84fb-9696f945b-4spf8" Oct 30 13:09:18.989593 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:18.989550 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9fa2d25-84a2-4041-8cba-9988db6ef2ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-751d84fb-666b55f879-5g5l5\" (UID: \"e9fa2d25-84a2-4041-8cba-9988db6ef2ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-751d84fb-666b55f879-5g5l5" Oct 30 13:09:18.989698 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:18.989624 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8cck\" (UniqueName: \"kubernetes.io/projected/e9fa2d25-84a2-4041-8cba-9988db6ef2ed-kube-api-access-m8cck\") pod \"payload-pre-processing-e2e-ait-751d84fb-666b55f879-5g5l5\" (UID: \"e9fa2d25-84a2-4041-8cba-9988db6ef2ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-751d84fb-666b55f879-5g5l5" Oct 30 13:09:18.990328 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:18.990293 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9fa2d25-84a2-4041-8cba-9988db6ef2ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-751d84fb-666b55f879-5g5l5\" (UID: \"e9fa2d25-84a2-4041-8cba-9988db6ef2ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-751d84fb-666b55f879-5g5l5" Oct 30 13:09:18.990426 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:18.990404 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5eecf33-3127-48f3-af3d-428dc1ca61da-plugins-config-volume\") pod \"payload-processing-e2e-ait-751d84fb-9696f945b-4spf8\" (UID: \"e5eecf33-3127-48f3-af3d-428dc1ca61da\") " pod="openshift-ingress/payload-processing-e2e-ait-751d84fb-9696f945b-4spf8" Oct 30 13:09:19.016787 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:19.016755 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th2kk\" (UniqueName: \"kubernetes.io/projected/e5eecf33-3127-48f3-af3d-428dc1ca61da-kube-api-access-th2kk\") pod \"payload-processing-e2e-ait-751d84fb-9696f945b-4spf8\" (UID: \"e5eecf33-3127-48f3-af3d-428dc1ca61da\") " pod="openshift-ingress/payload-processing-e2e-ait-751d84fb-9696f945b-4spf8" Oct 30 13:09:19.017235 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:19.017206 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8cck\" (UniqueName: \"kubernetes.io/projected/e9fa2d25-84a2-4041-8cba-9988db6ef2ed-kube-api-access-m8cck\") pod \"payload-pre-processing-e2e-ait-751d84fb-666b55f879-5g5l5\" (UID: \"e9fa2d25-84a2-4041-8cba-9988db6ef2ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-751d84fb-666b55f879-5g5l5" Oct 30 13:09:19.133064 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:19.132972 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-751d84fb-666b55f879-5g5l5" Oct 30 13:09:19.195066 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:19.195037 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-751d84fb-9696f945b-4spf8" Oct 30 13:09:19.466330 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:19.466294 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-751d84fb-666b55f879-5g5l5"] Oct 30 13:09:19.476825 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:09:19.476794 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9fa2d25_84a2_4041_8cba_9988db6ef2ed.slice/crio-a6d5f2ed651848f00e40526fc3904c34d8de490c68b29dc347a33998476dbd54 WatchSource:0}: Error finding container a6d5f2ed651848f00e40526fc3904c34d8de490c68b29dc347a33998476dbd54: Status 404 returned error can't find the container with id a6d5f2ed651848f00e40526fc3904c34d8de490c68b29dc347a33998476dbd54 Oct 30 13:09:19.538934 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:09:19.538906 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5eecf33_3127_48f3_af3d_428dc1ca61da.slice/crio-888e0b8ca0fff3ff31381c55516a61b62b68fa840d1f9c12d03331a75332bd84 WatchSource:0}: Error finding container 888e0b8ca0fff3ff31381c55516a61b62b68fa840d1f9c12d03331a75332bd84: Status 404 returned error can't find the container with id 888e0b8ca0fff3ff31381c55516a61b62b68fa840d1f9c12d03331a75332bd84 Oct 30 13:09:19.539944 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:19.539836 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-751d84fb-9696f945b-4spf8"] Oct 30 13:09:19.691037 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:19.691004 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-751d84fb-666b55f879-5g5l5" event={"ID":"e9fa2d25-84a2-4041-8cba-9988db6ef2ed","Type":"ContainerStarted","Data":"422da1f9b38ec1664f38469ce44b29f9c922ce79045d6134933b92f9bc1a3ade"} Oct 30 13:09:19.691435 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:19.691062 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-751d84fb-666b55f879-5g5l5" event={"ID":"e9fa2d25-84a2-4041-8cba-9988db6ef2ed","Type":"ContainerStarted","Data":"a6d5f2ed651848f00e40526fc3904c34d8de490c68b29dc347a33998476dbd54"} Oct 30 13:09:19.691435 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:19.691214 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-751d84fb-666b55f879-5g5l5" Oct 30 13:09:19.692841 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:19.692810 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-751d84fb-9696f945b-4spf8" event={"ID":"e5eecf33-3127-48f3-af3d-428dc1ca61da","Type":"ContainerStarted","Data":"139c6c57ea8e6bf1faccfc11d572e8d8bd46a11f865890c880a333fd4235f263"} Oct 30 13:09:19.692945 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:19.692854 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-751d84fb-9696f945b-4spf8" event={"ID":"e5eecf33-3127-48f3-af3d-428dc1ca61da","Type":"ContainerStarted","Data":"888e0b8ca0fff3ff31381c55516a61b62b68fa840d1f9c12d03331a75332bd84"} Oct 30 13:09:19.744461 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:19.744368 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-751d84fb-666b55f879-5g5l5" podStartSLOduration=1.7443532990000001 podStartE2EDuration="1.744353299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:19.739080353 +0000 UTC m=+1016.208496117" watchObservedRunningTime="2026-10-30 13:09:19.744353299 +0000 UTC m=+1016.213769044" Oct 30 13:09:19.813047 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:19.812983 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-751d84fb-9696f945b-4spf8" podStartSLOduration=1.812965451 podStartE2EDuration="1.812965451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:19.810671923 +0000 UTC m=+1016.280087697" watchObservedRunningTime="2026-10-30 13:09:19.812965451 +0000 UTC m=+1016.282381197" Oct 30 13:09:19.828354 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:19.828331 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b6dd98-6nxm4" Oct 30 13:09:19.906446 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:19.906417 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47c3d395-2a73-4415-901c-a007b2d167ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47c3d395-2a73-4415-901c-a007b2d167ff-maas-api-openshift-ca\") pod \"47c3d395-2a73-4415-901c-a007b2d167ff\" (UID: \"47c3d395-2a73-4415-901c-a007b2d167ff\") " Oct 30 13:09:19.906636 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:19.906508 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47c3d395-2a73-4415-901c-a007b2d167ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47c3d395-2a73-4415-901c-a007b2d167ff-maas-api-xks-ca\") pod \"47c3d395-2a73-4415-901c-a007b2d167ff\" (UID: \"47c3d395-2a73-4415-901c-a007b2d167ff\") " Oct 30 13:09:19.906708 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:19.906653 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47c3d395-2a73-4415-901c-a007b2d167ff-kube-api-access-zbj9k\" (UniqueName: \"kubernetes.io/projected/47c3d395-2a73-4415-901c-a007b2d167ff-kube-api-access-zbj9k\") pod \"47c3d395-2a73-4415-901c-a007b2d167ff\" (UID: \"47c3d395-2a73-4415-901c-a007b2d167ff\") " Oct 30 13:09:19.906893 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:19.906868 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47c3d395-2a73-4415-901c-a007b2d167ff-maas-api-openshift-ca" pod "47c3d395-2a73-4415-901c-a007b2d167ff" (UID: "47c3d395-2a73-4415-901c-a007b2d167ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:09:19.907356 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:19.907333 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47c3d395-2a73-4415-901c-a007b2d167ff-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:09:19.908849 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:19.908821 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47c3d395-2a73-4415-901c-a007b2d167ff-kube-api-access-zbj9k" pod "47c3d395-2a73-4415-901c-a007b2d167ff" (UID: "47c3d395-2a73-4415-901c-a007b2d167ff"). InnerVolumeSpecName "kube-api-access-zbj9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:09:19.909110 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:19.909089 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47c3d395-2a73-4415-901c-a007b2d167ff-maas-api-xks-ca" pod "47c3d395-2a73-4415-901c-a007b2d167ff" (UID: "47c3d395-2a73-4415-901c-a007b2d167ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:09:20.008416 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:20.008346 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47c3d395-2a73-4415-901c-a007b2d167ff-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:09:20.008416 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:20.008374 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbj9k\" (UniqueName: \"kubernetes.io/projected/47c3d395-2a73-4415-901c-a007b2d167ff-kube-api-access-zbj9k\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:09:20.594318 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:20.594283 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:20.706107 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:20.706071 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b6dd98-6nxm4" Oct 30 13:09:20.706528 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:20.706103 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b6dd98-6nxm4" event={"ID":"47c3d395-2a73-4415-901c-a007b2d167ff","Type":"ContainerDied","Data":"78291806020fb496b6a7f4d5866ee4bb7c349c39bbf4159de1a9a7cc2872a1a3"} Oct 30 13:09:20.706528 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:20.706177 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78291806020fb496b6a7f4d5866ee4bb7c349c39bbf4159de1a9a7cc2872a1a3" Oct 30 13:09:20.706528 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:20.706424 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-751d84fb-9696f945b-4spf8" Oct 30 13:09:21.316982 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:21.316942 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:21.441198 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:21.441154 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:21.648394 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:21.648317 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:22.062524 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:22.062491 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c"] Oct 30 13:09:22.063559 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:22.063535 5922 cpu_manager.go:414] "containerMap: removing container" podUID="47c3d395-2a73-4415-901c-a007b2d167ff" containerName="revoke-keys" Oct 30 13:09:22.063660 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:22.063561 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47c3d395-2a73-4415-901c-a007b2d167ff" containerName="revoke-keys" Oct 30 13:09:22.063820 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:22.063804 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="47c3d395-2a73-4415-901c-a007b2d167ff" containerName="revoke-keys" Oct 30 13:09:22.071736 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:22.071711 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" Oct 30 13:09:22.093594 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:22.093568 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c"] Oct 30 13:09:22.097989 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:22.097962 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-4a59cbc1-openshift-default-dockercfg-x7lnl\"" Oct 30 13:09:22.137751 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:22.137726 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:22.139002 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:22.138976 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99d3f2ce-87ad-4849-8976-7c388312e1e7-istiod-ca-cert\") pod \"e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c\" (UID: \"99d3f2ce-87ad-4849-8976-7c388312e1e7\") " pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" Oct 30 13:09:22.139101 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:22.139012 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52l7g\" (UniqueName: \"kubernetes.io/projected/99d3f2ce-87ad-4849-8976-7c388312e1e7-kube-api-access-52l7g\") pod \"e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c\" (UID: \"99d3f2ce-87ad-4849-8976-7c388312e1e7\") " pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" Oct 30 13:09:22.139101 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:22.139036 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99d3f2ce-87ad-4849-8976-7c388312e1e7-credential-socket\") pod \"e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c\" (UID: \"99d3f2ce-87ad-4849-8976-7c388312e1e7\") " pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" Oct 30 13:09:22.139191 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:22.139139 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99d3f2ce-87ad-4849-8976-7c388312e1e7-workload-socket\") pod \"e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c\" (UID: \"99d3f2ce-87ad-4849-8976-7c388312e1e7\") " pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" Oct 30 13:09:22.139191 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:22.139174 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99d3f2ce-87ad-4849-8976-7c388312e1e7-workload-certs\") pod \"e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c\" (UID: \"99d3f2ce-87ad-4849-8976-7c388312e1e7\") " pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" Oct 30 13:09:22.139302 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:22.139200 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99d3f2ce-87ad-4849-8976-7c388312e1e7-istio-data\") pod \"e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c\" (UID: \"99d3f2ce-87ad-4849-8976-7c388312e1e7\") " pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" Oct 30 13:09:22.139353 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:22.139324 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99d3f2ce-87ad-4849-8976-7c388312e1e7-istio-podinfo\") pod \"e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c\" (UID: \"99d3f2ce-87ad-4849-8976-7c388312e1e7\") " pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" Oct 30 13:09:22.139529 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:22.139511 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99d3f2ce-87ad-4849-8976-7c388312e1e7-istio-envoy\") pod \"e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c\" (UID: \"99d3f2ce-87ad-4849-8976-7c388312e1e7\") " pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" Oct 30 13:09:22.139594 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:22.139540 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99d3f2ce-87ad-4849-8976-7c388312e1e7-istio-token\") pod \"e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c\" (UID: \"99d3f2ce-87ad-4849-8976-7c388312e1e7\") " pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" Oct 30 13:09:22.241266 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:22.241235 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99d3f2ce-87ad-4849-8976-7c388312e1e7-workload-certs\") pod \"e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c\" (UID: \"99d3f2ce-87ad-4849-8976-7c388312e1e7\") " pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" Oct 30 13:09:22.241449 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:22.241291 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99d3f2ce-87ad-4849-8976-7c388312e1e7-istio-data\") pod \"e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c\" (UID: \"99d3f2ce-87ad-4849-8976-7c388312e1e7\") " pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" Oct 30 13:09:22.241449 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:22.241340 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99d3f2ce-87ad-4849-8976-7c388312e1e7-istio-podinfo\") pod \"e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c\" (UID: \"99d3f2ce-87ad-4849-8976-7c388312e1e7\") " pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" Oct 30 13:09:22.241577 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:22.241549 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99d3f2ce-87ad-4849-8976-7c388312e1e7-istio-envoy\") pod \"e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c\" (UID: \"99d3f2ce-87ad-4849-8976-7c388312e1e7\") " pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" Oct 30 13:09:22.241697 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:22.241580 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99d3f2ce-87ad-4849-8976-7c388312e1e7-istio-token\") pod \"e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c\" (UID: \"99d3f2ce-87ad-4849-8976-7c388312e1e7\") " pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" Oct 30 13:09:22.241758 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:22.241717 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99d3f2ce-87ad-4849-8976-7c388312e1e7-workload-certs\") pod \"e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c\" (UID: \"99d3f2ce-87ad-4849-8976-7c388312e1e7\") " pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" Oct 30 13:09:22.241872 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:22.241854 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99d3f2ce-87ad-4849-8976-7c388312e1e7-istiod-ca-cert\") pod \"e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c\" (UID: \"99d3f2ce-87ad-4849-8976-7c388312e1e7\") " pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" Oct 30 13:09:22.242037 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:22.242012 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99d3f2ce-87ad-4849-8976-7c388312e1e7-istio-data\") pod \"e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c\" (UID: \"99d3f2ce-87ad-4849-8976-7c388312e1e7\") " pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" Oct 30 13:09:22.242566 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:22.242539 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99d3f2ce-87ad-4849-8976-7c388312e1e7-istiod-ca-cert\") pod \"e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c\" (UID: \"99d3f2ce-87ad-4849-8976-7c388312e1e7\") " pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" Oct 30 13:09:22.242776 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:22.242592 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52l7g\" (UniqueName: \"kubernetes.io/projected/99d3f2ce-87ad-4849-8976-7c388312e1e7-kube-api-access-52l7g\") pod \"e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c\" (UID: \"99d3f2ce-87ad-4849-8976-7c388312e1e7\") " pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" Oct 30 13:09:22.242776 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:22.242648 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99d3f2ce-87ad-4849-8976-7c388312e1e7-credential-socket\") pod \"e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c\" (UID: \"99d3f2ce-87ad-4849-8976-7c388312e1e7\") " pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" Oct 30 13:09:22.242776 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:22.242705 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99d3f2ce-87ad-4849-8976-7c388312e1e7-workload-socket\") pod \"e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c\" (UID: \"99d3f2ce-87ad-4849-8976-7c388312e1e7\") " pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" Oct 30 13:09:22.242995 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:22.242972 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99d3f2ce-87ad-4849-8976-7c388312e1e7-credential-socket\") pod \"e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c\" (UID: \"99d3f2ce-87ad-4849-8976-7c388312e1e7\") " pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" Oct 30 13:09:22.243059 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:22.243008 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99d3f2ce-87ad-4849-8976-7c388312e1e7-workload-socket\") pod \"e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c\" (UID: \"99d3f2ce-87ad-4849-8976-7c388312e1e7\") " pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" Oct 30 13:09:22.244226 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:22.244180 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99d3f2ce-87ad-4849-8976-7c388312e1e7-istio-podinfo\") pod \"e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c\" (UID: \"99d3f2ce-87ad-4849-8976-7c388312e1e7\") " pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" Oct 30 13:09:22.244350 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:22.244291 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99d3f2ce-87ad-4849-8976-7c388312e1e7-istio-envoy\") pod \"e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c\" (UID: \"99d3f2ce-87ad-4849-8976-7c388312e1e7\") " pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" Oct 30 13:09:22.273655 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:22.268573 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99d3f2ce-87ad-4849-8976-7c388312e1e7-istio-token\") pod \"e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c\" (UID: \"99d3f2ce-87ad-4849-8976-7c388312e1e7\") " pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" Oct 30 13:09:22.273655 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:22.270050 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52l7g\" (UniqueName: \"kubernetes.io/projected/99d3f2ce-87ad-4849-8976-7c388312e1e7-kube-api-access-52l7g\") pod \"e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c\" (UID: \"99d3f2ce-87ad-4849-8976-7c388312e1e7\") " pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" Oct 30 13:09:22.362827 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:22.362739 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:22.386513 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:22.386478 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" Oct 30 13:09:22.726728 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:22.726700 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c"] Oct 30 13:09:22.726957 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:09:22.726936 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99d3f2ce_87ad_4849_8976_7c388312e1e7.slice/crio-b3e05952b1cb5fa72a2ad8157cb7b122b3b9dc218052e40d05fe27232daaf461 WatchSource:0}: Error finding container b3e05952b1cb5fa72a2ad8157cb7b122b3b9dc218052e40d05fe27232daaf461: Status 404 returned error can't find the container with id b3e05952b1cb5fa72a2ad8157cb7b122b3b9dc218052e40d05fe27232daaf461 Oct 30 13:09:22.730338 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:22.730297 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:09:22.730422 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:22.730395 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:09:22.730456 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:22.730440 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:09:23.201386 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:23.201353 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:23.554351 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:09:23.554272 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:09:23.740866 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:23.740824 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" event={"ID":"99d3f2ce-87ad-4849-8976-7c388312e1e7","Type":"ContainerStarted","Data":"357040f68e61a06dd187d19aae4a0c6b89155204e8612c471965c0cd3d35c6db"} Oct 30 13:09:23.741063 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:23.740931 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" event={"ID":"99d3f2ce-87ad-4849-8976-7c388312e1e7","Type":"ContainerStarted","Data":"b3e05952b1cb5fa72a2ad8157cb7b122b3b9dc218052e40d05fe27232daaf461"} Oct 30 13:09:23.802800 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:23.802751 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" podStartSLOduration=1.802737545 podStartE2EDuration="1.802737545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:23.800458684 +0000 UTC m=+1020.269874427" watchObservedRunningTime="2026-10-30 13:09:23.802737545 +0000 UTC m=+1020.272153290" Oct 30 13:09:24.387517 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:24.387483 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" Oct 30 13:09:24.393022 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:24.392999 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" Oct 30 13:09:24.749597 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:24.749568 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" Oct 30 13:09:24.750437 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:24.750419 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" Oct 30 13:09:25.137829 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:25.137736 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:25.480573 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:25.480494 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:25.606849 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:25.606822 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:26.701201 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:26.701162 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:26.832628 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:26.832591 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:26.951452 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:26.951377 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:27.197925 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:27.197883 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:27.343339 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:27.343246 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:27.415055 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:27.415017 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4a59cbc1-7b788ccfc5-k9zhp"] Oct 30 13:09:27.420764 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:27.420744 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4a59cbc1-7b788ccfc5-k9zhp" Oct 30 13:09:27.442361 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:27.442335 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4a59cbc1-7b788ccfc5-k9zhp"] Oct 30 13:09:27.447758 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:27.447726 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-4a59cbc1\"" Oct 30 13:09:27.447885 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:27.447776 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-4a59cbc1-dockercfg-8tx8j\"" Oct 30 13:09:27.468658 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:27.468630 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-470f78b0-ncbmp"] Oct 30 13:09:27.476866 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:27.476848 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-470f78b0-ncbmp" Oct 30 13:09:27.546637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:27.543259 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:09:27.557842 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:27.557811 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:09:27.558258 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:27.558238 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:09:27.559542 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:27.559512 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gjft\" (UniqueName: \"kubernetes.io/projected/c2d03639-8993-4d55-9d09-e4044319f98c-kube-api-access-6gjft\") pod \"payload-pre-processing-e2e-derive-4a59cbc1-7b788ccfc5-k9zhp\" (UID: \"c2d03639-8993-4d55-9d09-e4044319f98c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4a59cbc1-7b788ccfc5-k9zhp" Oct 30 13:09:27.559746 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:27.559724 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2d03639-8993-4d55-9d09-e4044319f98c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4a59cbc1-7b788ccfc5-k9zhp\" (UID: \"c2d03639-8993-4d55-9d09-e4044319f98c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4a59cbc1-7b788ccfc5-k9zhp" Oct 30 13:09:27.560209 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:27.560191 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:09:27.561685 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:27.561665 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-470f78b0-ncbmp"] Oct 30 13:09:27.570005 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:27.569838 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:27.660791 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:27.660709 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/954859db-438e-4108-b8fd-e4b0535b1998-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-470f78b0-ncbmp\" (UID: \"954859db-438e-4108-b8fd-e4b0535b1998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-470f78b0-ncbmp" Oct 30 13:09:27.660984 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:27.660880 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gjft\" (UniqueName: \"kubernetes.io/projected/c2d03639-8993-4d55-9d09-e4044319f98c-kube-api-access-6gjft\") pod \"payload-pre-processing-e2e-derive-4a59cbc1-7b788ccfc5-k9zhp\" (UID: \"c2d03639-8993-4d55-9d09-e4044319f98c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4a59cbc1-7b788ccfc5-k9zhp" Oct 30 13:09:27.660984 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:27.660931 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2d03639-8993-4d55-9d09-e4044319f98c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4a59cbc1-7b788ccfc5-k9zhp\" (UID: \"c2d03639-8993-4d55-9d09-e4044319f98c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4a59cbc1-7b788ccfc5-k9zhp" Oct 30 13:09:27.661112 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:27.660992 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/954859db-438e-4108-b8fd-e4b0535b1998-maas-api-xks-ca\") pod \"maas-api-revoke-sub-470f78b0-ncbmp\" (UID: \"954859db-438e-4108-b8fd-e4b0535b1998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-470f78b0-ncbmp" Oct 30 13:09:27.661112 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:27.661040 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2ptw\" (UniqueName: \"kubernetes.io/projected/954859db-438e-4108-b8fd-e4b0535b1998-kube-api-access-g2ptw\") pod \"maas-api-revoke-sub-470f78b0-ncbmp\" (UID: \"954859db-438e-4108-b8fd-e4b0535b1998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-470f78b0-ncbmp" Oct 30 13:09:27.661592 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:27.661573 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2d03639-8993-4d55-9d09-e4044319f98c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4a59cbc1-7b788ccfc5-k9zhp\" (UID: \"c2d03639-8993-4d55-9d09-e4044319f98c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4a59cbc1-7b788ccfc5-k9zhp" Oct 30 13:09:27.686013 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:27.685990 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gjft\" (UniqueName: \"kubernetes.io/projected/c2d03639-8993-4d55-9d09-e4044319f98c-kube-api-access-6gjft\") pod \"payload-pre-processing-e2e-derive-4a59cbc1-7b788ccfc5-k9zhp\" (UID: \"c2d03639-8993-4d55-9d09-e4044319f98c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4a59cbc1-7b788ccfc5-k9zhp" Oct 30 13:09:27.736886 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:27.736851 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4a59cbc1-7b788ccfc5-k9zhp" Oct 30 13:09:27.762002 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:27.761973 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2ptw\" (UniqueName: \"kubernetes.io/projected/954859db-438e-4108-b8fd-e4b0535b1998-kube-api-access-g2ptw\") pod \"maas-api-revoke-sub-470f78b0-ncbmp\" (UID: \"954859db-438e-4108-b8fd-e4b0535b1998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-470f78b0-ncbmp" Oct 30 13:09:27.762146 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:27.762097 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/954859db-438e-4108-b8fd-e4b0535b1998-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-470f78b0-ncbmp\" (UID: \"954859db-438e-4108-b8fd-e4b0535b1998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-470f78b0-ncbmp" Oct 30 13:09:27.762444 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:27.762419 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/954859db-438e-4108-b8fd-e4b0535b1998-maas-api-xks-ca\") pod \"maas-api-revoke-sub-470f78b0-ncbmp\" (UID: \"954859db-438e-4108-b8fd-e4b0535b1998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-470f78b0-ncbmp" Oct 30 13:09:27.763059 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:27.763036 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/954859db-438e-4108-b8fd-e4b0535b1998-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-470f78b0-ncbmp\" (UID: \"954859db-438e-4108-b8fd-e4b0535b1998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-470f78b0-ncbmp" Oct 30 13:09:27.765390 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:27.765371 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/954859db-438e-4108-b8fd-e4b0535b1998-maas-api-xks-ca\") pod \"maas-api-revoke-sub-470f78b0-ncbmp\" (UID: \"954859db-438e-4108-b8fd-e4b0535b1998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-470f78b0-ncbmp" Oct 30 13:09:27.808416 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:27.808383 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2ptw\" (UniqueName: \"kubernetes.io/projected/954859db-438e-4108-b8fd-e4b0535b1998-kube-api-access-g2ptw\") pod \"maas-api-revoke-sub-470f78b0-ncbmp\" (UID: \"954859db-438e-4108-b8fd-e4b0535b1998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-470f78b0-ncbmp" Oct 30 13:09:27.883940 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:27.883896 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-470f78b0-ncbmp" Oct 30 13:09:28.059986 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:09:28.059951 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2d03639_8993_4d55_9d09_e4044319f98c.slice/crio-625a6d93d2c7e1fcddcd08eab8c0abcb4a1f25ac41d92565c180b3ee293926a4 WatchSource:0}: Error finding container 625a6d93d2c7e1fcddcd08eab8c0abcb4a1f25ac41d92565c180b3ee293926a4: Status 404 returned error can't find the container with id 625a6d93d2c7e1fcddcd08eab8c0abcb4a1f25ac41d92565c180b3ee293926a4 Oct 30 13:09:28.063724 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:28.063702 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4a59cbc1-7b788ccfc5-k9zhp"] Oct 30 13:09:28.208593 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:09:28.208567 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod954859db_438e_4108_b8fd_e4b0535b1998.slice/crio-d3a1312d31ad05f42029be2f8bbef21e314dc2ff7e2d0f190c7a4a03a0a85282 WatchSource:0}: Error finding container d3a1312d31ad05f42029be2f8bbef21e314dc2ff7e2d0f190c7a4a03a0a85282: Status 404 returned error can't find the container with id d3a1312d31ad05f42029be2f8bbef21e314dc2ff7e2d0f190c7a4a03a0a85282 Oct 30 13:09:28.209294 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:28.209230 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-470f78b0-ncbmp"] Oct 30 13:09:28.789478 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:28.789393 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4a59cbc1-7b788ccfc5-k9zhp" event={"ID":"c2d03639-8993-4d55-9d09-e4044319f98c","Type":"ContainerStarted","Data":"406ef21ef396cfdec52a3faffb3e487d90e587909f45dd38506a79612a8fc6d2"} Oct 30 13:09:28.789478 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:28.789455 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4a59cbc1-7b788ccfc5-k9zhp" event={"ID":"c2d03639-8993-4d55-9d09-e4044319f98c","Type":"ContainerStarted","Data":"625a6d93d2c7e1fcddcd08eab8c0abcb4a1f25ac41d92565c180b3ee293926a4"} Oct 30 13:09:28.790018 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:28.789596 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-4a59cbc1-7b788ccfc5-k9zhp" Oct 30 13:09:28.791778 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:28.791745 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-470f78b0-ncbmp" event={"ID":"954859db-438e-4108-b8fd-e4b0535b1998","Type":"ContainerStarted","Data":"7ce43e29216ea6847c30d16aed2700d7d6d584c3b8eaa0d6cd18949dfdc90313"} Oct 30 13:09:28.791914 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:28.791789 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-470f78b0-ncbmp" event={"ID":"954859db-438e-4108-b8fd-e4b0535b1998","Type":"ContainerStarted","Data":"d3a1312d31ad05f42029be2f8bbef21e314dc2ff7e2d0f190c7a4a03a0a85282"} Oct 30 13:09:28.834161 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:28.834116 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-4a59cbc1-7b788ccfc5-k9zhp" podStartSLOduration=1.834102219 podStartE2EDuration="1.834102219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:28.831042397 +0000 UTC m=+1025.300458142" watchObservedRunningTime="2026-10-30 13:09:28.834102219 +0000 UTC m=+1025.303517963" Oct 30 13:09:28.874950 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:28.874897 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-470f78b0-ncbmp" podStartSLOduration=1.874883638 podStartE2EDuration="1.874883638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:28.871669933 +0000 UTC m=+1025.341085679" watchObservedRunningTime="2026-10-30 13:09:28.874883638 +0000 UTC m=+1025.344299382" Oct 30 13:09:29.270449 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:29.270411 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:29.803300 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:29.803263 5922 generic.go:356] "Generic (PLEG): container finished" podID="954859db-438e-4108-b8fd-e4b0535b1998" containerID="7ce43e29216ea6847c30d16aed2700d7d6d584c3b8eaa0d6cd18949dfdc90313" exitCode=0 Oct 30 13:09:29.803766 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:29.803349 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-470f78b0-ncbmp" event={"ID":"954859db-438e-4108-b8fd-e4b0535b1998","Type":"ContainerDied","Data":"7ce43e29216ea6847c30d16aed2700d7d6d584c3b8eaa0d6cd18949dfdc90313"} Oct 30 13:09:30.288637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:30.287885 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rl5vp"] Oct 30 13:09:30.315633 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:30.315390 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rl5vp" Oct 30 13:09:30.336352 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:30.332370 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 13:09:30.336352 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:30.332435 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 13:09:30.336352 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:30.332475 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 13:09:30.336352 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:30.333240 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-fqrgx\"" Oct 30 13:09:30.539702 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:30.518247 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ef89e987-6307-4b0b-9cdc-8059bf210075-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rl5vp\" (UID: \"ef89e987-6307-4b0b-9cdc-8059bf210075\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rl5vp" Oct 30 13:09:30.539702 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:30.518454 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef89e987-6307-4b0b-9cdc-8059bf210075-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rl5vp\" (UID: \"ef89e987-6307-4b0b-9cdc-8059bf210075\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rl5vp" Oct 30 13:09:30.539702 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:30.518550 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef89e987-6307-4b0b-9cdc-8059bf210075-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rl5vp\" (UID: \"ef89e987-6307-4b0b-9cdc-8059bf210075\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rl5vp" Oct 30 13:09:30.539702 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:30.518577 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ef89e987-6307-4b0b-9cdc-8059bf210075-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rl5vp\" (UID: \"ef89e987-6307-4b0b-9cdc-8059bf210075\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rl5vp" Oct 30 13:09:30.539702 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:30.518705 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ef89e987-6307-4b0b-9cdc-8059bf210075-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rl5vp\" (UID: \"ef89e987-6307-4b0b-9cdc-8059bf210075\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rl5vp" Oct 30 13:09:30.539702 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:30.518750 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ef89e987-6307-4b0b-9cdc-8059bf210075-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rl5vp\" (UID: \"ef89e987-6307-4b0b-9cdc-8059bf210075\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rl5vp" Oct 30 13:09:30.539702 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:30.518795 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ghf7\" (UniqueName: \"kubernetes.io/projected/ef89e987-6307-4b0b-9cdc-8059bf210075-kube-api-access-6ghf7\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rl5vp\" (UID: \"ef89e987-6307-4b0b-9cdc-8059bf210075\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rl5vp" Oct 30 13:09:30.552649 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:30.549046 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rl5vp"] Oct 30 13:09:30.629644 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:30.623689 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef89e987-6307-4b0b-9cdc-8059bf210075-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rl5vp\" (UID: \"ef89e987-6307-4b0b-9cdc-8059bf210075\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rl5vp" Oct 30 13:09:30.629644 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:30.623804 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef89e987-6307-4b0b-9cdc-8059bf210075-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rl5vp\" (UID: \"ef89e987-6307-4b0b-9cdc-8059bf210075\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rl5vp" Oct 30 13:09:30.629644 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:30.623833 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ef89e987-6307-4b0b-9cdc-8059bf210075-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rl5vp\" (UID: \"ef89e987-6307-4b0b-9cdc-8059bf210075\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rl5vp" Oct 30 13:09:30.629644 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:30.623972 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ef89e987-6307-4b0b-9cdc-8059bf210075-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rl5vp\" (UID: \"ef89e987-6307-4b0b-9cdc-8059bf210075\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rl5vp" Oct 30 13:09:30.629644 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:30.624029 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ef89e987-6307-4b0b-9cdc-8059bf210075-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rl5vp\" (UID: \"ef89e987-6307-4b0b-9cdc-8059bf210075\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rl5vp" Oct 30 13:09:30.629644 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:30.624086 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ghf7\" (UniqueName: \"kubernetes.io/projected/ef89e987-6307-4b0b-9cdc-8059bf210075-kube-api-access-6ghf7\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rl5vp\" (UID: \"ef89e987-6307-4b0b-9cdc-8059bf210075\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rl5vp" Oct 30 13:09:30.629644 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:30.624151 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ef89e987-6307-4b0b-9cdc-8059bf210075-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rl5vp\" (UID: \"ef89e987-6307-4b0b-9cdc-8059bf210075\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rl5vp" Oct 30 13:09:30.649430 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:30.649397 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ef89e987-6307-4b0b-9cdc-8059bf210075-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rl5vp\" (UID: \"ef89e987-6307-4b0b-9cdc-8059bf210075\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rl5vp" Oct 30 13:09:30.664926 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:30.664881 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ef89e987-6307-4b0b-9cdc-8059bf210075-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rl5vp\" (UID: \"ef89e987-6307-4b0b-9cdc-8059bf210075\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rl5vp" Oct 30 13:09:30.669181 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:30.669146 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ef89e987-6307-4b0b-9cdc-8059bf210075-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rl5vp\" (UID: \"ef89e987-6307-4b0b-9cdc-8059bf210075\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rl5vp" Oct 30 13:09:30.748071 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:30.737846 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ef89e987-6307-4b0b-9cdc-8059bf210075-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rl5vp\" (UID: \"ef89e987-6307-4b0b-9cdc-8059bf210075\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rl5vp" Oct 30 13:09:30.748071 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:30.743875 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef89e987-6307-4b0b-9cdc-8059bf210075-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rl5vp\" (UID: \"ef89e987-6307-4b0b-9cdc-8059bf210075\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rl5vp" Oct 30 13:09:30.772649 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:30.762867 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-751d84fb-666b55f879-5g5l5" Oct 30 13:09:30.822809 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:30.822709 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ghf7\" (UniqueName: \"kubernetes.io/projected/ef89e987-6307-4b0b-9cdc-8059bf210075-kube-api-access-6ghf7\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rl5vp\" (UID: \"ef89e987-6307-4b0b-9cdc-8059bf210075\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rl5vp" Oct 30 13:09:30.887636 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:30.886212 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef89e987-6307-4b0b-9cdc-8059bf210075-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rl5vp\" (UID: \"ef89e987-6307-4b0b-9cdc-8059bf210075\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rl5vp" Oct 30 13:09:31.130697 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:31.117732 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rl5vp" Oct 30 13:09:31.750686 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:31.723721 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" podUID="47517373-759d-4eeb-ab5e-6150b7e82bb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 13:09:31.844637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:31.755864 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" podUID="35619ab9-cbeb-45da-8506-3af63a273177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:31.844637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:31.757339 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" podUID="b487a069-f519-45a3-b1d8-f32f2431797b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 13:09:31.844637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:31.759776 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" podUID="a85de36b-f8c8-4ade-a468-c114080cb312" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:31.844637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:31.796423 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" podUID="5f353928-8cb4-49a3-b999-1b90e6cb31bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 13:09:31.844637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:31.841494 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" podUID="dad5bdad-cd6f-41f1-a124-540c3524ead7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 13:09:31.953588 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:31.952972 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p22vk" podUID="6870ecd9-9f61-4a6f-a753-1ca41fe77166" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:32.241190 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:32.241136 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" podUID="c0ea7fde-2ac5-432d-bb9b-534024186ac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:32.407715 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:32.375353 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-470f78b0-ncbmp" Oct 30 13:09:32.407715 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:32.383640 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/954859db-438e-4108-b8fd-e4b0535b1998-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/954859db-438e-4108-b8fd-e4b0535b1998-maas-api-openshift-ca\") pod \"954859db-438e-4108-b8fd-e4b0535b1998\" (UID: \"954859db-438e-4108-b8fd-e4b0535b1998\") " Oct 30 13:09:32.407715 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:32.383740 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/954859db-438e-4108-b8fd-e4b0535b1998-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/954859db-438e-4108-b8fd-e4b0535b1998-maas-api-xks-ca\") pod \"954859db-438e-4108-b8fd-e4b0535b1998\" (UID: \"954859db-438e-4108-b8fd-e4b0535b1998\") " Oct 30 13:09:32.407715 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:32.384049 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/954859db-438e-4108-b8fd-e4b0535b1998-kube-api-access-g2ptw\" (UniqueName: \"kubernetes.io/projected/954859db-438e-4108-b8fd-e4b0535b1998-kube-api-access-g2ptw\") pod \"954859db-438e-4108-b8fd-e4b0535b1998\" (UID: \"954859db-438e-4108-b8fd-e4b0535b1998\") " Oct 30 13:09:32.408678 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:32.408647 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/954859db-438e-4108-b8fd-e4b0535b1998-maas-api-openshift-ca" pod "954859db-438e-4108-b8fd-e4b0535b1998" (UID: "954859db-438e-4108-b8fd-e4b0535b1998"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:09:32.411481 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:32.411451 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-751d84fb-9696f945b-4spf8" Oct 30 13:09:32.487898 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:32.487865 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/954859db-438e-4108-b8fd-e4b0535b1998-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:09:32.775015 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:32.774973 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" podUID="a85de36b-f8c8-4ade-a468-c114080cb312" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:32.775514 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:32.775486 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" podUID="dfd83cc8-6503-45a8-8260-c0edb8752471" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 13:09:32.776281 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:32.776261 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" Oct 30 13:09:32.782637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:32.780483 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" podUID="14ae0aea-473f-4ca0-bcc0-f617b9839c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:32.792005 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:32.788511 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" podUID="7751779d-b082-409a-a20f-ff0ff6263808" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 13:09:32.792005 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:32.788849 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" podUID="35619ab9-cbeb-45da-8506-3af63a273177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 13:09:32.792005 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:32.789090 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" podUID="b487a069-f519-45a3-b1d8-f32f2431797b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 13:09:32.792005 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:32.789313 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" podUID="47517373-759d-4eeb-ab5e-6150b7e82bb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 13:09:32.792005 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:32.789581 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" Oct 30 13:09:32.808072 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:32.807732 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" podUID="5f353928-8cb4-49a3-b999-1b90e6cb31bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 13:09:32.809740 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:32.809696 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" Oct 30 13:09:32.810189 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:32.810171 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" Oct 30 13:09:32.810433 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:32.810406 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" Oct 30 13:09:32.816649 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:32.814477 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/954859db-438e-4108-b8fd-e4b0535b1998-kube-api-access-g2ptw" pod "954859db-438e-4108-b8fd-e4b0535b1998" (UID: "954859db-438e-4108-b8fd-e4b0535b1998"). InnerVolumeSpecName "kube-api-access-g2ptw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:09:32.852788 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:32.852752 5922 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:09:32.853394 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:32.853360 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" podUID="c61d2f25-754d-4538-b154-b9095bffe96f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:32.893645 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:32.856261 5922 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" podUID="b487a069-f519-45a3-b1d8-f32f2431797b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:32.934708 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:32.859755 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2ptw\" (UniqueName: \"kubernetes.io/projected/954859db-438e-4108-b8fd-e4b0535b1998-kube-api-access-g2ptw\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:09:32.934708 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:32.860376 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" podUID="dad5bdad-cd6f-41f1-a124-540c3524ead7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:32.934708 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:32.924016 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" Oct 30 13:09:32.940121 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:32.940091 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/954859db-438e-4108-b8fd-e4b0535b1998-maas-api-xks-ca" pod "954859db-438e-4108-b8fd-e4b0535b1998" (UID: "954859db-438e-4108-b8fd-e4b0535b1998"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:09:33.040663 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:33.025897 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/954859db-438e-4108-b8fd-e4b0535b1998-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:09:33.142734 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:33.142701 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:33.173787 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:33.173750 5922 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" podUID="a85de36b-f8c8-4ade-a468-c114080cb312" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": context deadline exceeded" Oct 30 13:09:33.178636 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:33.175364 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" podUID="a296df26-3a3b-40ef-a3ab-1866ecf7dcbc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:33.243980 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:33.243937 5922 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" podUID="35619ab9-cbeb-45da-8506-3af63a273177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:33.299852 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:33.297669 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" podUID="c0ea7fde-2ac5-432d-bb9b-534024186ac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:33.299852 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:33.298381 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" Oct 30 13:09:33.779664 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:33.779027 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" podUID="dfd83cc8-6503-45a8-8260-c0edb8752471" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:33.793749 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:33.793707 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" podUID="14ae0aea-473f-4ca0-bcc0-f617b9839c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:33.798987 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:33.798935 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" podUID="a85de36b-f8c8-4ade-a468-c114080cb312" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 13:09:33.821665 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:33.819531 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" podUID="5f353928-8cb4-49a3-b999-1b90e6cb31bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 13:09:33.854636 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:33.846341 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" podUID="47517373-759d-4eeb-ab5e-6150b7e82bb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:34.169124 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:33.846484 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" podUID="35619ab9-cbeb-45da-8506-3af63a273177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 13:09:34.176713 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:33.848074 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" podUID="7751779d-b082-409a-a20f-ff0ff6263808" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 13:09:34.201715 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:33.848115 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" podUID="b487a069-f519-45a3-b1d8-f32f2431797b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:34.205632 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:33.950202 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" podUID="dad5bdad-cd6f-41f1-a124-540c3524ead7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:34.438689 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:34.190544 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" podUID="a296df26-3a3b-40ef-a3ab-1866ecf7dcbc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:34.869935 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:34.869880 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" podUID="5f353928-8cb4-49a3-b999-1b90e6cb31bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 13:09:34.918371 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:34.918323 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" podUID="14ae0aea-473f-4ca0-bcc0-f617b9839c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:34.950452 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:34.950410 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" podUID="c0ea7fde-2ac5-432d-bb9b-534024186ac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 13:09:34.994909 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:34.984604 5922 patch_prober.go:29] interesting pod/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:09:34.994909 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:34.984707 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" podUID="cecdab1d-a927-4d7a-b270-ba7908c88d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:35.015485 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:35.015444 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" podUID="a85de36b-f8c8-4ade-a468-c114080cb312" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 13:09:35.066248 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:09:35.063921 5922 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.336s" Oct 30 13:09:35.066248 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:35.064003 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" Oct 30 13:09:35.066248 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:35.064032 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-470f78b0-ncbmp" event={"ID":"954859db-438e-4108-b8fd-e4b0535b1998","Type":"ContainerDied","Data":"d3a1312d31ad05f42029be2f8bbef21e314dc2ff7e2d0f190c7a4a03a0a85282"} Oct 30 13:09:35.066248 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:35.064069 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3a1312d31ad05f42029be2f8bbef21e314dc2ff7e2d0f190c7a4a03a0a85282" Oct 30 13:09:35.066248 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:35.064158 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" Oct 30 13:09:35.066248 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:35.064196 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" Oct 30 13:09:35.066248 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:35.064229 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" Oct 30 13:09:35.066248 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:35.064259 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" Oct 30 13:09:35.072826 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:35.066903 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-470f78b0-ncbmp" Oct 30 13:09:35.086119 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:35.086095 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" Oct 30 13:09:35.143864 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:35.143782 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" Oct 30 13:09:35.144050 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:35.144035 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rl5vp"] Oct 30 13:09:35.144198 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:35.144189 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" Oct 30 13:09:35.144325 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:35.144316 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" Oct 30 13:09:35.144444 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:35.144435 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" Oct 30 13:09:35.144563 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:35.144555 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" Oct 30 13:09:35.144716 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:35.144706 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" Oct 30 13:09:35.144831 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:35.144823 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" Oct 30 13:09:35.145682 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:35.145661 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" Oct 30 13:09:35.145958 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:35.145942 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" Oct 30 13:09:35.696176 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:35.696134 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rl5vp" event={"ID":"ef89e987-6307-4b0b-9cdc-8059bf210075","Type":"ContainerStarted","Data":"006acdfc2532f669cf3a106b6abaedb7749f95c36a789cf4c036f98692ad824c"} Oct 30 13:09:35.982869 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:35.982787 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:37.058175 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:37.058138 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dad25b3a-8jt96"] Oct 30 13:09:37.069645 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:37.069587 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dad25b3a-8jt96"] Oct 30 13:09:37.525153 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:37.525122 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:09:37.525236 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:37.525185 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:09:37.647636 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:37.647571 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50735aee-920f-4412-a82e-655ce2c43094" path="/var/lib/kubelet/pods/50735aee-920f-4412-a82e-655ce2c43094/volumes" Oct 30 13:09:37.714494 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:37.714460 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rl5vp" event={"ID":"ef89e987-6307-4b0b-9cdc-8059bf210075","Type":"ContainerStarted","Data":"aeb303dee201887666d547e9aa369ce09d96c7e8d96512aa49d33765c36bcb8b"} Oct 30 13:09:37.714683 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:37.714530 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rl5vp" Oct 30 13:09:37.763762 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:37.763718 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rl5vp" podStartSLOduration=5.294888608 podStartE2EDuration="7.76370369s" totalImagesPullingTime="2.468815082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:37.760967866 +0000 UTC m=+1034.230383629" watchObservedRunningTime="2026-10-30 13:09:37.76370369 +0000 UTC m=+1034.233119434" Oct 30 13:09:38.236394 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:38.234791 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:38.724399 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:38.724369 5922 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-rl5vp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:09:38.724583 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:38.724438 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rl5vp" podUID="ef89e987-6307-4b0b-9cdc-8059bf210075" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:09:38.985797 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:38.985710 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:39.038305 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:39.038273 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef480c1f-ffhjq"] Oct 30 13:09:39.049345 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:39.049315 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef480c1f-ffhjq"] Oct 30 13:09:39.647946 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:39.647919 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6e2aea3-e4de-4607-9d53-a5f30cba70e9" path="/var/lib/kubelet/pods/c6e2aea3-e4de-4607-9d53-a5f30cba70e9/volumes" Oct 30 13:09:39.806042 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:39.806014 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-4a59cbc1-7b788ccfc5-k9zhp" Oct 30 13:09:40.164930 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:40.164882 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:40.657478 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:40.657440 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:40.781594 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:40.781564 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:40.933684 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:40.929032 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:41.060710 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:41.060674 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:41.724289 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:41.724255 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rl5vp" Oct 30 13:09:42.968328 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:42.968293 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:44.180501 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:44.180471 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:44.331453 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:44.331417 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:45.000538 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:45.000498 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:45.902520 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:45.902462 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:46.017746 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:46.017705 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:46.968510 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:46.968381 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:48.038132 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:48.038089 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b14df5e-jl9pc"] Oct 30 13:09:48.048791 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:48.048763 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b14df5e-jl9pc"] Oct 30 13:09:48.276284 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:48.276242 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:09:48.752955 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:48.752919 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:48.881883 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:48.881840 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:49.005835 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:49.002514 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:49.647590 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:49.647558 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f0a5c3b-02f4-433d-b710-aba5fb022ff8" path="/var/lib/kubelet/pods/4f0a5c3b-02f4-433d-b710-aba5fb022ff8/volumes" Oct 30 13:09:50.732923 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:50.732880 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m"] Oct 30 13:09:50.734493 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:50.734465 5922 cpu_manager.go:414] "containerMap: removing container" podUID="954859db-438e-4108-b8fd-e4b0535b1998" containerName="revoke-keys" Oct 30 13:09:50.734493 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:50.734493 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="954859db-438e-4108-b8fd-e4b0535b1998" containerName="revoke-keys" Oct 30 13:09:50.734753 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:50.734724 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="954859db-438e-4108-b8fd-e4b0535b1998" containerName="revoke-keys" Oct 30 13:09:50.740218 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:50.740194 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" Oct 30 13:09:50.752346 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:50.752302 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m"] Oct 30 13:09:50.933742 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:50.933709 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g48cf\" (UniqueName: \"kubernetes.io/projected/49ffe88e-df16-4759-9a79-68dc934427fb-kube-api-access-g48cf\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m\" (UID: \"49ffe88e-df16-4759-9a79-68dc934427fb\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" Oct 30 13:09:50.933918 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:50.933849 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49ffe88e-df16-4759-9a79-68dc934427fb-istio-token\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m\" (UID: \"49ffe88e-df16-4759-9a79-68dc934427fb\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" Oct 30 13:09:50.933918 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:50.933882 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49ffe88e-df16-4759-9a79-68dc934427fb-istio-podinfo\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m\" (UID: \"49ffe88e-df16-4759-9a79-68dc934427fb\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" Oct 30 13:09:50.933918 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:50.933909 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ffe88e-df16-4759-9a79-68dc934427fb-workload-socket\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m\" (UID: \"49ffe88e-df16-4759-9a79-68dc934427fb\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" Oct 30 13:09:50.934032 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:50.933931 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49ffe88e-df16-4759-9a79-68dc934427fb-workload-certs\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m\" (UID: \"49ffe88e-df16-4759-9a79-68dc934427fb\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" Oct 30 13:09:50.934032 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:50.933964 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49ffe88e-df16-4759-9a79-68dc934427fb-istio-data\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m\" (UID: \"49ffe88e-df16-4759-9a79-68dc934427fb\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" Oct 30 13:09:50.934032 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:50.934005 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49ffe88e-df16-4759-9a79-68dc934427fb-istiod-ca-cert\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m\" (UID: \"49ffe88e-df16-4759-9a79-68dc934427fb\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" Oct 30 13:09:50.934032 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:50.934023 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49ffe88e-df16-4759-9a79-68dc934427fb-istio-envoy\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m\" (UID: \"49ffe88e-df16-4759-9a79-68dc934427fb\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" Oct 30 13:09:50.934338 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:50.934315 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ffe88e-df16-4759-9a79-68dc934427fb-credential-socket\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m\" (UID: \"49ffe88e-df16-4759-9a79-68dc934427fb\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" Oct 30 13:09:51.035561 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:51.035475 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g48cf\" (UniqueName: \"kubernetes.io/projected/49ffe88e-df16-4759-9a79-68dc934427fb-kube-api-access-g48cf\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m\" (UID: \"49ffe88e-df16-4759-9a79-68dc934427fb\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" Oct 30 13:09:51.035742 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:51.035570 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49ffe88e-df16-4759-9a79-68dc934427fb-istio-token\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m\" (UID: \"49ffe88e-df16-4759-9a79-68dc934427fb\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" Oct 30 13:09:51.035742 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:51.035630 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49ffe88e-df16-4759-9a79-68dc934427fb-istio-podinfo\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m\" (UID: \"49ffe88e-df16-4759-9a79-68dc934427fb\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" Oct 30 13:09:51.035742 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:51.035675 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ffe88e-df16-4759-9a79-68dc934427fb-workload-socket\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m\" (UID: \"49ffe88e-df16-4759-9a79-68dc934427fb\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" Oct 30 13:09:51.035742 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:51.035702 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49ffe88e-df16-4759-9a79-68dc934427fb-workload-certs\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m\" (UID: \"49ffe88e-df16-4759-9a79-68dc934427fb\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" Oct 30 13:09:51.035999 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:51.035755 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49ffe88e-df16-4759-9a79-68dc934427fb-istio-data\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m\" (UID: \"49ffe88e-df16-4759-9a79-68dc934427fb\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" Oct 30 13:09:51.035999 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:51.035814 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49ffe88e-df16-4759-9a79-68dc934427fb-istiod-ca-cert\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m\" (UID: \"49ffe88e-df16-4759-9a79-68dc934427fb\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" Oct 30 13:09:51.035999 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:51.035844 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49ffe88e-df16-4759-9a79-68dc934427fb-istio-envoy\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m\" (UID: \"49ffe88e-df16-4759-9a79-68dc934427fb\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" Oct 30 13:09:51.035999 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:51.035975 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ffe88e-df16-4759-9a79-68dc934427fb-credential-socket\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m\" (UID: \"49ffe88e-df16-4759-9a79-68dc934427fb\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" Oct 30 13:09:51.036204 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:51.036129 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49ffe88e-df16-4759-9a79-68dc934427fb-istio-data\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m\" (UID: \"49ffe88e-df16-4759-9a79-68dc934427fb\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" Oct 30 13:09:51.036204 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:51.036184 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49ffe88e-df16-4759-9a79-68dc934427fb-workload-certs\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m\" (UID: \"49ffe88e-df16-4759-9a79-68dc934427fb\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" Oct 30 13:09:51.036387 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:51.036361 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ffe88e-df16-4759-9a79-68dc934427fb-credential-socket\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m\" (UID: \"49ffe88e-df16-4759-9a79-68dc934427fb\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" Oct 30 13:09:51.036831 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:51.036805 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ffe88e-df16-4759-9a79-68dc934427fb-workload-socket\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m\" (UID: \"49ffe88e-df16-4759-9a79-68dc934427fb\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" Oct 30 13:09:51.036973 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:51.036953 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49ffe88e-df16-4759-9a79-68dc934427fb-istiod-ca-cert\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m\" (UID: \"49ffe88e-df16-4759-9a79-68dc934427fb\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" Oct 30 13:09:51.038403 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:51.038380 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49ffe88e-df16-4759-9a79-68dc934427fb-istio-envoy\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m\" (UID: \"49ffe88e-df16-4759-9a79-68dc934427fb\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" Oct 30 13:09:51.038785 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:51.038763 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49ffe88e-df16-4759-9a79-68dc934427fb-istio-podinfo\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m\" (UID: \"49ffe88e-df16-4759-9a79-68dc934427fb\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" Oct 30 13:09:51.068725 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:51.066311 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49ffe88e-df16-4759-9a79-68dc934427fb-istio-token\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m\" (UID: \"49ffe88e-df16-4759-9a79-68dc934427fb\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" Oct 30 13:09:51.075993 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:51.075324 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g48cf\" (UniqueName: \"kubernetes.io/projected/49ffe88e-df16-4759-9a79-68dc934427fb-kube-api-access-g48cf\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m\" (UID: \"49ffe88e-df16-4759-9a79-68dc934427fb\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" Oct 30 13:09:51.078902 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:51.078876 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" Oct 30 13:09:51.436415 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:09:51.436386 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ffe88e_df16_4759_9a79_68dc934427fb.slice/crio-12e203e156059cbcafb5fcfe0ffe267b673449809530585840b33fb410e508ea WatchSource:0}: Error finding container 12e203e156059cbcafb5fcfe0ffe267b673449809530585840b33fb410e508ea: Status 404 returned error can't find the container with id 12e203e156059cbcafb5fcfe0ffe267b673449809530585840b33fb410e508ea Oct 30 13:09:51.437069 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:51.437048 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m"] Oct 30 13:09:51.438999 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:51.438967 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:09:51.439070 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:51.439033 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:09:51.439070 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:51.439063 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:09:51.641341 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:51.641301 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:52.048758 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:52.048719 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" event={"ID":"49ffe88e-df16-4759-9a79-68dc934427fb","Type":"ContainerStarted","Data":"3c0071616a73c6daf5ec02ef82d3eab5952db36646dac9d434f90f1e42480b75"} Oct 30 13:09:52.049153 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:52.048783 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" event={"ID":"49ffe88e-df16-4759-9a79-68dc934427fb","Type":"ContainerStarted","Data":"12e203e156059cbcafb5fcfe0ffe267b673449809530585840b33fb410e508ea"} Oct 30 13:09:52.079906 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:52.079879 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" Oct 30 13:09:52.081088 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:52.081064 5922 patch_prober.go:29] interesting pod/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 30 13:09:52.081192 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:52.081106 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" podUID="49ffe88e-df16-4759-9a79-68dc934427fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 30 13:09:52.115882 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:52.115840 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" podStartSLOduration=2.115827164 podStartE2EDuration="2.115827164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:52.112781741 +0000 UTC m=+1048.582197486" watchObservedRunningTime="2026-10-30 13:09:52.115827164 +0000 UTC m=+1048.585242909" Oct 30 13:09:53.086795 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:53.086750 5922 patch_prober.go:29] interesting pod/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 30 13:09:53.087253 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:53.086811 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" podUID="49ffe88e-df16-4759-9a79-68dc934427fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 30 13:09:53.391777 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:53.391683 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:53.559694 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:09:53.559659 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:09:53.933017 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:53.932984 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:54.087084 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:54.086845 5922 patch_prober.go:29] interesting pod/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 30 13:09:54.087084 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:54.086926 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" podUID="49ffe88e-df16-4759-9a79-68dc934427fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 30 13:09:55.086675 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:55.085045 5922 patch_prober.go:29] interesting pod/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 30 13:09:55.086675 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:55.085112 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" podUID="49ffe88e-df16-4759-9a79-68dc934427fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 30 13:09:55.488801 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:55.488766 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:56.082637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:56.079454 5922 patch_prober.go:29] interesting pod/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 30 13:09:56.082637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:56.079510 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" podUID="49ffe88e-df16-4759-9a79-68dc934427fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 30 13:09:57.093759 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:57.087911 5922 patch_prober.go:29] interesting pod/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 30 13:09:57.098785 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:57.098260 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" podUID="49ffe88e-df16-4759-9a79-68dc934427fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 30 13:09:57.388064 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:57.387975 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:58.079921 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:58.079887 5922 patch_prober.go:29] interesting pod/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 30 13:09:58.080100 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:58.079946 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" podUID="49ffe88e-df16-4759-9a79-68dc934427fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 30 13:09:59.042767 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:59.042738 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5db1191-qqmdj"] Oct 30 13:09:59.054993 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:59.054965 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5db1191-qqmdj"] Oct 30 13:09:59.085152 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:59.085126 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" Oct 30 13:09:59.085501 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:59.085472 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" Oct 30 13:09:59.086071 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:59.086051 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" Oct 30 13:09:59.199556 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:59.199516 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:59.645693 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:09:59.645663 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc9f17c6-e5e1-4a1e-9525-a511055902d3" path="/var/lib/kubelet/pods/bc9f17c6-e5e1-4a1e-9525-a511055902d3/volumes" Oct 30 13:10:00.857100 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:00.857064 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:01.284474 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:01.284438 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:01.438922 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:01.438894 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:03.478075 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:03.478033 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:03.601153 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:03.601121 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:03.715891 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:03.715836 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:05.039766 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:05.039727 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d278e5f4-vk9z7"] Oct 30 13:10:05.054697 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:05.054662 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c97ae1fc-mqwzp"] Oct 30 13:10:05.062648 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:05.062601 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d278e5f4-vk9z7"] Oct 30 13:10:05.071624 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:05.071570 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c97ae1fc-mqwzp"] Oct 30 13:10:05.646367 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:05.646334 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a94f1261-e840-4407-87c9-395005a540b1" path="/var/lib/kubelet/pods/a94f1261-e840-4407-87c9-395005a540b1/volumes" Oct 30 13:10:05.646898 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:05.646882 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1cc7211-6b43-4fed-874d-749577c986dc" path="/var/lib/kubelet/pods/c1cc7211-6b43-4fed-874d-749577c986dc/volumes" Oct 30 13:10:06.037055 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:06.037026 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edbca662-tc4ch"] Oct 30 13:10:06.041867 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:06.041835 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edbca662-tc4ch"] Oct 30 13:10:06.187103 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:06.187075 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:07.653182 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:07.653151 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa" path="/var/lib/kubelet/pods/0b77e948-6004-45d0-8cfa-a1cbd3fdb0fa/volumes" Oct 30 13:10:07.973996 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:07.973964 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:08.104073 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:08.104040 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d84fb-6kns6"] Oct 30 13:10:08.111805 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:08.111778 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d84fb-6kns6" Oct 30 13:10:08.135602 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:08.135492 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d84fb-6kns6"] Oct 30 13:10:08.139750 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:08.139596 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:10:08.139750 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:08.139667 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:10:08.139750 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:08.139711 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:10:08.139750 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:08.139742 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:10:08.213759 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:08.213725 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26zl7\" (UniqueName: \"kubernetes.io/projected/93b58167-8ba3-4a56-94a0-5010c4323c8e-kube-api-access-26zl7\") pod \"maas-api-revoke-keys-e2e-ait-751d84fb-6kns6\" (UID: \"93b58167-8ba3-4a56-94a0-5010c4323c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d84fb-6kns6" Oct 30 13:10:08.213921 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:08.213760 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93b58167-8ba3-4a56-94a0-5010c4323c8e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-751d84fb-6kns6\" (UID: \"93b58167-8ba3-4a56-94a0-5010c4323c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d84fb-6kns6" Oct 30 13:10:08.213983 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:08.213970 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93b58167-8ba3-4a56-94a0-5010c4323c8e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-751d84fb-6kns6\" (UID: \"93b58167-8ba3-4a56-94a0-5010c4323c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d84fb-6kns6" Oct 30 13:10:08.315730 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:08.315656 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26zl7\" (UniqueName: \"kubernetes.io/projected/93b58167-8ba3-4a56-94a0-5010c4323c8e-kube-api-access-26zl7\") pod \"maas-api-revoke-keys-e2e-ait-751d84fb-6kns6\" (UID: \"93b58167-8ba3-4a56-94a0-5010c4323c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d84fb-6kns6" Oct 30 13:10:08.315730 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:08.315687 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93b58167-8ba3-4a56-94a0-5010c4323c8e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-751d84fb-6kns6\" (UID: \"93b58167-8ba3-4a56-94a0-5010c4323c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d84fb-6kns6" Oct 30 13:10:08.315943 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:08.315855 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93b58167-8ba3-4a56-94a0-5010c4323c8e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-751d84fb-6kns6\" (UID: \"93b58167-8ba3-4a56-94a0-5010c4323c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d84fb-6kns6" Oct 30 13:10:08.316483 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:08.316463 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93b58167-8ba3-4a56-94a0-5010c4323c8e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-751d84fb-6kns6\" (UID: \"93b58167-8ba3-4a56-94a0-5010c4323c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d84fb-6kns6" Oct 30 13:10:08.318224 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:08.318197 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93b58167-8ba3-4a56-94a0-5010c4323c8e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-751d84fb-6kns6\" (UID: \"93b58167-8ba3-4a56-94a0-5010c4323c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d84fb-6kns6" Oct 30 13:10:08.340935 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:08.340902 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26zl7\" (UniqueName: \"kubernetes.io/projected/93b58167-8ba3-4a56-94a0-5010c4323c8e-kube-api-access-26zl7\") pod \"maas-api-revoke-keys-e2e-ait-751d84fb-6kns6\" (UID: \"93b58167-8ba3-4a56-94a0-5010c4323c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d84fb-6kns6" Oct 30 13:10:08.433906 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:08.433879 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d84fb-6kns6" Oct 30 13:10:08.750050 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:08.750016 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d84fb-6kns6"] Oct 30 13:10:08.751057 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:10:08.751025 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93b58167_8ba3_4a56_94a0_5010c4323c8e.slice/crio-5c641164a6b02c2e84744546d8ef587c90c18cfce8dcd0e4688ede470e6fa6b3 WatchSource:0}: Error finding container 5c641164a6b02c2e84744546d8ef587c90c18cfce8dcd0e4688ede470e6fa6b3: Status 404 returned error can't find the container with id 5c641164a6b02c2e84744546d8ef587c90c18cfce8dcd0e4688ede470e6fa6b3 Oct 30 13:10:09.043726 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:09.043633 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0aed1791-cwtk2"] Oct 30 13:10:09.055275 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:09.055246 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0aed1791-cwtk2"] Oct 30 13:10:09.230068 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:09.230032 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d84fb-6kns6" event={"ID":"93b58167-8ba3-4a56-94a0-5010c4323c8e","Type":"ContainerStarted","Data":"cb9f276c1c16ddcf5712aea5471d1a07a01c35542d0ab9a5adfb48b2308ca603"} Oct 30 13:10:09.230245 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:09.230089 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d84fb-6kns6" event={"ID":"93b58167-8ba3-4a56-94a0-5010c4323c8e","Type":"ContainerStarted","Data":"5c641164a6b02c2e84744546d8ef587c90c18cfce8dcd0e4688ede470e6fa6b3"} Oct 30 13:10:09.284322 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:09.284275 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d84fb-6kns6" podStartSLOduration=1.284260184 podStartE2EDuration="1.284260184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:09.281833131 +0000 UTC m=+1065.751248876" watchObservedRunningTime="2026-10-30 13:10:09.284260184 +0000 UTC m=+1065.753675928" Oct 30 13:10:09.646693 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:09.646665 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="215248bc-d335-49c0-96a0-6835913f5b35" path="/var/lib/kubelet/pods/215248bc-d335-49c0-96a0-6835913f5b35/volumes" Oct 30 13:10:10.162410 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:10.162359 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:10.240977 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:10.240944 5922 generic.go:356] "Generic (PLEG): container finished" podID="93b58167-8ba3-4a56-94a0-5010c4323c8e" containerID="cb9f276c1c16ddcf5712aea5471d1a07a01c35542d0ab9a5adfb48b2308ca603" exitCode=0 Oct 30 13:10:10.241170 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:10.241032 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d84fb-6kns6" event={"ID":"93b58167-8ba3-4a56-94a0-5010c4323c8e","Type":"ContainerDied","Data":"cb9f276c1c16ddcf5712aea5471d1a07a01c35542d0ab9a5adfb48b2308ca603"} Oct 30 13:10:11.432212 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:11.432179 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d84fb-6kns6" Oct 30 13:10:11.461800 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:11.461768 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93b58167-8ba3-4a56-94a0-5010c4323c8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93b58167-8ba3-4a56-94a0-5010c4323c8e-maas-api-xks-ca\") pod \"93b58167-8ba3-4a56-94a0-5010c4323c8e\" (UID: \"93b58167-8ba3-4a56-94a0-5010c4323c8e\") " Oct 30 13:10:11.461935 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:11.461863 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93b58167-8ba3-4a56-94a0-5010c4323c8e-kube-api-access-26zl7\" (UniqueName: \"kubernetes.io/projected/93b58167-8ba3-4a56-94a0-5010c4323c8e-kube-api-access-26zl7\") pod \"93b58167-8ba3-4a56-94a0-5010c4323c8e\" (UID: \"93b58167-8ba3-4a56-94a0-5010c4323c8e\") " Oct 30 13:10:11.462072 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:11.462048 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93b58167-8ba3-4a56-94a0-5010c4323c8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93b58167-8ba3-4a56-94a0-5010c4323c8e-maas-api-openshift-ca\") pod \"93b58167-8ba3-4a56-94a0-5010c4323c8e\" (UID: \"93b58167-8ba3-4a56-94a0-5010c4323c8e\") " Oct 30 13:10:11.462352 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:11.462330 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93b58167-8ba3-4a56-94a0-5010c4323c8e-maas-api-openshift-ca" pod "93b58167-8ba3-4a56-94a0-5010c4323c8e" (UID: "93b58167-8ba3-4a56-94a0-5010c4323c8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:10:11.462699 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:11.462673 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93b58167-8ba3-4a56-94a0-5010c4323c8e-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:10:11.464327 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:11.464302 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93b58167-8ba3-4a56-94a0-5010c4323c8e-maas-api-xks-ca" pod "93b58167-8ba3-4a56-94a0-5010c4323c8e" (UID: "93b58167-8ba3-4a56-94a0-5010c4323c8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:10:11.464433 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:11.464340 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93b58167-8ba3-4a56-94a0-5010c4323c8e-kube-api-access-26zl7" pod "93b58167-8ba3-4a56-94a0-5010c4323c8e" (UID: "93b58167-8ba3-4a56-94a0-5010c4323c8e"). InnerVolumeSpecName "kube-api-access-26zl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:11.564469 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:11.564356 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93b58167-8ba3-4a56-94a0-5010c4323c8e-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:10:11.564469 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:11.564386 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26zl7\" (UniqueName: \"kubernetes.io/projected/93b58167-8ba3-4a56-94a0-5010c4323c8e-kube-api-access-26zl7\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:10:12.264312 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:12.264279 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d84fb-6kns6" event={"ID":"93b58167-8ba3-4a56-94a0-5010c4323c8e","Type":"ContainerDied","Data":"5c641164a6b02c2e84744546d8ef587c90c18cfce8dcd0e4688ede470e6fa6b3"} Oct 30 13:10:12.264501 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:12.264324 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d84fb-6kns6" Oct 30 13:10:12.264501 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:12.264335 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c641164a6b02c2e84744546d8ef587c90c18cfce8dcd0e4688ede470e6fa6b3" Oct 30 13:10:12.461941 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:12.461904 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:14.078524 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:14.078472 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" podUID="47517373-759d-4eeb-ab5e-6150b7e82bb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:14.117744 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:14.117697 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" podUID="a296df26-3a3b-40ef-a3ab-1866ecf7dcbc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:14.119204 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:14.117984 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" podUID="c0ea7fde-2ac5-432d-bb9b-534024186ac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:14.129744 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:14.129708 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" podUID="14ae0aea-473f-4ca0-bcc0-f617b9839c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:14.156015 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:14.155974 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" podUID="7751779d-b082-409a-a20f-ff0ff6263808" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:14.159989 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:14.156257 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" podUID="dad5bdad-cd6f-41f1-a124-540c3524ead7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:14.160839 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:14.156305 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" podUID="35619ab9-cbeb-45da-8506-3af63a273177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:14.220810 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:14.220762 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" podUID="b487a069-f519-45a3-b1d8-f32f2431797b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 13:10:14.452693 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:14.430746 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" podUID="dfd83cc8-6503-45a8-8260-c0edb8752471" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:14.901832 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:14.901731 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" podUID="5f353928-8cb4-49a3-b999-1b90e6cb31bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:14.925769 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:14.925724 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" podUID="a85de36b-f8c8-4ade-a468-c114080cb312" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:15.062638 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:15.059379 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a85a078-798zq"] Oct 30 13:10:15.118729 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:15.118694 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a85a078-798zq"] Oct 30 13:10:15.119348 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:15.119315 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" podUID="a296df26-3a3b-40ef-a3ab-1866ecf7dcbc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.110:8000/ready\": context deadline exceeded" Oct 30 13:10:15.133631 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:15.086167 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" podUID="47517373-759d-4eeb-ab5e-6150b7e82bb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:15.134156 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:15.122498 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" Oct 30 13:10:15.134372 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:15.134355 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" Oct 30 13:10:15.152682 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:15.122806 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" podUID="c0ea7fde-2ac5-432d-bb9b-534024186ac9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 13:10:15.153153 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:15.130255 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" podUID="14ae0aea-473f-4ca0-bcc0-f617b9839c8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:15.153584 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:15.153563 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" Oct 30 13:10:15.153778 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:15.153763 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" Oct 30 13:10:15.161048 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:15.161022 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" podUID="dad5bdad-cd6f-41f1-a124-540c3524ead7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 13:10:15.161474 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:15.161448 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" podUID="35619ab9-cbeb-45da-8506-3af63a273177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:15.162037 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:15.162017 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" Oct 30 13:10:15.167013 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:15.166996 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" Oct 30 13:10:15.441258 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:15.432585 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" podUID="dfd83cc8-6503-45a8-8260-c0edb8752471" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:15.441258 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:15.433262 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" Oct 30 13:10:15.604772 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:15.604738 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" Oct 30 13:10:15.922336 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:15.922290 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aac684d3-17d4-4d51-a781-81a824f44956" path="/var/lib/kubelet/pods/aac684d3-17d4-4d51-a781-81a824f44956/volumes" Oct 30 13:10:15.923229 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:15.923211 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" Oct 30 13:10:15.927289 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:15.924508 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz" Oct 30 13:10:15.927580 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:15.927566 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" Oct 30 13:10:15.928411 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:15.928396 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" Oct 30 13:10:15.928771 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:15.928752 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" Oct 30 13:10:15.950304 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:15.950275 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" Oct 30 13:10:16.394737 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:16.394692 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4a59cbc1-j79tv"] Oct 30 13:10:16.395811 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:16.395783 5922 cpu_manager.go:414] "containerMap: removing container" podUID="93b58167-8ba3-4a56-94a0-5010c4323c8e" containerName="revoke-keys" Oct 30 13:10:16.395811 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:16.395809 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93b58167-8ba3-4a56-94a0-5010c4323c8e" containerName="revoke-keys" Oct 30 13:10:16.396028 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:16.396013 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="93b58167-8ba3-4a56-94a0-5010c4323c8e" containerName="revoke-keys" Oct 30 13:10:16.399728 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:16.399700 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4a59cbc1-j79tv" Oct 30 13:10:16.400671 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:16.400466 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4a59cbc1-j79tv"] Oct 30 13:10:16.416280 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:16.416255 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:10:16.416467 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:16.416265 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:10:16.416467 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:16.416347 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:10:16.416467 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:16.416351 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:10:16.515684 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:16.515645 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab24fa9e-2607-497f-abe5-760e5e55c4d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-4a59cbc1-j79tv\" (UID: \"ab24fa9e-2607-497f-abe5-760e5e55c4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4a59cbc1-j79tv" Oct 30 13:10:16.515858 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:16.515758 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8j7z\" (UniqueName: \"kubernetes.io/projected/ab24fa9e-2607-497f-abe5-760e5e55c4d6-kube-api-access-s8j7z\") pod \"maas-api-revoke-keys-e2e-derive-4a59cbc1-j79tv\" (UID: \"ab24fa9e-2607-497f-abe5-760e5e55c4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4a59cbc1-j79tv" Oct 30 13:10:16.515921 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:16.515908 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab24fa9e-2607-497f-abe5-760e5e55c4d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-4a59cbc1-j79tv\" (UID: \"ab24fa9e-2607-497f-abe5-760e5e55c4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4a59cbc1-j79tv" Oct 30 13:10:16.617221 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:16.617185 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab24fa9e-2607-497f-abe5-760e5e55c4d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-4a59cbc1-j79tv\" (UID: \"ab24fa9e-2607-497f-abe5-760e5e55c4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4a59cbc1-j79tv" Oct 30 13:10:16.617390 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:16.617250 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8j7z\" (UniqueName: \"kubernetes.io/projected/ab24fa9e-2607-497f-abe5-760e5e55c4d6-kube-api-access-s8j7z\") pod \"maas-api-revoke-keys-e2e-derive-4a59cbc1-j79tv\" (UID: \"ab24fa9e-2607-497f-abe5-760e5e55c4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4a59cbc1-j79tv" Oct 30 13:10:16.617455 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:16.617386 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab24fa9e-2607-497f-abe5-760e5e55c4d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-4a59cbc1-j79tv\" (UID: \"ab24fa9e-2607-497f-abe5-760e5e55c4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4a59cbc1-j79tv" Oct 30 13:10:16.617998 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:16.617972 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab24fa9e-2607-497f-abe5-760e5e55c4d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-4a59cbc1-j79tv\" (UID: \"ab24fa9e-2607-497f-abe5-760e5e55c4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4a59cbc1-j79tv" Oct 30 13:10:16.619820 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:16.619795 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab24fa9e-2607-497f-abe5-760e5e55c4d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-4a59cbc1-j79tv\" (UID: \"ab24fa9e-2607-497f-abe5-760e5e55c4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4a59cbc1-j79tv" Oct 30 13:10:16.642089 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:16.642063 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8j7z\" (UniqueName: \"kubernetes.io/projected/ab24fa9e-2607-497f-abe5-760e5e55c4d6-kube-api-access-s8j7z\") pod \"maas-api-revoke-keys-e2e-derive-4a59cbc1-j79tv\" (UID: \"ab24fa9e-2607-497f-abe5-760e5e55c4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4a59cbc1-j79tv" Oct 30 13:10:16.713816 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:16.713788 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4a59cbc1-j79tv" Oct 30 13:10:16.743945 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:16.743918 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d33f9772-6rtmj"] Oct 30 13:10:16.747718 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:16.747700 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33f9772-6rtmj" Oct 30 13:10:16.803946 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:16.803914 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d33f9772-6rtmj"] Oct 30 13:10:16.825281 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:16.825254 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0faf6428-6b88-40c6-b3a2-bc963b9f3bc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d33f9772-6rtmj\" (UID: \"0faf6428-6b88-40c6-b3a2-bc963b9f3bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33f9772-6rtmj" Oct 30 13:10:16.825439 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:16.825287 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0faf6428-6b88-40c6-b3a2-bc963b9f3bc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d33f9772-6rtmj\" (UID: \"0faf6428-6b88-40c6-b3a2-bc963b9f3bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33f9772-6rtmj" Oct 30 13:10:16.825490 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:16.825442 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd7sj\" (UniqueName: \"kubernetes.io/projected/0faf6428-6b88-40c6-b3a2-bc963b9f3bc0-kube-api-access-jd7sj\") pod \"maas-api-revoke-sub-d33f9772-6rtmj\" (UID: \"0faf6428-6b88-40c6-b3a2-bc963b9f3bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33f9772-6rtmj" Oct 30 13:10:16.888247 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:16.888195 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:16.945034 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:16.927216 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0faf6428-6b88-40c6-b3a2-bc963b9f3bc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d33f9772-6rtmj\" (UID: \"0faf6428-6b88-40c6-b3a2-bc963b9f3bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33f9772-6rtmj" Oct 30 13:10:16.945034 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:16.927263 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0faf6428-6b88-40c6-b3a2-bc963b9f3bc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d33f9772-6rtmj\" (UID: \"0faf6428-6b88-40c6-b3a2-bc963b9f3bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33f9772-6rtmj" Oct 30 13:10:16.945034 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:16.927371 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd7sj\" (UniqueName: \"kubernetes.io/projected/0faf6428-6b88-40c6-b3a2-bc963b9f3bc0-kube-api-access-jd7sj\") pod \"maas-api-revoke-sub-d33f9772-6rtmj\" (UID: \"0faf6428-6b88-40c6-b3a2-bc963b9f3bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33f9772-6rtmj" Oct 30 13:10:16.945034 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:16.932157 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0faf6428-6b88-40c6-b3a2-bc963b9f3bc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d33f9772-6rtmj\" (UID: \"0faf6428-6b88-40c6-b3a2-bc963b9f3bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33f9772-6rtmj" Oct 30 13:10:16.945034 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:16.934883 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0faf6428-6b88-40c6-b3a2-bc963b9f3bc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d33f9772-6rtmj\" (UID: \"0faf6428-6b88-40c6-b3a2-bc963b9f3bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33f9772-6rtmj" Oct 30 13:10:16.954930 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:16.954903 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd7sj\" (UniqueName: \"kubernetes.io/projected/0faf6428-6b88-40c6-b3a2-bc963b9f3bc0-kube-api-access-jd7sj\") pod \"maas-api-revoke-sub-d33f9772-6rtmj\" (UID: \"0faf6428-6b88-40c6-b3a2-bc963b9f3bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33f9772-6rtmj" Oct 30 13:10:17.056319 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:10:17.056289 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab24fa9e_2607_497f_abe5_760e5e55c4d6.slice/crio-846ff78fe7fe280e805c87f916aa79abd9cda4db43ccbdd91250b9b9e52cc2b5 WatchSource:0}: Error finding container 846ff78fe7fe280e805c87f916aa79abd9cda4db43ccbdd91250b9b9e52cc2b5: Status 404 returned error can't find the container with id 846ff78fe7fe280e805c87f916aa79abd9cda4db43ccbdd91250b9b9e52cc2b5 Oct 30 13:10:17.057193 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:17.057171 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4a59cbc1-j79tv"] Oct 30 13:10:17.081636 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:17.076840 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33f9772-6rtmj" Oct 30 13:10:17.414464 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:10:17.414435 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0faf6428_6b88_40c6_b3a2_bc963b9f3bc0.slice/crio-f2f17fd94ff174a2a89631480fa2b4d9e9cce3df9acdc02a5407d37ca1abb4de WatchSource:0}: Error finding container f2f17fd94ff174a2a89631480fa2b4d9e9cce3df9acdc02a5407d37ca1abb4de: Status 404 returned error can't find the container with id f2f17fd94ff174a2a89631480fa2b4d9e9cce3df9acdc02a5407d37ca1abb4de Oct 30 13:10:17.415525 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:17.415497 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d33f9772-6rtmj"] Oct 30 13:10:17.650367 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:17.650293 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4a59cbc1-j79tv" event={"ID":"ab24fa9e-2607-497f-abe5-760e5e55c4d6","Type":"ContainerStarted","Data":"35f1cda2c709c6dd298c22aba5749904bbdd03a3156a5fba08bd70d4759fce11"} Oct 30 13:10:17.650367 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:17.650345 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4a59cbc1-j79tv" event={"ID":"ab24fa9e-2607-497f-abe5-760e5e55c4d6","Type":"ContainerStarted","Data":"846ff78fe7fe280e805c87f916aa79abd9cda4db43ccbdd91250b9b9e52cc2b5"} Oct 30 13:10:17.650367 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:17.650362 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33f9772-6rtmj" event={"ID":"0faf6428-6b88-40c6-b3a2-bc963b9f3bc0","Type":"ContainerStarted","Data":"f2f17fd94ff174a2a89631480fa2b4d9e9cce3df9acdc02a5407d37ca1abb4de"} Oct 30 13:10:17.720141 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:17.720088 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4a59cbc1-j79tv" podStartSLOduration=1.720074566 podStartE2EDuration="1.720074566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:17.715375419 +0000 UTC m=+1074.184791164" watchObservedRunningTime="2026-10-30 13:10:17.720074566 +0000 UTC m=+1074.189490310" Oct 30 13:10:18.562072 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:18.562037 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-751d84fb-9696f945b-4spf8"] Oct 30 13:10:18.597628 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:18.597583 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-751d84fb-666b55f879-5g5l5"] Oct 30 13:10:18.661220 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:18.661186 5922 generic.go:356] "Generic (PLEG): container finished" podID="ab24fa9e-2607-497f-abe5-760e5e55c4d6" containerID="35f1cda2c709c6dd298c22aba5749904bbdd03a3156a5fba08bd70d4759fce11" exitCode=0 Oct 30 13:10:18.661398 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:18.661315 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4a59cbc1-j79tv" event={"ID":"ab24fa9e-2607-497f-abe5-760e5e55c4d6","Type":"ContainerDied","Data":"35f1cda2c709c6dd298c22aba5749904bbdd03a3156a5fba08bd70d4759fce11"} Oct 30 13:10:18.663842 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:18.663809 5922 generic.go:356] "Generic (PLEG): container finished" podID="0faf6428-6b88-40c6-b3a2-bc963b9f3bc0" containerID="68d5dd32bdc2d1b092bf92165ddb66d1b4be0f7dd9e0af71a99a35f730df0757" exitCode=0 Oct 30 13:10:18.664010 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:18.663993 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33f9772-6rtmj" event={"ID":"0faf6428-6b88-40c6-b3a2-bc963b9f3bc0","Type":"ContainerDied","Data":"68d5dd32bdc2d1b092bf92165ddb66d1b4be0f7dd9e0af71a99a35f730df0757"} Oct 30 13:10:18.714865 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:18.714833 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:18.835807 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:18.835714 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:19.054457 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:19.054423 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc8032e6-xsrnf"] Oct 30 13:10:19.065677 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:19.065595 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc8032e6-xsrnf"] Oct 30 13:10:19.266738 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:19.266705 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:19.647841 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:19.647765 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0484e59d-627c-4eab-a3b5-2436e0a6c7a0" path="/var/lib/kubelet/pods/0484e59d-627c-4eab-a3b5-2436e0a6c7a0/volumes" Oct 30 13:10:19.886866 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:19.886837 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4a59cbc1-j79tv" Oct 30 13:10:19.953702 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:19.949510 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33f9772-6rtmj" Oct 30 13:10:19.986635 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:19.983332 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab24fa9e-2607-497f-abe5-760e5e55c4d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab24fa9e-2607-497f-abe5-760e5e55c4d6-maas-api-xks-ca\") pod \"ab24fa9e-2607-497f-abe5-760e5e55c4d6\" (UID: \"ab24fa9e-2607-497f-abe5-760e5e55c4d6\") " Oct 30 13:10:19.986635 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:19.983397 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab24fa9e-2607-497f-abe5-760e5e55c4d6-kube-api-access-s8j7z\" (UniqueName: \"kubernetes.io/projected/ab24fa9e-2607-497f-abe5-760e5e55c4d6-kube-api-access-s8j7z\") pod \"ab24fa9e-2607-497f-abe5-760e5e55c4d6\" (UID: \"ab24fa9e-2607-497f-abe5-760e5e55c4d6\") " Oct 30 13:10:19.986635 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:19.983507 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab24fa9e-2607-497f-abe5-760e5e55c4d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab24fa9e-2607-497f-abe5-760e5e55c4d6-maas-api-openshift-ca\") pod \"ab24fa9e-2607-497f-abe5-760e5e55c4d6\" (UID: \"ab24fa9e-2607-497f-abe5-760e5e55c4d6\") " Oct 30 13:10:19.986635 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:19.986123 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab24fa9e-2607-497f-abe5-760e5e55c4d6-maas-api-openshift-ca" pod "ab24fa9e-2607-497f-abe5-760e5e55c4d6" (UID: "ab24fa9e-2607-497f-abe5-760e5e55c4d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:10:19.997259 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:19.994242 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab24fa9e-2607-497f-abe5-760e5e55c4d6-kube-api-access-s8j7z" pod "ab24fa9e-2607-497f-abe5-760e5e55c4d6" (UID: "ab24fa9e-2607-497f-abe5-760e5e55c4d6"). InnerVolumeSpecName "kube-api-access-s8j7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:19.997259 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:19.995569 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab24fa9e-2607-497f-abe5-760e5e55c4d6-maas-api-xks-ca" pod "ab24fa9e-2607-497f-abe5-760e5e55c4d6" (UID: "ab24fa9e-2607-497f-abe5-760e5e55c4d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:10:20.085146 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:20.085110 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0faf6428-6b88-40c6-b3a2-bc963b9f3bc0-kube-api-access-jd7sj\" (UniqueName: \"kubernetes.io/projected/0faf6428-6b88-40c6-b3a2-bc963b9f3bc0-kube-api-access-jd7sj\") pod \"0faf6428-6b88-40c6-b3a2-bc963b9f3bc0\" (UID: \"0faf6428-6b88-40c6-b3a2-bc963b9f3bc0\") " Oct 30 13:10:20.085327 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:20.085169 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0faf6428-6b88-40c6-b3a2-bc963b9f3bc0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0faf6428-6b88-40c6-b3a2-bc963b9f3bc0-maas-api-openshift-ca\") pod \"0faf6428-6b88-40c6-b3a2-bc963b9f3bc0\" (UID: \"0faf6428-6b88-40c6-b3a2-bc963b9f3bc0\") " Oct 30 13:10:20.085327 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:20.085214 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0faf6428-6b88-40c6-b3a2-bc963b9f3bc0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0faf6428-6b88-40c6-b3a2-bc963b9f3bc0-maas-api-xks-ca\") pod \"0faf6428-6b88-40c6-b3a2-bc963b9f3bc0\" (UID: \"0faf6428-6b88-40c6-b3a2-bc963b9f3bc0\") " Oct 30 13:10:20.085707 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:20.085671 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0faf6428-6b88-40c6-b3a2-bc963b9f3bc0-maas-api-openshift-ca" pod "0faf6428-6b88-40c6-b3a2-bc963b9f3bc0" (UID: "0faf6428-6b88-40c6-b3a2-bc963b9f3bc0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:10:20.086151 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:20.086125 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab24fa9e-2607-497f-abe5-760e5e55c4d6-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:10:20.086271 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:20.086156 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab24fa9e-2607-497f-abe5-760e5e55c4d6-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:10:20.086271 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:20.086171 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8j7z\" (UniqueName: \"kubernetes.io/projected/ab24fa9e-2607-497f-abe5-760e5e55c4d6-kube-api-access-s8j7z\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:10:20.086271 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:20.086186 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0faf6428-6b88-40c6-b3a2-bc963b9f3bc0-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:10:20.087463 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:20.087436 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0faf6428-6b88-40c6-b3a2-bc963b9f3bc0-kube-api-access-jd7sj" pod "0faf6428-6b88-40c6-b3a2-bc963b9f3bc0" (UID: "0faf6428-6b88-40c6-b3a2-bc963b9f3bc0"). InnerVolumeSpecName "kube-api-access-jd7sj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:20.087564 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:20.087540 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0faf6428-6b88-40c6-b3a2-bc963b9f3bc0-maas-api-xks-ca" pod "0faf6428-6b88-40c6-b3a2-bc963b9f3bc0" (UID: "0faf6428-6b88-40c6-b3a2-bc963b9f3bc0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:10:20.187560 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:20.187477 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd7sj\" (UniqueName: \"kubernetes.io/projected/0faf6428-6b88-40c6-b3a2-bc963b9f3bc0-kube-api-access-jd7sj\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:10:20.187560 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:20.187508 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0faf6428-6b88-40c6-b3a2-bc963b9f3bc0-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:10:20.687498 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:20.687469 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33f9772-6rtmj" Oct 30 13:10:20.687984 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:20.687516 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33f9772-6rtmj" event={"ID":"0faf6428-6b88-40c6-b3a2-bc963b9f3bc0","Type":"ContainerDied","Data":"f2f17fd94ff174a2a89631480fa2b4d9e9cce3df9acdc02a5407d37ca1abb4de"} Oct 30 13:10:20.687984 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:20.687578 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2f17fd94ff174a2a89631480fa2b4d9e9cce3df9acdc02a5407d37ca1abb4de" Oct 30 13:10:20.693665 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:20.689740 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4a59cbc1-j79tv" event={"ID":"ab24fa9e-2607-497f-abe5-760e5e55c4d6","Type":"ContainerDied","Data":"846ff78fe7fe280e805c87f916aa79abd9cda4db43ccbdd91250b9b9e52cc2b5"} Oct 30 13:10:20.693665 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:20.689767 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4a59cbc1-j79tv" Oct 30 13:10:20.693665 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:20.689787 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="846ff78fe7fe280e805c87f916aa79abd9cda4db43ccbdd91250b9b9e52cc2b5" Oct 30 13:10:21.542999 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:21.542965 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:22.712158 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:22.712122 5922 generic.go:356] "Generic (PLEG): container finished" podID="99d3f2ce-87ad-4849-8976-7c388312e1e7" containerID="357040f68e61a06dd187d19aae4a0c6b89155204e8612c471965c0cd3d35c6db" exitCode=0 Oct 30 13:10:22.712585 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:22.712199 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" event={"ID":"99d3f2ce-87ad-4849-8976-7c388312e1e7","Type":"ContainerDied","Data":"357040f68e61a06dd187d19aae4a0c6b89155204e8612c471965c0cd3d35c6db"} Oct 30 13:10:22.712585 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:22.712566 5922 scope.go:122] "RemoveContainer" containerID="357040f68e61a06dd187d19aae4a0c6b89155204e8612c471965c0cd3d35c6db" Oct 30 13:10:22.714077 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:22.714051 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:10:22.714144 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:22.714117 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:10:22.714188 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:22.714174 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:10:23.553365 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:10:23.553322 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:10:23.562513 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:23.562487 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-751d84fb-9696f945b-4spf8" podUID="e5eecf33-3127-48f3-af3d-428dc1ca61da" containerName="payload-processing" containerID="cri-o://139c6c57ea8e6bf1faccfc11d572e8d8bd46a11f865890c880a333fd4235f263" gracePeriod=25 Oct 30 13:10:23.598744 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:23.598716 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-751d84fb-666b55f879-5g5l5" podUID="e9fa2d25-84a2-4041-8cba-9988db6ef2ed" containerName="payload-pre-processing" containerID="cri-o://422da1f9b38ec1664f38469ce44b29f9c922ce79045d6134933b92f9bc1a3ade" gracePeriod=25 Oct 30 13:10:23.674506 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:23.674477 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:23.749286 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:23.749223 5922 generic.go:356] "Generic (PLEG): container finished" podID="e5eecf33-3127-48f3-af3d-428dc1ca61da" containerID="139c6c57ea8e6bf1faccfc11d572e8d8bd46a11f865890c880a333fd4235f263" exitCode=0 Oct 30 13:10:23.749806 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:23.749343 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-751d84fb-9696f945b-4spf8" event={"ID":"e5eecf33-3127-48f3-af3d-428dc1ca61da","Type":"ContainerDied","Data":"139c6c57ea8e6bf1faccfc11d572e8d8bd46a11f865890c880a333fd4235f263"} Oct 30 13:10:23.752566 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:23.752535 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" event={"ID":"99d3f2ce-87ad-4849-8976-7c388312e1e7","Type":"ContainerStarted","Data":"57d0b32569a17bb01048eea9963047f15952754e0f1dedbf7c8c8c9fb8e583af"} Oct 30 13:10:23.754769 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:23.754740 5922 generic.go:356] "Generic (PLEG): container finished" podID="e9fa2d25-84a2-4041-8cba-9988db6ef2ed" containerID="422da1f9b38ec1664f38469ce44b29f9c922ce79045d6134933b92f9bc1a3ade" exitCode=0 Oct 30 13:10:23.754926 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:23.754815 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-751d84fb-666b55f879-5g5l5" event={"ID":"e9fa2d25-84a2-4041-8cba-9988db6ef2ed","Type":"ContainerDied","Data":"422da1f9b38ec1664f38469ce44b29f9c922ce79045d6134933b92f9bc1a3ade"} Oct 30 13:10:24.018016 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:24.017980 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-751d84fb-9696f945b-4spf8" Oct 30 13:10:24.042439 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:24.042411 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5eecf33-3127-48f3-af3d-428dc1ca61da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5eecf33-3127-48f3-af3d-428dc1ca61da-plugins-config-volume\") pod \"e5eecf33-3127-48f3-af3d-428dc1ca61da\" (UID: \"e5eecf33-3127-48f3-af3d-428dc1ca61da\") " Oct 30 13:10:24.042727 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:24.042711 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5eecf33-3127-48f3-af3d-428dc1ca61da-kube-api-access-th2kk\" (UniqueName: \"kubernetes.io/projected/e5eecf33-3127-48f3-af3d-428dc1ca61da-kube-api-access-th2kk\") pod \"e5eecf33-3127-48f3-af3d-428dc1ca61da\" (UID: \"e5eecf33-3127-48f3-af3d-428dc1ca61da\") " Oct 30 13:10:24.042817 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:24.042794 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5eecf33-3127-48f3-af3d-428dc1ca61da-plugins-config-volume" pod "e5eecf33-3127-48f3-af3d-428dc1ca61da" (UID: "e5eecf33-3127-48f3-af3d-428dc1ca61da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:10:24.043127 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:24.043106 5922 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5eecf33-3127-48f3-af3d-428dc1ca61da-plugins-config-volume\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:10:24.044813 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:24.044793 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5eecf33-3127-48f3-af3d-428dc1ca61da-kube-api-access-th2kk" pod "e5eecf33-3127-48f3-af3d-428dc1ca61da" (UID: "e5eecf33-3127-48f3-af3d-428dc1ca61da"). InnerVolumeSpecName "kube-api-access-th2kk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:24.114891 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:24.114826 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-751d84fb-666b55f879-5g5l5" Oct 30 13:10:24.143743 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:24.143712 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9fa2d25-84a2-4041-8cba-9988db6ef2ed-kube-api-access-m8cck\" (UniqueName: \"kubernetes.io/projected/e9fa2d25-84a2-4041-8cba-9988db6ef2ed-kube-api-access-m8cck\") pod \"e9fa2d25-84a2-4041-8cba-9988db6ef2ed\" (UID: \"e9fa2d25-84a2-4041-8cba-9988db6ef2ed\") " Oct 30 13:10:24.143907 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:24.143831 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9fa2d25-84a2-4041-8cba-9988db6ef2ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9fa2d25-84a2-4041-8cba-9988db6ef2ed-plugins-config-volume\") pod \"e9fa2d25-84a2-4041-8cba-9988db6ef2ed\" (UID: \"e9fa2d25-84a2-4041-8cba-9988db6ef2ed\") " Oct 30 13:10:24.144231 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:24.144202 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9fa2d25-84a2-4041-8cba-9988db6ef2ed-plugins-config-volume" pod "e9fa2d25-84a2-4041-8cba-9988db6ef2ed" (UID: "e9fa2d25-84a2-4041-8cba-9988db6ef2ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:10:24.144591 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:24.144568 5922 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9fa2d25-84a2-4041-8cba-9988db6ef2ed-plugins-config-volume\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:10:24.144699 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:24.144595 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th2kk\" (UniqueName: \"kubernetes.io/projected/e5eecf33-3127-48f3-af3d-428dc1ca61da-kube-api-access-th2kk\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:10:24.146299 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:24.146274 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9fa2d25-84a2-4041-8cba-9988db6ef2ed-kube-api-access-m8cck" pod "e9fa2d25-84a2-4041-8cba-9988db6ef2ed" (UID: "e9fa2d25-84a2-4041-8cba-9988db6ef2ed"). InnerVolumeSpecName "kube-api-access-m8cck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:24.245516 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:24.245476 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8cck\" (UniqueName: \"kubernetes.io/projected/e9fa2d25-84a2-4041-8cba-9988db6ef2ed-kube-api-access-m8cck\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:10:24.386870 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:24.386772 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" Oct 30 13:10:24.388098 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:24.388068 5922 patch_prober.go:29] interesting pod/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 30 13:10:24.388189 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:24.388129 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" podUID="99d3f2ce-87ad-4849-8976-7c388312e1e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 30 13:10:24.749592 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:24.749561 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" Oct 30 13:10:24.790637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:24.789714 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-751d84fb-666b55f879-5g5l5" Oct 30 13:10:24.790637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:24.789768 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-751d84fb-666b55f879-5g5l5" event={"ID":"e9fa2d25-84a2-4041-8cba-9988db6ef2ed","Type":"ContainerDied","Data":"a6d5f2ed651848f00e40526fc3904c34d8de490c68b29dc347a33998476dbd54"} Oct 30 13:10:24.790637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:24.789877 5922 scope.go:122] "RemoveContainer" containerID="422da1f9b38ec1664f38469ce44b29f9c922ce79045d6134933b92f9bc1a3ade" Oct 30 13:10:24.794214 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:24.794185 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-751d84fb-9696f945b-4spf8" Oct 30 13:10:24.794355 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:24.794238 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-751d84fb-9696f945b-4spf8" event={"ID":"e5eecf33-3127-48f3-af3d-428dc1ca61da","Type":"ContainerDied","Data":"888e0b8ca0fff3ff31381c55516a61b62b68fa840d1f9c12d03331a75332bd84"} Oct 30 13:10:24.806909 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:24.806888 5922 scope.go:122] "RemoveContainer" containerID="139c6c57ea8e6bf1faccfc11d572e8d8bd46a11f865890c880a333fd4235f263" Oct 30 13:10:24.851785 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:24.851755 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-751d84fb-666b55f879-5g5l5"] Oct 30 13:10:24.873635 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:24.865298 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-751d84fb-666b55f879-5g5l5"] Oct 30 13:10:24.919389 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:24.919359 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-751d84fb-9696f945b-4spf8"] Oct 30 13:10:24.925110 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:24.925088 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-751d84fb-9696f945b-4spf8"] Oct 30 13:10:25.387057 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:25.387016 5922 patch_prober.go:29] interesting pod/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 30 13:10:25.387267 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:25.387082 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" podUID="99d3f2ce-87ad-4849-8976-7c388312e1e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 30 13:10:25.646169 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:25.646095 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5eecf33-3127-48f3-af3d-428dc1ca61da" path="/var/lib/kubelet/pods/e5eecf33-3127-48f3-af3d-428dc1ca61da/volumes" Oct 30 13:10:25.646641 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:25.646601 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9fa2d25-84a2-4041-8cba-9988db6ef2ed" path="/var/lib/kubelet/pods/e9fa2d25-84a2-4041-8cba-9988db6ef2ed/volumes" Oct 30 13:10:25.660168 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:25.660134 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:26.386954 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:26.386918 5922 patch_prober.go:29] interesting pod/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 30 13:10:26.387390 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:26.386975 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" podUID="99d3f2ce-87ad-4849-8976-7c388312e1e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 30 13:10:26.894474 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:26.894440 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4a59cbc1-7b788ccfc5-k9zhp"] Oct 30 13:10:27.012871 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:27.012838 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:27.138750 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:27.138714 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:27.346299 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:27.346265 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-999b8bdc-fmt85"] Oct 30 13:10:27.347225 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:27.347187 5922 cpu_manager.go:414] "containerMap: removing container" podUID="e5eecf33-3127-48f3-af3d-428dc1ca61da" containerName="payload-processing" Oct 30 13:10:27.347225 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:27.347223 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5eecf33-3127-48f3-af3d-428dc1ca61da" containerName="payload-processing" Oct 30 13:10:27.347398 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:27.347247 5922 cpu_manager.go:414] "containerMap: removing container" podUID="ab24fa9e-2607-497f-abe5-760e5e55c4d6" containerName="revoke-keys" Oct 30 13:10:27.347398 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:27.347254 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab24fa9e-2607-497f-abe5-760e5e55c4d6" containerName="revoke-keys" Oct 30 13:10:27.347398 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:27.347323 5922 cpu_manager.go:414] "containerMap: removing container" podUID="e9fa2d25-84a2-4041-8cba-9988db6ef2ed" containerName="payload-pre-processing" Oct 30 13:10:27.347398 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:27.347331 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9fa2d25-84a2-4041-8cba-9988db6ef2ed" containerName="payload-pre-processing" Oct 30 13:10:27.347398 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:27.347354 5922 cpu_manager.go:414] "containerMap: removing container" podUID="0faf6428-6b88-40c6-b3a2-bc963b9f3bc0" containerName="revoke-keys" Oct 30 13:10:27.347398 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:27.347361 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0faf6428-6b88-40c6-b3a2-bc963b9f3bc0" containerName="revoke-keys" Oct 30 13:10:27.347581 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:27.347530 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9fa2d25-84a2-4041-8cba-9988db6ef2ed" containerName="payload-pre-processing" Oct 30 13:10:27.347581 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:27.347560 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="0faf6428-6b88-40c6-b3a2-bc963b9f3bc0" containerName="revoke-keys" Oct 30 13:10:27.347667 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:27.347579 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5eecf33-3127-48f3-af3d-428dc1ca61da" containerName="payload-processing" Oct 30 13:10:27.347667 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:27.347600 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab24fa9e-2607-497f-abe5-760e5e55c4d6" containerName="revoke-keys" Oct 30 13:10:27.353935 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:27.353915 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-999b8bdc-fmt85" Oct 30 13:10:27.397113 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:27.397083 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:10:27.398008 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:27.397985 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:10:27.398710 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:27.398687 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:10:27.398804 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:27.398777 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:10:27.399903 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:27.399875 5922 patch_prober.go:29] interesting pod/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 30 13:10:27.400005 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:27.399927 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" podUID="99d3f2ce-87ad-4849-8976-7c388312e1e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 30 13:10:27.410759 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:27.410474 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-999b8bdc-fmt85"] Oct 30 13:10:27.416719 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:27.416688 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f904654e-bf69-45ec-9293-a555868acb57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-999b8bdc-fmt85\" (UID: \"f904654e-bf69-45ec-9293-a555868acb57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-999b8bdc-fmt85" Oct 30 13:10:27.416836 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:27.416816 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f904654e-bf69-45ec-9293-a555868acb57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-999b8bdc-fmt85\" (UID: \"f904654e-bf69-45ec-9293-a555868acb57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-999b8bdc-fmt85" Oct 30 13:10:27.416968 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:27.416951 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p94hf\" (UniqueName: \"kubernetes.io/projected/f904654e-bf69-45ec-9293-a555868acb57-kube-api-access-p94hf\") pod \"maas-api-revoke-sub-999b8bdc-fmt85\" (UID: \"f904654e-bf69-45ec-9293-a555868acb57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-999b8bdc-fmt85" Oct 30 13:10:27.518469 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:27.518438 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f904654e-bf69-45ec-9293-a555868acb57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-999b8bdc-fmt85\" (UID: \"f904654e-bf69-45ec-9293-a555868acb57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-999b8bdc-fmt85" Oct 30 13:10:27.518647 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:27.518526 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f904654e-bf69-45ec-9293-a555868acb57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-999b8bdc-fmt85\" (UID: \"f904654e-bf69-45ec-9293-a555868acb57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-999b8bdc-fmt85" Oct 30 13:10:27.518647 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:27.518592 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p94hf\" (UniqueName: \"kubernetes.io/projected/f904654e-bf69-45ec-9293-a555868acb57-kube-api-access-p94hf\") pod \"maas-api-revoke-sub-999b8bdc-fmt85\" (UID: \"f904654e-bf69-45ec-9293-a555868acb57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-999b8bdc-fmt85" Oct 30 13:10:27.519275 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:27.519248 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f904654e-bf69-45ec-9293-a555868acb57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-999b8bdc-fmt85\" (UID: \"f904654e-bf69-45ec-9293-a555868acb57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-999b8bdc-fmt85" Oct 30 13:10:27.520958 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:27.520928 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f904654e-bf69-45ec-9293-a555868acb57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-999b8bdc-fmt85\" (UID: \"f904654e-bf69-45ec-9293-a555868acb57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-999b8bdc-fmt85" Oct 30 13:10:27.544336 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:27.544274 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p94hf\" (UniqueName: \"kubernetes.io/projected/f904654e-bf69-45ec-9293-a555868acb57-kube-api-access-p94hf\") pod \"maas-api-revoke-sub-999b8bdc-fmt85\" (UID: \"f904654e-bf69-45ec-9293-a555868acb57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-999b8bdc-fmt85" Oct 30 13:10:27.708934 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:27.708819 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-999b8bdc-fmt85" Oct 30 13:10:27.902882 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:27.902569 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:28.148311 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:28.148284 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-999b8bdc-fmt85"] Oct 30 13:10:28.148554 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:10:28.148534 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf904654e_bf69_45ec_9293_a555868acb57.slice/crio-8f8cee8e2e4d5b4f5d58b3e8988b451d39729449292c98fbecbc9810c90730e6 WatchSource:0}: Error finding container 8f8cee8e2e4d5b4f5d58b3e8988b451d39729449292c98fbecbc9810c90730e6: Status 404 returned error can't find the container with id 8f8cee8e2e4d5b4f5d58b3e8988b451d39729449292c98fbecbc9810c90730e6 Oct 30 13:10:28.388636 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:28.388222 5922 patch_prober.go:29] interesting pod/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 30 13:10:28.388636 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:28.388281 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" podUID="99d3f2ce-87ad-4849-8976-7c388312e1e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 30 13:10:28.854656 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:28.850779 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-999b8bdc-fmt85" event={"ID":"f904654e-bf69-45ec-9293-a555868acb57","Type":"ContainerStarted","Data":"b0e194606ccb4bf0affeef4db95b72c405d4a7c4e285eefdf7001f935b2767e1"} Oct 30 13:10:28.854656 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:28.850848 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-999b8bdc-fmt85" event={"ID":"f904654e-bf69-45ec-9293-a555868acb57","Type":"ContainerStarted","Data":"8f8cee8e2e4d5b4f5d58b3e8988b451d39729449292c98fbecbc9810c90730e6"} Oct 30 13:10:28.905814 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:28.905761 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-999b8bdc-fmt85" podStartSLOduration=1.9057468370000001 podStartE2EDuration="1.905746837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:28.90502025 +0000 UTC m=+1085.374435996" watchObservedRunningTime="2026-10-30 13:10:28.905746837 +0000 UTC m=+1085.375162578" Oct 30 13:10:29.056309 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:29.056272 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:29.222334 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:29.222299 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:29.307259 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:29.307227 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d84fb-6kns6"] Oct 30 13:10:29.313406 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:29.313373 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d84fb-6kns6"] Oct 30 13:10:29.387859 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:29.387818 5922 patch_prober.go:29] interesting pod/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 30 13:10:29.388057 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:29.387883 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" podUID="99d3f2ce-87ad-4849-8976-7c388312e1e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 30 13:10:29.503669 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:29.503510 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:29.626637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:29.620980 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:29.656861 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:29.656827 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93b58167-8ba3-4a56-94a0-5010c4323c8e" path="/var/lib/kubelet/pods/93b58167-8ba3-4a56-94a0-5010c4323c8e/volumes" Oct 30 13:10:29.862018 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:29.861933 5922 generic.go:356] "Generic (PLEG): container finished" podID="f904654e-bf69-45ec-9293-a555868acb57" containerID="b0e194606ccb4bf0affeef4db95b72c405d4a7c4e285eefdf7001f935b2767e1" exitCode=0 Oct 30 13:10:29.862018 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:29.861996 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-999b8bdc-fmt85" event={"ID":"f904654e-bf69-45ec-9293-a555868acb57","Type":"ContainerDied","Data":"b0e194606ccb4bf0affeef4db95b72c405d4a7c4e285eefdf7001f935b2767e1"} Oct 30 13:10:30.159480 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:30.159389 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:30.278324 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:30.278280 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:30.391854 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:30.391825 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" Oct 30 13:10:30.392877 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:30.392854 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" Oct 30 13:10:30.483122 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:30.483092 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:30.995768 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:30.995684 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:31.160420 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:31.160351 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-999b8bdc-fmt85" Oct 30 13:10:31.350410 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:31.350377 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f904654e-bf69-45ec-9293-a555868acb57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f904654e-bf69-45ec-9293-a555868acb57-maas-api-openshift-ca\") pod \"f904654e-bf69-45ec-9293-a555868acb57\" (UID: \"f904654e-bf69-45ec-9293-a555868acb57\") " Oct 30 13:10:31.350410 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:31.350417 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f904654e-bf69-45ec-9293-a555868acb57-kube-api-access-p94hf\" (UniqueName: \"kubernetes.io/projected/f904654e-bf69-45ec-9293-a555868acb57-kube-api-access-p94hf\") pod \"f904654e-bf69-45ec-9293-a555868acb57\" (UID: \"f904654e-bf69-45ec-9293-a555868acb57\") " Oct 30 13:10:31.350665 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:31.350636 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f904654e-bf69-45ec-9293-a555868acb57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f904654e-bf69-45ec-9293-a555868acb57-maas-api-xks-ca\") pod \"f904654e-bf69-45ec-9293-a555868acb57\" (UID: \"f904654e-bf69-45ec-9293-a555868acb57\") " Oct 30 13:10:31.350764 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:31.350742 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f904654e-bf69-45ec-9293-a555868acb57-maas-api-openshift-ca" pod "f904654e-bf69-45ec-9293-a555868acb57" (UID: "f904654e-bf69-45ec-9293-a555868acb57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:10:31.351262 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:31.351240 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f904654e-bf69-45ec-9293-a555868acb57-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:10:31.352574 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:31.352543 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f904654e-bf69-45ec-9293-a555868acb57-maas-api-xks-ca" pod "f904654e-bf69-45ec-9293-a555868acb57" (UID: "f904654e-bf69-45ec-9293-a555868acb57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:10:31.352712 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:31.352608 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f904654e-bf69-45ec-9293-a555868acb57-kube-api-access-p94hf" pod "f904654e-bf69-45ec-9293-a555868acb57" (UID: "f904654e-bf69-45ec-9293-a555868acb57"). InnerVolumeSpecName "kube-api-access-p94hf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:31.452507 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:31.452480 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f904654e-bf69-45ec-9293-a555868acb57-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:10:31.452507 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:31.452507 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p94hf\" (UniqueName: \"kubernetes.io/projected/f904654e-bf69-45ec-9293-a555868acb57-kube-api-access-p94hf\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:10:31.556247 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:31.556216 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:31.885391 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:31.885316 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-999b8bdc-fmt85" Oct 30 13:10:31.885391 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:31.885352 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-999b8bdc-fmt85" event={"ID":"f904654e-bf69-45ec-9293-a555868acb57","Type":"ContainerDied","Data":"8f8cee8e2e4d5b4f5d58b3e8988b451d39729449292c98fbecbc9810c90730e6"} Oct 30 13:10:31.885568 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:31.885406 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f8cee8e2e4d5b4f5d58b3e8988b451d39729449292c98fbecbc9810c90730e6" Oct 30 13:10:31.895344 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:31.895315 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-4a59cbc1-7b788ccfc5-k9zhp" podUID="c2d03639-8993-4d55-9d09-e4044319f98c" containerName="payload-pre-processing" containerID="cri-o://406ef21ef396cfdec52a3faffb3e487d90e587909f45dd38506a79612a8fc6d2" gracePeriod=25 Oct 30 13:10:32.239755 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:32.239726 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4a59cbc1-7b788ccfc5-k9zhp" Oct 30 13:10:32.267777 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:32.267733 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2d03639-8993-4d55-9d09-e4044319f98c-kube-api-access-6gjft\" (UniqueName: \"kubernetes.io/projected/c2d03639-8993-4d55-9d09-e4044319f98c-kube-api-access-6gjft\") pod \"c2d03639-8993-4d55-9d09-e4044319f98c\" (UID: \"c2d03639-8993-4d55-9d09-e4044319f98c\") " Oct 30 13:10:32.275804 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:32.275774 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2d03639-8993-4d55-9d09-e4044319f98c-kube-api-access-6gjft" pod "c2d03639-8993-4d55-9d09-e4044319f98c" (UID: "c2d03639-8993-4d55-9d09-e4044319f98c"). InnerVolumeSpecName "kube-api-access-6gjft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:32.363375 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:32.363337 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:10:32.369175 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:32.369149 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2d03639-8993-4d55-9d09-e4044319f98c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2d03639-8993-4d55-9d09-e4044319f98c-plugins-config-volume\") pod \"c2d03639-8993-4d55-9d09-e4044319f98c\" (UID: \"c2d03639-8993-4d55-9d09-e4044319f98c\") " Oct 30 13:10:32.369533 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:32.369510 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2d03639-8993-4d55-9d09-e4044319f98c-plugins-config-volume" pod "c2d03639-8993-4d55-9d09-e4044319f98c" (UID: "c2d03639-8993-4d55-9d09-e4044319f98c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:10:32.369851 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:32.369832 5922 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2d03639-8993-4d55-9d09-e4044319f98c-plugins-config-volume\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:10:32.369931 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:32.369854 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gjft\" (UniqueName: \"kubernetes.io/projected/c2d03639-8993-4d55-9d09-e4044319f98c-kube-api-access-6gjft\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:10:32.909663 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:32.902224 5922 generic.go:356] "Generic (PLEG): container finished" podID="c2d03639-8993-4d55-9d09-e4044319f98c" containerID="406ef21ef396cfdec52a3faffb3e487d90e587909f45dd38506a79612a8fc6d2" exitCode=0 Oct 30 13:10:32.909663 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:32.902290 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4a59cbc1-7b788ccfc5-k9zhp" Oct 30 13:10:32.909663 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:32.902338 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4a59cbc1-7b788ccfc5-k9zhp" event={"ID":"c2d03639-8993-4d55-9d09-e4044319f98c","Type":"ContainerDied","Data":"406ef21ef396cfdec52a3faffb3e487d90e587909f45dd38506a79612a8fc6d2"} Oct 30 13:10:32.909663 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:32.902384 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4a59cbc1-7b788ccfc5-k9zhp" event={"ID":"c2d03639-8993-4d55-9d09-e4044319f98c","Type":"ContainerDied","Data":"625a6d93d2c7e1fcddcd08eab8c0abcb4a1f25ac41d92565c180b3ee293926a4"} Oct 30 13:10:32.909663 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:32.902417 5922 scope.go:122] "RemoveContainer" containerID="406ef21ef396cfdec52a3faffb3e487d90e587909f45dd38506a79612a8fc6d2" Oct 30 13:10:32.917220 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:32.917191 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:32.919177 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:32.919148 5922 scope.go:122] "RemoveContainer" containerID="406ef21ef396cfdec52a3faffb3e487d90e587909f45dd38506a79612a8fc6d2" Oct 30 13:10:32.919539 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:10:32.919517 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"406ef21ef396cfdec52a3faffb3e487d90e587909f45dd38506a79612a8fc6d2\": container with ID starting with 406ef21ef396cfdec52a3faffb3e487d90e587909f45dd38506a79612a8fc6d2 not found: ID does not exist" containerID="406ef21ef396cfdec52a3faffb3e487d90e587909f45dd38506a79612a8fc6d2" Oct 30 13:10:32.919640 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:32.919549 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"406ef21ef396cfdec52a3faffb3e487d90e587909f45dd38506a79612a8fc6d2"} err="failed to get container status \"406ef21ef396cfdec52a3faffb3e487d90e587909f45dd38506a79612a8fc6d2\": rpc error: code = NotFound desc = could not find container \"406ef21ef396cfdec52a3faffb3e487d90e587909f45dd38506a79612a8fc6d2\": container with ID starting with 406ef21ef396cfdec52a3faffb3e487d90e587909f45dd38506a79612a8fc6d2 not found: ID does not exist" Oct 30 13:10:32.962018 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:32.961984 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4a59cbc1-7b788ccfc5-k9zhp"] Oct 30 13:10:32.974944 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:32.974912 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4a59cbc1-7b788ccfc5-k9zhp"] Oct 30 13:10:33.210700 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:33.210581 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:33.648469 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:33.648393 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2d03639-8993-4d55-9d09-e4044319f98c" path="/var/lib/kubelet/pods/c2d03639-8993-4d55-9d09-e4044319f98c/volumes" Oct 30 13:10:35.648865 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:35.648789 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk"] Oct 30 13:10:35.649451 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:35.649436 5922 cpu_manager.go:414] "containerMap: removing container" podUID="f904654e-bf69-45ec-9293-a555868acb57" containerName="revoke-keys" Oct 30 13:10:35.649495 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:35.649452 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f904654e-bf69-45ec-9293-a555868acb57" containerName="revoke-keys" Oct 30 13:10:35.649495 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:35.649488 5922 cpu_manager.go:414] "containerMap: removing container" podUID="c2d03639-8993-4d55-9d09-e4044319f98c" containerName="payload-pre-processing" Oct 30 13:10:35.649559 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:35.649494 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2d03639-8993-4d55-9d09-e4044319f98c" containerName="payload-pre-processing" Oct 30 13:10:35.649598 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:35.649590 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2d03639-8993-4d55-9d09-e4044319f98c" containerName="payload-pre-processing" Oct 30 13:10:35.649652 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:35.649604 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="f904654e-bf69-45ec-9293-a555868acb57" containerName="revoke-keys" Oct 30 13:10:35.654883 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:35.654863 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" Oct 30 13:10:35.677592 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:35.677564 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk"] Oct 30 13:10:35.748986 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:35.748951 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f0665fa-cce6-4a67-97c7-88982d036ac8-istio-data\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk\" (UID: \"9f0665fa-cce6-4a67-97c7-88982d036ac8\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" Oct 30 13:10:35.749159 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:35.749071 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f0665fa-cce6-4a67-97c7-88982d036ac8-istio-envoy\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk\" (UID: \"9f0665fa-cce6-4a67-97c7-88982d036ac8\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" Oct 30 13:10:35.749159 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:35.749122 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f0665fa-cce6-4a67-97c7-88982d036ac8-istiod-ca-cert\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk\" (UID: \"9f0665fa-cce6-4a67-97c7-88982d036ac8\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" Oct 30 13:10:35.749159 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:35.749151 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f0665fa-cce6-4a67-97c7-88982d036ac8-istio-podinfo\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk\" (UID: \"9f0665fa-cce6-4a67-97c7-88982d036ac8\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" Oct 30 13:10:35.749461 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:35.749437 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kt7b\" (UniqueName: \"kubernetes.io/projected/9f0665fa-cce6-4a67-97c7-88982d036ac8-kube-api-access-8kt7b\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk\" (UID: \"9f0665fa-cce6-4a67-97c7-88982d036ac8\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" Oct 30 13:10:35.749597 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:35.749577 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f0665fa-cce6-4a67-97c7-88982d036ac8-istio-token\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk\" (UID: \"9f0665fa-cce6-4a67-97c7-88982d036ac8\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" Oct 30 13:10:35.749732 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:35.749670 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f0665fa-cce6-4a67-97c7-88982d036ac8-workload-socket\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk\" (UID: \"9f0665fa-cce6-4a67-97c7-88982d036ac8\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" Oct 30 13:10:35.749831 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:35.749806 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f0665fa-cce6-4a67-97c7-88982d036ac8-workload-certs\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk\" (UID: \"9f0665fa-cce6-4a67-97c7-88982d036ac8\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" Oct 30 13:10:35.749919 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:35.749901 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f0665fa-cce6-4a67-97c7-88982d036ac8-credential-socket\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk\" (UID: \"9f0665fa-cce6-4a67-97c7-88982d036ac8\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" Oct 30 13:10:35.851199 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:35.851170 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f0665fa-cce6-4a67-97c7-88982d036ac8-credential-socket\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk\" (UID: \"9f0665fa-cce6-4a67-97c7-88982d036ac8\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" Oct 30 13:10:35.851336 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:35.851213 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f0665fa-cce6-4a67-97c7-88982d036ac8-istio-data\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk\" (UID: \"9f0665fa-cce6-4a67-97c7-88982d036ac8\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" Oct 30 13:10:35.851336 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:35.851274 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f0665fa-cce6-4a67-97c7-88982d036ac8-istio-envoy\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk\" (UID: \"9f0665fa-cce6-4a67-97c7-88982d036ac8\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" Oct 30 13:10:35.851336 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:35.851306 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f0665fa-cce6-4a67-97c7-88982d036ac8-istiod-ca-cert\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk\" (UID: \"9f0665fa-cce6-4a67-97c7-88982d036ac8\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" Oct 30 13:10:35.851489 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:35.851340 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f0665fa-cce6-4a67-97c7-88982d036ac8-istio-podinfo\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk\" (UID: \"9f0665fa-cce6-4a67-97c7-88982d036ac8\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" Oct 30 13:10:35.851547 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:35.851491 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kt7b\" (UniqueName: \"kubernetes.io/projected/9f0665fa-cce6-4a67-97c7-88982d036ac8-kube-api-access-8kt7b\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk\" (UID: \"9f0665fa-cce6-4a67-97c7-88982d036ac8\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" Oct 30 13:10:35.851607 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:35.851569 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f0665fa-cce6-4a67-97c7-88982d036ac8-istio-token\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk\" (UID: \"9f0665fa-cce6-4a67-97c7-88982d036ac8\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" Oct 30 13:10:35.851684 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:35.851624 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f0665fa-cce6-4a67-97c7-88982d036ac8-workload-socket\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk\" (UID: \"9f0665fa-cce6-4a67-97c7-88982d036ac8\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" Oct 30 13:10:35.851684 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:35.851669 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f0665fa-cce6-4a67-97c7-88982d036ac8-workload-certs\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk\" (UID: \"9f0665fa-cce6-4a67-97c7-88982d036ac8\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" Oct 30 13:10:35.851783 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:35.851702 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f0665fa-cce6-4a67-97c7-88982d036ac8-credential-socket\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk\" (UID: \"9f0665fa-cce6-4a67-97c7-88982d036ac8\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" Oct 30 13:10:35.851783 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:35.851709 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f0665fa-cce6-4a67-97c7-88982d036ac8-istio-data\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk\" (UID: \"9f0665fa-cce6-4a67-97c7-88982d036ac8\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" Oct 30 13:10:35.852011 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:35.851969 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f0665fa-cce6-4a67-97c7-88982d036ac8-workload-socket\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk\" (UID: \"9f0665fa-cce6-4a67-97c7-88982d036ac8\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" Oct 30 13:10:35.852105 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:35.852016 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f0665fa-cce6-4a67-97c7-88982d036ac8-workload-certs\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk\" (UID: \"9f0665fa-cce6-4a67-97c7-88982d036ac8\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" Oct 30 13:10:35.852105 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:35.852067 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f0665fa-cce6-4a67-97c7-88982d036ac8-istiod-ca-cert\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk\" (UID: \"9f0665fa-cce6-4a67-97c7-88982d036ac8\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" Oct 30 13:10:35.854333 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:35.854299 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f0665fa-cce6-4a67-97c7-88982d036ac8-istio-envoy\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk\" (UID: \"9f0665fa-cce6-4a67-97c7-88982d036ac8\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" Oct 30 13:10:35.854471 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:35.854447 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f0665fa-cce6-4a67-97c7-88982d036ac8-istio-podinfo\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk\" (UID: \"9f0665fa-cce6-4a67-97c7-88982d036ac8\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" Oct 30 13:10:35.878529 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:35.878500 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f0665fa-cce6-4a67-97c7-88982d036ac8-istio-token\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk\" (UID: \"9f0665fa-cce6-4a67-97c7-88982d036ac8\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" Oct 30 13:10:35.878874 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:35.878853 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kt7b\" (UniqueName: \"kubernetes.io/projected/9f0665fa-cce6-4a67-97c7-88982d036ac8-kube-api-access-8kt7b\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk\" (UID: \"9f0665fa-cce6-4a67-97c7-88982d036ac8\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" Oct 30 13:10:35.972353 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:35.972322 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:35.973065 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:35.973044 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" Oct 30 13:10:36.313245 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:36.313203 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk"] Oct 30 13:10:36.313378 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:10:36.313358 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f0665fa_cce6_4a67_97c7_88982d036ac8.slice/crio-343bcb54b492d93b9735f8283d21f5a38ebb026b5c14d7b8941f04eb533a4962 WatchSource:0}: Error finding container 343bcb54b492d93b9735f8283d21f5a38ebb026b5c14d7b8941f04eb533a4962: Status 404 returned error can't find the container with id 343bcb54b492d93b9735f8283d21f5a38ebb026b5c14d7b8941f04eb533a4962 Oct 30 13:10:36.315908 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:36.315872 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:10:36.316002 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:36.315937 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:10:36.316002 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:36.315966 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:10:36.379189 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:36.379153 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-253f7c37-9ngjf"] Oct 30 13:10:36.385843 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:36.385824 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-253f7c37-9ngjf" Oct 30 13:10:36.401652 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:36.401627 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-253f7c37-9ngjf"] Oct 30 13:10:36.407924 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:36.407892 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:10:36.407924 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:36.407892 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:10:36.408104 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:36.407920 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:10:36.408104 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:36.407970 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:10:36.463020 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:36.462980 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxdfc\" (UniqueName: \"kubernetes.io/projected/564dfb19-1d58-44c9-9836-6c7d1bf80bad-kube-api-access-cxdfc\") pod \"maas-api-revoke-keys-e2e-ait-253f7c37-9ngjf\" (UID: \"564dfb19-1d58-44c9-9836-6c7d1bf80bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-253f7c37-9ngjf" Oct 30 13:10:36.463196 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:36.463139 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/564dfb19-1d58-44c9-9836-6c7d1bf80bad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-253f7c37-9ngjf\" (UID: \"564dfb19-1d58-44c9-9836-6c7d1bf80bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-253f7c37-9ngjf" Oct 30 13:10:36.463326 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:36.463309 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/564dfb19-1d58-44c9-9836-6c7d1bf80bad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-253f7c37-9ngjf\" (UID: \"564dfb19-1d58-44c9-9836-6c7d1bf80bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-253f7c37-9ngjf" Oct 30 13:10:36.565218 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:36.565138 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/564dfb19-1d58-44c9-9836-6c7d1bf80bad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-253f7c37-9ngjf\" (UID: \"564dfb19-1d58-44c9-9836-6c7d1bf80bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-253f7c37-9ngjf" Oct 30 13:10:36.565384 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:36.565303 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/564dfb19-1d58-44c9-9836-6c7d1bf80bad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-253f7c37-9ngjf\" (UID: \"564dfb19-1d58-44c9-9836-6c7d1bf80bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-253f7c37-9ngjf" Oct 30 13:10:36.565484 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:36.565467 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxdfc\" (UniqueName: \"kubernetes.io/projected/564dfb19-1d58-44c9-9836-6c7d1bf80bad-kube-api-access-cxdfc\") pod \"maas-api-revoke-keys-e2e-ait-253f7c37-9ngjf\" (UID: \"564dfb19-1d58-44c9-9836-6c7d1bf80bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-253f7c37-9ngjf" Oct 30 13:10:36.565943 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:36.565923 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/564dfb19-1d58-44c9-9836-6c7d1bf80bad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-253f7c37-9ngjf\" (UID: \"564dfb19-1d58-44c9-9836-6c7d1bf80bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-253f7c37-9ngjf" Oct 30 13:10:36.567398 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:36.567380 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/564dfb19-1d58-44c9-9836-6c7d1bf80bad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-253f7c37-9ngjf\" (UID: \"564dfb19-1d58-44c9-9836-6c7d1bf80bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-253f7c37-9ngjf" Oct 30 13:10:36.601350 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:36.601323 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxdfc\" (UniqueName: \"kubernetes.io/projected/564dfb19-1d58-44c9-9836-6c7d1bf80bad-kube-api-access-cxdfc\") pod \"maas-api-revoke-keys-e2e-ait-253f7c37-9ngjf\" (UID: \"564dfb19-1d58-44c9-9836-6c7d1bf80bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-253f7c37-9ngjf" Oct 30 13:10:36.703160 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:36.703128 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-253f7c37-9ngjf" Oct 30 13:10:36.800018 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:36.799794 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:37.000413 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:37.000381 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:37.002113 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:37.002086 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" event={"ID":"9f0665fa-cce6-4a67-97c7-88982d036ac8","Type":"ContainerStarted","Data":"02e0c93506f7ac881565bd42ac37a7ba949eb12d9e29e0795dd0340f8c79a603"} Oct 30 13:10:37.002232 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:37.002142 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" event={"ID":"9f0665fa-cce6-4a67-97c7-88982d036ac8","Type":"ContainerStarted","Data":"343bcb54b492d93b9735f8283d21f5a38ebb026b5c14d7b8941f04eb533a4962"} Oct 30 13:10:37.037285 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:10:37.037262 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod564dfb19_1d58_44c9_9836_6c7d1bf80bad.slice/crio-d54f335e151106519fb9dce0acc3b2a9e44fb4fe65b2e4e4ea23834e44e80714 WatchSource:0}: Error finding container d54f335e151106519fb9dce0acc3b2a9e44fb4fe65b2e4e4ea23834e44e80714: Status 404 returned error can't find the container with id d54f335e151106519fb9dce0acc3b2a9e44fb4fe65b2e4e4ea23834e44e80714 Oct 30 13:10:37.037429 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:37.037414 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-253f7c37-9ngjf"] Oct 30 13:10:37.052684 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:37.052642 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" podStartSLOduration=2.052628563 podStartE2EDuration="2.052628563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:37.047481909 +0000 UTC m=+1093.516897683" watchObservedRunningTime="2026-10-30 13:10:37.052628563 +0000 UTC m=+1093.522044305" Oct 30 13:10:37.919561 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:37.919526 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:37.987235 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:37.987195 5922 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 13:10:37.987415 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:37.987259 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" podUID="9f0665fa-cce6-4a67-97c7-88982d036ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 13:10:37.987534 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:37.987515 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" Oct 30 13:10:38.015150 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:38.015109 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-253f7c37-9ngjf" event={"ID":"564dfb19-1d58-44c9-9836-6c7d1bf80bad","Type":"ContainerStarted","Data":"eda2bdd0c668e8aeed3acf0f014ad1a65c5d4172ae9f6e745c2334db368cac84"} Oct 30 13:10:38.015300 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:38.015179 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-253f7c37-9ngjf" event={"ID":"564dfb19-1d58-44c9-9836-6c7d1bf80bad","Type":"ContainerStarted","Data":"d54f335e151106519fb9dce0acc3b2a9e44fb4fe65b2e4e4ea23834e44e80714"} Oct 30 13:10:38.038181 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:38.038152 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:38.059524 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:38.059485 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-253f7c37-9ngjf" podStartSLOduration=2.059471317 podStartE2EDuration="2.059471317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:38.05672906 +0000 UTC m=+1094.526144863" watchObservedRunningTime="2026-10-30 13:10:38.059471317 +0000 UTC m=+1094.528887087" Oct 30 13:10:38.097131 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:38.097103 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0fd5fcba-688fdb5fb7-m2s7t"] Oct 30 13:10:38.101928 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:38.101908 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0fd5fcba-688fdb5fb7-m2s7t" Oct 30 13:10:38.122249 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:38.122221 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-0fd5fcba-dockercfg-kc4mg\"" Oct 30 13:10:38.124770 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:38.124739 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0fd5fcba-688fdb5fb7-m2s7t"] Oct 30 13:10:38.133015 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:38.132994 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-0fd5fcba\"" Oct 30 13:10:38.261532 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:38.256899 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd060f1-18db-4577-b842-09a746874192-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-0fd5fcba-688fdb5fb7-m2s7t\" (UID: \"4cd060f1-18db-4577-b842-09a746874192\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0fd5fcba-688fdb5fb7-m2s7t" Oct 30 13:10:38.261532 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:38.257131 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58dgf\" (UniqueName: \"kubernetes.io/projected/4cd060f1-18db-4577-b842-09a746874192-kube-api-access-58dgf\") pod \"payload-pre-processing-e2e-migrate-0fd5fcba-688fdb5fb7-m2s7t\" (UID: \"4cd060f1-18db-4577-b842-09a746874192\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0fd5fcba-688fdb5fb7-m2s7t" Oct 30 13:10:38.287831 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:38.287023 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0fd5fcba-7b5c755894-dh5cw"] Oct 30 13:10:38.362648 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:38.358952 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0fd5fcba-7b5c755894-dh5cw"] Oct 30 13:10:38.362648 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:38.360020 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58dgf\" (UniqueName: \"kubernetes.io/projected/4cd060f1-18db-4577-b842-09a746874192-kube-api-access-58dgf\") pod \"payload-pre-processing-e2e-migrate-0fd5fcba-688fdb5fb7-m2s7t\" (UID: \"4cd060f1-18db-4577-b842-09a746874192\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0fd5fcba-688fdb5fb7-m2s7t" Oct 30 13:10:38.362648 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:38.360180 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd060f1-18db-4577-b842-09a746874192-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-0fd5fcba-688fdb5fb7-m2s7t\" (UID: \"4cd060f1-18db-4577-b842-09a746874192\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0fd5fcba-688fdb5fb7-m2s7t" Oct 30 13:10:38.362648 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:38.361284 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd060f1-18db-4577-b842-09a746874192-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-0fd5fcba-688fdb5fb7-m2s7t\" (UID: \"4cd060f1-18db-4577-b842-09a746874192\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0fd5fcba-688fdb5fb7-m2s7t" Oct 30 13:10:38.362648 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:38.361394 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-0fd5fcba-7b5c755894-dh5cw" Oct 30 13:10:38.419672 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:38.419581 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58dgf\" (UniqueName: \"kubernetes.io/projected/4cd060f1-18db-4577-b842-09a746874192-kube-api-access-58dgf\") pod \"payload-pre-processing-e2e-migrate-0fd5fcba-688fdb5fb7-m2s7t\" (UID: \"4cd060f1-18db-4577-b842-09a746874192\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0fd5fcba-688fdb5fb7-m2s7t" Oct 30 13:10:38.430873 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:38.430845 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0fd5fcba-688fdb5fb7-m2s7t" Oct 30 13:10:38.465653 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:38.462057 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ead3af3-ce7e-40db-988e-d2bcd10fa01f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-0fd5fcba-7b5c755894-dh5cw\" (UID: \"8ead3af3-ce7e-40db-988e-d2bcd10fa01f\") " pod="openshift-ingress/payload-processing-e2e-migrate-0fd5fcba-7b5c755894-dh5cw" Oct 30 13:10:38.465653 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:38.462440 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlxtq\" (UniqueName: \"kubernetes.io/projected/8ead3af3-ce7e-40db-988e-d2bcd10fa01f-kube-api-access-nlxtq\") pod \"payload-processing-e2e-migrate-0fd5fcba-7b5c755894-dh5cw\" (UID: \"8ead3af3-ce7e-40db-988e-d2bcd10fa01f\") " pod="openshift-ingress/payload-processing-e2e-migrate-0fd5fcba-7b5c755894-dh5cw" Oct 30 13:10:38.620758 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:38.617674 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlxtq\" (UniqueName: \"kubernetes.io/projected/8ead3af3-ce7e-40db-988e-d2bcd10fa01f-kube-api-access-nlxtq\") pod \"payload-processing-e2e-migrate-0fd5fcba-7b5c755894-dh5cw\" (UID: \"8ead3af3-ce7e-40db-988e-d2bcd10fa01f\") " pod="openshift-ingress/payload-processing-e2e-migrate-0fd5fcba-7b5c755894-dh5cw" Oct 30 13:10:38.620758 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:38.617910 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ead3af3-ce7e-40db-988e-d2bcd10fa01f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-0fd5fcba-7b5c755894-dh5cw\" (UID: \"8ead3af3-ce7e-40db-988e-d2bcd10fa01f\") " pod="openshift-ingress/payload-processing-e2e-migrate-0fd5fcba-7b5c755894-dh5cw" Oct 30 13:10:38.620758 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:38.619077 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ead3af3-ce7e-40db-988e-d2bcd10fa01f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-0fd5fcba-7b5c755894-dh5cw\" (UID: \"8ead3af3-ce7e-40db-988e-d2bcd10fa01f\") " pod="openshift-ingress/payload-processing-e2e-migrate-0fd5fcba-7b5c755894-dh5cw" Oct 30 13:10:38.647854 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:38.647815 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlxtq\" (UniqueName: \"kubernetes.io/projected/8ead3af3-ce7e-40db-988e-d2bcd10fa01f-kube-api-access-nlxtq\") pod \"payload-processing-e2e-migrate-0fd5fcba-7b5c755894-dh5cw\" (UID: \"8ead3af3-ce7e-40db-988e-d2bcd10fa01f\") " pod="openshift-ingress/payload-processing-e2e-migrate-0fd5fcba-7b5c755894-dh5cw" Oct 30 13:10:38.720749 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:38.720710 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-0fd5fcba-7b5c755894-dh5cw" Oct 30 13:10:38.798083 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:10:38.797892 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cd060f1_18db_4577_b842_09a746874192.slice/crio-a1f2b6c853abcf96323035fbd196edf96c42716f1340e2b1e1c93823ff748044 WatchSource:0}: Error finding container a1f2b6c853abcf96323035fbd196edf96c42716f1340e2b1e1c93823ff748044: Status 404 returned error can't find the container with id a1f2b6c853abcf96323035fbd196edf96c42716f1340e2b1e1c93823ff748044 Oct 30 13:10:38.799663 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:38.798995 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0fd5fcba-688fdb5fb7-m2s7t"] Oct 30 13:10:38.976556 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:38.975090 5922 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 13:10:38.976556 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:38.976161 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" podUID="9f0665fa-cce6-4a67-97c7-88982d036ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 13:10:39.030024 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:39.029992 5922 generic.go:356] "Generic (PLEG): container finished" podID="564dfb19-1d58-44c9-9836-6c7d1bf80bad" containerID="eda2bdd0c668e8aeed3acf0f014ad1a65c5d4172ae9f6e745c2334db368cac84" exitCode=0 Oct 30 13:10:39.030158 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:39.030074 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-253f7c37-9ngjf" event={"ID":"564dfb19-1d58-44c9-9836-6c7d1bf80bad","Type":"ContainerDied","Data":"eda2bdd0c668e8aeed3acf0f014ad1a65c5d4172ae9f6e745c2334db368cac84"} Oct 30 13:10:39.041987 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:39.041400 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0fd5fcba-688fdb5fb7-m2s7t" event={"ID":"4cd060f1-18db-4577-b842-09a746874192","Type":"ContainerStarted","Data":"9950d3b2c01caa7558fa7ba5f65dc5c10fd781dd9ae65d5ad52c8f2197f6bb47"} Oct 30 13:10:39.041987 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:39.041456 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0fd5fcba-688fdb5fb7-m2s7t" event={"ID":"4cd060f1-18db-4577-b842-09a746874192","Type":"ContainerStarted","Data":"a1f2b6c853abcf96323035fbd196edf96c42716f1340e2b1e1c93823ff748044"} Oct 30 13:10:39.041987 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:39.041492 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0fd5fcba-688fdb5fb7-m2s7t" Oct 30 13:10:39.097907 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:39.097811 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0fd5fcba-7b5c755894-dh5cw"] Oct 30 13:10:39.144676 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:39.140865 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0fd5fcba-688fdb5fb7-m2s7t" podStartSLOduration=1.140848697 podStartE2EDuration="1.140848697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:39.134270974 +0000 UTC m=+1095.603686739" watchObservedRunningTime="2026-10-30 13:10:39.140848697 +0000 UTC m=+1095.610264443" Oct 30 13:10:39.217160 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:39.217128 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:39.513666 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:39.513629 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea4b8cb1-tl947"] Oct 30 13:10:39.520778 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:39.519948 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea4b8cb1-tl947" Oct 30 13:10:39.540073 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:39.540048 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwmc6\" (UniqueName: \"kubernetes.io/projected/1f3c3956-902f-46f0-be07-e4d57765ff4a-kube-api-access-hwmc6\") pod \"maas-api-revoke-sub-ea4b8cb1-tl947\" (UID: \"1f3c3956-902f-46f0-be07-e4d57765ff4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea4b8cb1-tl947" Oct 30 13:10:39.540303 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:39.540287 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f3c3956-902f-46f0-be07-e4d57765ff4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea4b8cb1-tl947\" (UID: \"1f3c3956-902f-46f0-be07-e4d57765ff4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea4b8cb1-tl947" Oct 30 13:10:39.540552 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:39.540534 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f3c3956-902f-46f0-be07-e4d57765ff4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea4b8cb1-tl947\" (UID: \"1f3c3956-902f-46f0-be07-e4d57765ff4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea4b8cb1-tl947" Oct 30 13:10:39.545450 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:39.545374 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea4b8cb1-tl947"] Oct 30 13:10:39.598465 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:39.598417 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:39.641780 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:39.641750 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f3c3956-902f-46f0-be07-e4d57765ff4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea4b8cb1-tl947\" (UID: \"1f3c3956-902f-46f0-be07-e4d57765ff4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea4b8cb1-tl947" Oct 30 13:10:39.641918 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:39.641897 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwmc6\" (UniqueName: \"kubernetes.io/projected/1f3c3956-902f-46f0-be07-e4d57765ff4a-kube-api-access-hwmc6\") pod \"maas-api-revoke-sub-ea4b8cb1-tl947\" (UID: \"1f3c3956-902f-46f0-be07-e4d57765ff4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea4b8cb1-tl947" Oct 30 13:10:39.641999 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:39.641985 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f3c3956-902f-46f0-be07-e4d57765ff4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea4b8cb1-tl947\" (UID: \"1f3c3956-902f-46f0-be07-e4d57765ff4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea4b8cb1-tl947" Oct 30 13:10:39.642630 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:39.642591 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f3c3956-902f-46f0-be07-e4d57765ff4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea4b8cb1-tl947\" (UID: \"1f3c3956-902f-46f0-be07-e4d57765ff4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea4b8cb1-tl947" Oct 30 13:10:39.644236 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:39.644209 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f3c3956-902f-46f0-be07-e4d57765ff4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea4b8cb1-tl947\" (UID: \"1f3c3956-902f-46f0-be07-e4d57765ff4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea4b8cb1-tl947" Oct 30 13:10:39.689896 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:39.689864 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwmc6\" (UniqueName: \"kubernetes.io/projected/1f3c3956-902f-46f0-be07-e4d57765ff4a-kube-api-access-hwmc6\") pod \"maas-api-revoke-sub-ea4b8cb1-tl947\" (UID: \"1f3c3956-902f-46f0-be07-e4d57765ff4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea4b8cb1-tl947" Oct 30 13:10:39.793910 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:39.793827 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42ee9d-6trfh"] Oct 30 13:10:39.798571 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:39.798549 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42ee9d-6trfh" Oct 30 13:10:39.809560 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:39.809539 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42ee9d-6trfh"] Oct 30 13:10:39.844122 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:39.844086 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea4b8cb1-tl947" Oct 30 13:10:39.845923 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:39.845897 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-42ee9d-6trfh\" (UID: \"acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42ee9d-6trfh" Oct 30 13:10:39.846035 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:39.846017 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-42ee9d-6trfh\" (UID: \"acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42ee9d-6trfh" Oct 30 13:10:39.846151 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:39.846132 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkzjp\" (UniqueName: \"kubernetes.io/projected/acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4-kube-api-access-tkzjp\") pod \"maas-api-revoke-keys-e2e-worker-w0-42ee9d-6trfh\" (UID: \"acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42ee9d-6trfh" Oct 30 13:10:39.947656 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:39.947459 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-42ee9d-6trfh\" (UID: \"acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42ee9d-6trfh" Oct 30 13:10:39.947656 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:39.947571 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-42ee9d-6trfh\" (UID: \"acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42ee9d-6trfh" Oct 30 13:10:39.948015 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:39.947995 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkzjp\" (UniqueName: \"kubernetes.io/projected/acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4-kube-api-access-tkzjp\") pod \"maas-api-revoke-keys-e2e-worker-w0-42ee9d-6trfh\" (UID: \"acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42ee9d-6trfh" Oct 30 13:10:39.951044 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:39.948146 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-42ee9d-6trfh\" (UID: \"acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42ee9d-6trfh" Oct 30 13:10:39.953777 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:39.953726 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-42ee9d-6trfh\" (UID: \"acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42ee9d-6trfh" Oct 30 13:10:40.002633 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:39.979980 5922 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 13:10:40.002633 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:39.980034 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" podUID="9f0665fa-cce6-4a67-97c7-88982d036ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 13:10:40.002633 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:39.998132 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkzjp\" (UniqueName: \"kubernetes.io/projected/acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4-kube-api-access-tkzjp\") pod \"maas-api-revoke-keys-e2e-worker-w0-42ee9d-6trfh\" (UID: \"acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42ee9d-6trfh" Oct 30 13:10:40.051161 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:40.051019 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-0fd5fcba-7b5c755894-dh5cw" event={"ID":"8ead3af3-ce7e-40db-988e-d2bcd10fa01f","Type":"ContainerStarted","Data":"ff770e3fee9d14b64762865721f17a4a98e6fe0e658b040aff28317c4c998776"} Oct 30 13:10:40.051655 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:40.051589 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-0fd5fcba-7b5c755894-dh5cw" Oct 30 13:10:40.051778 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:40.051669 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-0fd5fcba-7b5c755894-dh5cw" event={"ID":"8ead3af3-ce7e-40db-988e-d2bcd10fa01f","Type":"ContainerStarted","Data":"da02b96f70fec787640dc9efa3916169f7e132b0eef814d5b1a691fe6ad0905e"} Oct 30 13:10:40.118173 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:40.118144 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42ee9d-6trfh" Oct 30 13:10:40.123022 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:40.122968 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-0fd5fcba-7b5c755894-dh5cw" podStartSLOduration=2.122951716 podStartE2EDuration="2.122951716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:40.116806169 +0000 UTC m=+1096.586221937" watchObservedRunningTime="2026-10-30 13:10:40.122951716 +0000 UTC m=+1096.592367462" Oct 30 13:10:40.257447 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:40.257383 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea4b8cb1-tl947"] Oct 30 13:10:40.328159 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:40.328133 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-253f7c37-9ngjf" Oct 30 13:10:40.406652 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:40.366153 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/564dfb19-1d58-44c9-9836-6c7d1bf80bad-kube-api-access-cxdfc\" (UniqueName: \"kubernetes.io/projected/564dfb19-1d58-44c9-9836-6c7d1bf80bad-kube-api-access-cxdfc\") pod \"564dfb19-1d58-44c9-9836-6c7d1bf80bad\" (UID: \"564dfb19-1d58-44c9-9836-6c7d1bf80bad\") " Oct 30 13:10:40.406652 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:40.366339 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/564dfb19-1d58-44c9-9836-6c7d1bf80bad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/564dfb19-1d58-44c9-9836-6c7d1bf80bad-maas-api-openshift-ca\") pod \"564dfb19-1d58-44c9-9836-6c7d1bf80bad\" (UID: \"564dfb19-1d58-44c9-9836-6c7d1bf80bad\") " Oct 30 13:10:40.406652 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:40.366584 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/564dfb19-1d58-44c9-9836-6c7d1bf80bad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/564dfb19-1d58-44c9-9836-6c7d1bf80bad-maas-api-xks-ca\") pod \"564dfb19-1d58-44c9-9836-6c7d1bf80bad\" (UID: \"564dfb19-1d58-44c9-9836-6c7d1bf80bad\") " Oct 30 13:10:40.409647 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:40.407888 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/564dfb19-1d58-44c9-9836-6c7d1bf80bad-maas-api-openshift-ca" pod "564dfb19-1d58-44c9-9836-6c7d1bf80bad" (UID: "564dfb19-1d58-44c9-9836-6c7d1bf80bad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:10:40.409647 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:40.408291 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/564dfb19-1d58-44c9-9836-6c7d1bf80bad-maas-api-xks-ca" pod "564dfb19-1d58-44c9-9836-6c7d1bf80bad" (UID: "564dfb19-1d58-44c9-9836-6c7d1bf80bad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:10:40.419747 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:40.419677 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/564dfb19-1d58-44c9-9836-6c7d1bf80bad-kube-api-access-cxdfc" pod "564dfb19-1d58-44c9-9836-6c7d1bf80bad" (UID: "564dfb19-1d58-44c9-9836-6c7d1bf80bad"). InnerVolumeSpecName "kube-api-access-cxdfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:40.487807 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:40.487776 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/564dfb19-1d58-44c9-9836-6c7d1bf80bad-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:10:40.488036 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:40.488019 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxdfc\" (UniqueName: \"kubernetes.io/projected/564dfb19-1d58-44c9-9836-6c7d1bf80bad-kube-api-access-cxdfc\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:10:40.488150 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:40.488138 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/564dfb19-1d58-44c9-9836-6c7d1bf80bad-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:10:40.554498 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:40.554416 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42ee9d-6trfh"] Oct 30 13:10:40.590390 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:40.590255 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:40.992155 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:40.988703 5922 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 13:10:40.992155 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:40.988762 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" podUID="9f0665fa-cce6-4a67-97c7-88982d036ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 13:10:41.073896 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:41.072712 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42ee9d-6trfh" event={"ID":"acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4","Type":"ContainerStarted","Data":"fa7691e1fd67c1f691497b2362b3c83a82437e03ff09c0f9901b99c1e5d74fb6"} Oct 30 13:10:41.073896 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:41.072783 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42ee9d-6trfh" event={"ID":"acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4","Type":"ContainerStarted","Data":"6a00840c819609bb525ce03be7cefe128ab64b88fab386eb3c2ec65e41b6d0de"} Oct 30 13:10:41.077787 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:41.076951 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea4b8cb1-tl947" event={"ID":"1f3c3956-902f-46f0-be07-e4d57765ff4a","Type":"ContainerStarted","Data":"74cdbdcd347bc2725a25d46db6364ec9246906ae40e4dc19fd5144ad6109c65a"} Oct 30 13:10:41.077787 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:41.077003 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea4b8cb1-tl947" event={"ID":"1f3c3956-902f-46f0-be07-e4d57765ff4a","Type":"ContainerStarted","Data":"eeb7cdc92be3669f4d574ff76e8cc22334f98455babd02a7f0fc2fe56635b61a"} Oct 30 13:10:41.083811 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:41.080737 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-253f7c37-9ngjf" Oct 30 13:10:41.084929 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:41.084517 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-253f7c37-9ngjf" event={"ID":"564dfb19-1d58-44c9-9836-6c7d1bf80bad","Type":"ContainerDied","Data":"d54f335e151106519fb9dce0acc3b2a9e44fb4fe65b2e4e4ea23834e44e80714"} Oct 30 13:10:41.084929 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:41.084565 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d54f335e151106519fb9dce0acc3b2a9e44fb4fe65b2e4e4ea23834e44e80714" Oct 30 13:10:41.121168 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:41.121118 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42ee9d-6trfh" podStartSLOduration=2.121103582 podStartE2EDuration="2.121103582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:41.117163671 +0000 UTC m=+1097.586579443" watchObservedRunningTime="2026-10-30 13:10:41.121103582 +0000 UTC m=+1097.590519366" Oct 30 13:10:41.182672 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:41.180889 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea4b8cb1-tl947" podStartSLOduration=2.180867276 podStartE2EDuration="2.180867276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:41.179045967 +0000 UTC m=+1097.648461740" watchObservedRunningTime="2026-10-30 13:10:41.180867276 +0000 UTC m=+1097.650283022" Oct 30 13:10:41.852487 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:41.852422 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:41.981683 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:41.981414 5922 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 13:10:41.981683 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:41.981484 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" podUID="9f0665fa-cce6-4a67-97c7-88982d036ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 13:10:42.109807 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:42.109191 5922 generic.go:356] "Generic (PLEG): container finished" podID="acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4" containerID="fa7691e1fd67c1f691497b2362b3c83a82437e03ff09c0f9901b99c1e5d74fb6" exitCode=0 Oct 30 13:10:42.109807 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:42.109368 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42ee9d-6trfh" event={"ID":"acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4","Type":"ContainerDied","Data":"fa7691e1fd67c1f691497b2362b3c83a82437e03ff09c0f9901b99c1e5d74fb6"} Oct 30 13:10:42.111670 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:42.111641 5922 generic.go:356] "Generic (PLEG): container finished" podID="1f3c3956-902f-46f0-be07-e4d57765ff4a" containerID="74cdbdcd347bc2725a25d46db6364ec9246906ae40e4dc19fd5144ad6109c65a" exitCode=0 Oct 30 13:10:42.111829 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:42.111671 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea4b8cb1-tl947" event={"ID":"1f3c3956-902f-46f0-be07-e4d57765ff4a","Type":"ContainerDied","Data":"74cdbdcd347bc2725a25d46db6364ec9246906ae40e4dc19fd5144ad6109c65a"} Oct 30 13:10:42.974024 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:42.973993 5922 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 13:10:42.974212 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:42.974040 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" podUID="9f0665fa-cce6-4a67-97c7-88982d036ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 13:10:43.162317 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:43.162271 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:43.316826 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:43.316792 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42ee9d-6trfh" Oct 30 13:10:43.326542 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:43.326491 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:43.377017 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:43.376988 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea4b8cb1-tl947" Oct 30 13:10:43.389310 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:43.389277 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4a59cbc1-j79tv"] Oct 30 13:10:43.459217 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:43.459175 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4a59cbc1-j79tv"] Oct 30 13:10:43.464314 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:43.464278 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4-kube-api-access-tkzjp\" (UniqueName: \"kubernetes.io/projected/acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4-kube-api-access-tkzjp\") pod \"acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4\" (UID: \"acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4\") " Oct 30 13:10:43.465897 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:43.465862 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4-maas-api-openshift-ca\") pod \"acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4\" (UID: \"acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4\") " Oct 30 13:10:43.466177 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:43.466156 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4-maas-api-xks-ca\") pod \"acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4\" (UID: \"acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4\") " Oct 30 13:10:43.468587 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:43.468559 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4-maas-api-openshift-ca" pod "acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4" (UID: "acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:10:43.479289 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:43.476179 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:43.479289 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:43.476364 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4-maas-api-xks-ca" pod "acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4" (UID: "acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:10:43.481347 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:43.480682 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4-kube-api-access-tkzjp" pod "acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4" (UID: "acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4"). InnerVolumeSpecName "kube-api-access-tkzjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:43.545116 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:43.545030 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c"] Oct 30 13:10:43.545290 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:43.545266 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" podUID="99d3f2ce-87ad-4849-8976-7c388312e1e7" containerName="istio-proxy" containerID="cri-o://57d0b32569a17bb01048eea9963047f15952754e0f1dedbf7c8c8c9fb8e583af" gracePeriod=30 Oct 30 13:10:43.569667 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:43.569584 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f3c3956-902f-46f0-be07-e4d57765ff4a-kube-api-access-hwmc6\" (UniqueName: \"kubernetes.io/projected/1f3c3956-902f-46f0-be07-e4d57765ff4a-kube-api-access-hwmc6\") pod \"1f3c3956-902f-46f0-be07-e4d57765ff4a\" (UID: \"1f3c3956-902f-46f0-be07-e4d57765ff4a\") " Oct 30 13:10:43.569930 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:43.569907 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f3c3956-902f-46f0-be07-e4d57765ff4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f3c3956-902f-46f0-be07-e4d57765ff4a-maas-api-openshift-ca\") pod \"1f3c3956-902f-46f0-be07-e4d57765ff4a\" (UID: \"1f3c3956-902f-46f0-be07-e4d57765ff4a\") " Oct 30 13:10:43.570046 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:43.570015 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f3c3956-902f-46f0-be07-e4d57765ff4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f3c3956-902f-46f0-be07-e4d57765ff4a-maas-api-xks-ca\") pod \"1f3c3956-902f-46f0-be07-e4d57765ff4a\" (UID: \"1f3c3956-902f-46f0-be07-e4d57765ff4a\") " Oct 30 13:10:43.572043 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:43.571790 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:10:43.572043 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:43.571819 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:10:43.572043 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:43.571835 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkzjp\" (UniqueName: \"kubernetes.io/projected/acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4-kube-api-access-tkzjp\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:10:43.572477 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:43.572264 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f3c3956-902f-46f0-be07-e4d57765ff4a-maas-api-openshift-ca" pod "1f3c3956-902f-46f0-be07-e4d57765ff4a" (UID: "1f3c3956-902f-46f0-be07-e4d57765ff4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:10:43.574045 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:43.574017 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f3c3956-902f-46f0-be07-e4d57765ff4a-kube-api-access-hwmc6" pod "1f3c3956-902f-46f0-be07-e4d57765ff4a" (UID: "1f3c3956-902f-46f0-be07-e4d57765ff4a"). InnerVolumeSpecName "kube-api-access-hwmc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:43.576220 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:43.576193 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f3c3956-902f-46f0-be07-e4d57765ff4a-maas-api-xks-ca" pod "1f3c3956-902f-46f0-be07-e4d57765ff4a" (UID: "1f3c3956-902f-46f0-be07-e4d57765ff4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:10:43.599664 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:43.596409 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:43.649733 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:43.649699 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab24fa9e-2607-497f-abe5-760e5e55c4d6" path="/var/lib/kubelet/pods/ab24fa9e-2607-497f-abe5-760e5e55c4d6/volumes" Oct 30 13:10:43.673475 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:43.673447 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f3c3956-902f-46f0-be07-e4d57765ff4a-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:10:43.673475 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:43.673476 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwmc6\" (UniqueName: \"kubernetes.io/projected/1f3c3956-902f-46f0-be07-e4d57765ff4a-kube-api-access-hwmc6\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:10:43.673635 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:43.673486 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f3c3956-902f-46f0-be07-e4d57765ff4a-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:10:43.978871 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:43.978842 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" Oct 30 13:10:43.979379 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:43.979311 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" Oct 30 13:10:43.980625 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:43.980582 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" Oct 30 13:10:44.134192 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:44.134165 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42ee9d-6trfh" Oct 30 13:10:44.134192 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:44.134182 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42ee9d-6trfh" event={"ID":"acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4","Type":"ContainerDied","Data":"6a00840c819609bb525ce03be7cefe128ab64b88fab386eb3c2ec65e41b6d0de"} Oct 30 13:10:44.134431 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:44.134232 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a00840c819609bb525ce03be7cefe128ab64b88fab386eb3c2ec65e41b6d0de" Oct 30 13:10:44.135957 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:44.135930 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea4b8cb1-tl947" event={"ID":"1f3c3956-902f-46f0-be07-e4d57765ff4a","Type":"ContainerDied","Data":"eeb7cdc92be3669f4d574ff76e8cc22334f98455babd02a7f0fc2fe56635b61a"} Oct 30 13:10:44.136084 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:44.135985 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eeb7cdc92be3669f4d574ff76e8cc22334f98455babd02a7f0fc2fe56635b61a" Oct 30 13:10:44.136084 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:44.135944 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea4b8cb1-tl947" Oct 30 13:10:44.651424 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:44.651345 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:44.657665 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:44.657626 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm"] Oct 30 13:10:44.680370 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:44.680341 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm"] Oct 30 13:10:44.860634 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:44.845105 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:45.000512 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:45.000474 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:45.394786 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:45.394708 5922 patch_prober.go:29] interesting pod/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:10:45.394786 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:45.394769 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" podUID="99d3f2ce-87ad-4849-8976-7c388312e1e7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:10:45.805911 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:45.805875 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:46.501500 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:46.501467 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a976c65-zdf47"] Oct 30 13:10:46.502227 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:46.502209 5922 cpu_manager.go:414] "containerMap: removing container" podUID="1f3c3956-902f-46f0-be07-e4d57765ff4a" containerName="revoke-keys" Oct 30 13:10:46.502287 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:46.502231 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f3c3956-902f-46f0-be07-e4d57765ff4a" containerName="revoke-keys" Oct 30 13:10:46.502287 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:46.502259 5922 cpu_manager.go:414] "containerMap: removing container" podUID="564dfb19-1d58-44c9-9836-6c7d1bf80bad" containerName="revoke-keys" Oct 30 13:10:46.502287 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:46.502264 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="564dfb19-1d58-44c9-9836-6c7d1bf80bad" containerName="revoke-keys" Oct 30 13:10:46.502287 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:46.502272 5922 cpu_manager.go:414] "containerMap: removing container" podUID="acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4" containerName="revoke-keys" Oct 30 13:10:46.502287 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:46.502277 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4" containerName="revoke-keys" Oct 30 13:10:46.502444 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:46.502393 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f3c3956-902f-46f0-be07-e4d57765ff4a" containerName="revoke-keys" Oct 30 13:10:46.502444 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:46.502406 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="564dfb19-1d58-44c9-9836-6c7d1bf80bad" containerName="revoke-keys" Oct 30 13:10:46.502444 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:46.502412 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4" containerName="revoke-keys" Oct 30 13:10:46.507592 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:46.507571 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a976c65-zdf47" Oct 30 13:10:46.521651 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:46.521624 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b7cc9a-3532-4b96-b523-fb5eeb9f6258-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a976c65-zdf47\" (UID: \"d6b7cc9a-3532-4b96-b523-fb5eeb9f6258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a976c65-zdf47" Oct 30 13:10:46.521865 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:46.521839 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b7cc9a-3532-4b96-b523-fb5eeb9f6258-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a976c65-zdf47\" (UID: \"d6b7cc9a-3532-4b96-b523-fb5eeb9f6258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a976c65-zdf47" Oct 30 13:10:46.521957 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:46.521938 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvhxn\" (UniqueName: \"kubernetes.io/projected/d6b7cc9a-3532-4b96-b523-fb5eeb9f6258-kube-api-access-zvhxn\") pod \"maas-api-revoke-sub-6a976c65-zdf47\" (UID: \"d6b7cc9a-3532-4b96-b523-fb5eeb9f6258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a976c65-zdf47" Oct 30 13:10:46.526423 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:46.526398 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a976c65-zdf47"] Oct 30 13:10:46.532647 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:46.532582 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:10:46.535747 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:46.535725 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:10:46.535851 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:46.535748 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:10:46.535851 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:46.535748 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:10:46.623294 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:46.623263 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b7cc9a-3532-4b96-b523-fb5eeb9f6258-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a976c65-zdf47\" (UID: \"d6b7cc9a-3532-4b96-b523-fb5eeb9f6258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a976c65-zdf47" Oct 30 13:10:46.623408 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:46.623309 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvhxn\" (UniqueName: \"kubernetes.io/projected/d6b7cc9a-3532-4b96-b523-fb5eeb9f6258-kube-api-access-zvhxn\") pod \"maas-api-revoke-sub-6a976c65-zdf47\" (UID: \"d6b7cc9a-3532-4b96-b523-fb5eeb9f6258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a976c65-zdf47" Oct 30 13:10:46.623408 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:46.623388 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b7cc9a-3532-4b96-b523-fb5eeb9f6258-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a976c65-zdf47\" (UID: \"d6b7cc9a-3532-4b96-b523-fb5eeb9f6258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a976c65-zdf47" Oct 30 13:10:46.624015 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:46.623992 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b7cc9a-3532-4b96-b523-fb5eeb9f6258-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a976c65-zdf47\" (UID: \"d6b7cc9a-3532-4b96-b523-fb5eeb9f6258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a976c65-zdf47" Oct 30 13:10:46.625713 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:46.625692 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b7cc9a-3532-4b96-b523-fb5eeb9f6258-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a976c65-zdf47\" (UID: \"d6b7cc9a-3532-4b96-b523-fb5eeb9f6258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a976c65-zdf47" Oct 30 13:10:46.648380 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:46.648351 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvhxn\" (UniqueName: \"kubernetes.io/projected/d6b7cc9a-3532-4b96-b523-fb5eeb9f6258-kube-api-access-zvhxn\") pod \"maas-api-revoke-sub-6a976c65-zdf47\" (UID: \"d6b7cc9a-3532-4b96-b523-fb5eeb9f6258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a976c65-zdf47" Oct 30 13:10:46.822420 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:46.822349 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a976c65-zdf47" Oct 30 13:10:47.000866 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:47.000806 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:47.130279 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:47.130188 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:47.161946 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:47.156344 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a976c65-zdf47"] Oct 30 13:10:47.164667 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:10:47.164624 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b7cc9a_3532_4b96_b523_fb5eeb9f6258.slice/crio-1aea4e2e9538ef7eb33530a3348452c4e913dc59c61abe1274201415b374bc82 WatchSource:0}: Error finding container 1aea4e2e9538ef7eb33530a3348452c4e913dc59c61abe1274201415b374bc82: Status 404 returned error can't find the container with id 1aea4e2e9538ef7eb33530a3348452c4e913dc59c61abe1274201415b374bc82 Oct 30 13:10:47.176478 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:47.176449 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a976c65-zdf47" event={"ID":"d6b7cc9a-3532-4b96-b523-fb5eeb9f6258","Type":"ContainerStarted","Data":"1aea4e2e9538ef7eb33530a3348452c4e913dc59c61abe1274201415b374bc82"} Oct 30 13:10:47.732825 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:47.732786 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:48.187337 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:48.187298 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a976c65-zdf47" event={"ID":"d6b7cc9a-3532-4b96-b523-fb5eeb9f6258","Type":"ContainerStarted","Data":"55b97076ea9178de76095090b9db4024adf581c09b608a8c8a704fa7d5e67af9"} Oct 30 13:10:48.230971 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:48.230925 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a976c65-zdf47" podStartSLOduration=2.230911277 podStartE2EDuration="2.230911277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:48.229106787 +0000 UTC m=+1104.698522542" watchObservedRunningTime="2026-10-30 13:10:48.230911277 +0000 UTC m=+1104.700327067" Oct 30 13:10:48.834373 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:48.834313 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" Oct 30 13:10:48.963380 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:48.963346 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99d3f2ce-87ad-4849-8976-7c388312e1e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99d3f2ce-87ad-4849-8976-7c388312e1e7-istiod-ca-cert\") pod \"99d3f2ce-87ad-4849-8976-7c388312e1e7\" (UID: \"99d3f2ce-87ad-4849-8976-7c388312e1e7\") " Oct 30 13:10:48.963551 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:48.963396 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99d3f2ce-87ad-4849-8976-7c388312e1e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99d3f2ce-87ad-4849-8976-7c388312e1e7-istio-data\") pod \"99d3f2ce-87ad-4849-8976-7c388312e1e7\" (UID: \"99d3f2ce-87ad-4849-8976-7c388312e1e7\") " Oct 30 13:10:48.963666 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:48.963554 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99d3f2ce-87ad-4849-8976-7c388312e1e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99d3f2ce-87ad-4849-8976-7c388312e1e7-istio-envoy\") pod \"99d3f2ce-87ad-4849-8976-7c388312e1e7\" (UID: \"99d3f2ce-87ad-4849-8976-7c388312e1e7\") " Oct 30 13:10:48.963666 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:48.963601 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99d3f2ce-87ad-4849-8976-7c388312e1e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99d3f2ce-87ad-4849-8976-7c388312e1e7-istio-podinfo\") pod \"99d3f2ce-87ad-4849-8976-7c388312e1e7\" (UID: \"99d3f2ce-87ad-4849-8976-7c388312e1e7\") " Oct 30 13:10:48.963803 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:48.963728 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99d3f2ce-87ad-4849-8976-7c388312e1e7-kube-api-access-52l7g\" (UniqueName: \"kubernetes.io/projected/99d3f2ce-87ad-4849-8976-7c388312e1e7-kube-api-access-52l7g\") pod \"99d3f2ce-87ad-4849-8976-7c388312e1e7\" (UID: \"99d3f2ce-87ad-4849-8976-7c388312e1e7\") " Oct 30 13:10:48.963803 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:48.963751 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99d3f2ce-87ad-4849-8976-7c388312e1e7-istiod-ca-cert" pod "99d3f2ce-87ad-4849-8976-7c388312e1e7" (UID: "99d3f2ce-87ad-4849-8976-7c388312e1e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:10:48.963803 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:48.963789 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99d3f2ce-87ad-4849-8976-7c388312e1e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99d3f2ce-87ad-4849-8976-7c388312e1e7-credential-socket\") pod \"99d3f2ce-87ad-4849-8976-7c388312e1e7\" (UID: \"99d3f2ce-87ad-4849-8976-7c388312e1e7\") " Oct 30 13:10:48.963965 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:48.963854 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99d3f2ce-87ad-4849-8976-7c388312e1e7-istio-token\" (UniqueName: \"kubernetes.io/projected/99d3f2ce-87ad-4849-8976-7c388312e1e7-istio-token\") pod \"99d3f2ce-87ad-4849-8976-7c388312e1e7\" (UID: \"99d3f2ce-87ad-4849-8976-7c388312e1e7\") " Oct 30 13:10:48.963965 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:48.963916 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99d3f2ce-87ad-4849-8976-7c388312e1e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99d3f2ce-87ad-4849-8976-7c388312e1e7-workload-socket\") pod \"99d3f2ce-87ad-4849-8976-7c388312e1e7\" (UID: \"99d3f2ce-87ad-4849-8976-7c388312e1e7\") " Oct 30 13:10:48.963965 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:48.963956 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99d3f2ce-87ad-4849-8976-7c388312e1e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99d3f2ce-87ad-4849-8976-7c388312e1e7-workload-certs\") pod \"99d3f2ce-87ad-4849-8976-7c388312e1e7\" (UID: \"99d3f2ce-87ad-4849-8976-7c388312e1e7\") " Oct 30 13:10:48.964126 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:48.963981 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d3f2ce-87ad-4849-8976-7c388312e1e7-istio-data" pod "99d3f2ce-87ad-4849-8976-7c388312e1e7" (UID: "99d3f2ce-87ad-4849-8976-7c388312e1e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:10:48.964469 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:48.964217 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d3f2ce-87ad-4849-8976-7c388312e1e7-credential-socket" pod "99d3f2ce-87ad-4849-8976-7c388312e1e7" (UID: "99d3f2ce-87ad-4849-8976-7c388312e1e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:10:48.964469 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:48.964227 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d3f2ce-87ad-4849-8976-7c388312e1e7-workload-socket" pod "99d3f2ce-87ad-4849-8976-7c388312e1e7" (UID: "99d3f2ce-87ad-4849-8976-7c388312e1e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:10:48.964469 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:48.964436 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d3f2ce-87ad-4849-8976-7c388312e1e7-workload-certs" pod "99d3f2ce-87ad-4849-8976-7c388312e1e7" (UID: "99d3f2ce-87ad-4849-8976-7c388312e1e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:10:48.964755 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:48.964732 5922 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99d3f2ce-87ad-4849-8976-7c388312e1e7-credential-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:10:48.964815 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:48.964753 5922 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99d3f2ce-87ad-4849-8976-7c388312e1e7-workload-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:10:48.964815 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:48.964768 5922 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99d3f2ce-87ad-4849-8976-7c388312e1e7-workload-certs\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:10:48.964815 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:48.964782 5922 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99d3f2ce-87ad-4849-8976-7c388312e1e7-istiod-ca-cert\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:10:48.964815 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:48.964795 5922 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99d3f2ce-87ad-4849-8976-7c388312e1e7-istio-data\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:10:48.965999 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:48.965971 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99d3f2ce-87ad-4849-8976-7c388312e1e7-istio-podinfo" pod "99d3f2ce-87ad-4849-8976-7c388312e1e7" (UID: "99d3f2ce-87ad-4849-8976-7c388312e1e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:10:48.966346 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:48.966317 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99d3f2ce-87ad-4849-8976-7c388312e1e7-kube-api-access-52l7g" pod "99d3f2ce-87ad-4849-8976-7c388312e1e7" (UID: "99d3f2ce-87ad-4849-8976-7c388312e1e7"). InnerVolumeSpecName "kube-api-access-52l7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:48.967008 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:48.966977 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99d3f2ce-87ad-4849-8976-7c388312e1e7-istio-token" pod "99d3f2ce-87ad-4849-8976-7c388312e1e7" (UID: "99d3f2ce-87ad-4849-8976-7c388312e1e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:48.967194 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:48.967177 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d3f2ce-87ad-4849-8976-7c388312e1e7-istio-envoy" pod "99d3f2ce-87ad-4849-8976-7c388312e1e7" (UID: "99d3f2ce-87ad-4849-8976-7c388312e1e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:10:49.066648 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:49.066588 5922 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99d3f2ce-87ad-4849-8976-7c388312e1e7-istio-envoy\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:10:49.066782 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:49.066656 5922 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99d3f2ce-87ad-4849-8976-7c388312e1e7-istio-podinfo\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:10:49.066782 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:49.066672 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52l7g\" (UniqueName: \"kubernetes.io/projected/99d3f2ce-87ad-4849-8976-7c388312e1e7-kube-api-access-52l7g\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:10:49.066782 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:49.066686 5922 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99d3f2ce-87ad-4849-8976-7c388312e1e7-istio-token\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:10:49.198730 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:49.198700 5922 generic.go:356] "Generic (PLEG): container finished" podID="d6b7cc9a-3532-4b96-b523-fb5eeb9f6258" containerID="55b97076ea9178de76095090b9db4024adf581c09b608a8c8a704fa7d5e67af9" exitCode=0 Oct 30 13:10:49.199186 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:49.198738 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a976c65-zdf47" event={"ID":"d6b7cc9a-3532-4b96-b523-fb5eeb9f6258","Type":"ContainerDied","Data":"55b97076ea9178de76095090b9db4024adf581c09b608a8c8a704fa7d5e67af9"} Oct 30 13:10:49.200673 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:49.200648 5922 generic.go:356] "Generic (PLEG): container finished" podID="99d3f2ce-87ad-4849-8976-7c388312e1e7" containerID="57d0b32569a17bb01048eea9963047f15952754e0f1dedbf7c8c8c9fb8e583af" exitCode=0 Oct 30 13:10:49.200792 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:49.200691 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" Oct 30 13:10:49.200792 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:49.200725 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" event={"ID":"99d3f2ce-87ad-4849-8976-7c388312e1e7","Type":"ContainerDied","Data":"57d0b32569a17bb01048eea9963047f15952754e0f1dedbf7c8c8c9fb8e583af"} Oct 30 13:10:49.200792 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:49.200773 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c" event={"ID":"99d3f2ce-87ad-4849-8976-7c388312e1e7","Type":"ContainerDied","Data":"b3e05952b1cb5fa72a2ad8157cb7b122b3b9dc218052e40d05fe27232daaf461"} Oct 30 13:10:49.200922 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:49.200803 5922 scope.go:122] "RemoveContainer" containerID="57d0b32569a17bb01048eea9963047f15952754e0f1dedbf7c8c8c9fb8e583af" Oct 30 13:10:49.214124 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:49.214102 5922 scope.go:122] "RemoveContainer" containerID="357040f68e61a06dd187d19aae4a0c6b89155204e8612c471965c0cd3d35c6db" Oct 30 13:10:49.224347 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:49.224322 5922 scope.go:122] "RemoveContainer" containerID="57d0b32569a17bb01048eea9963047f15952754e0f1dedbf7c8c8c9fb8e583af" Oct 30 13:10:49.224647 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:10:49.224589 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57d0b32569a17bb01048eea9963047f15952754e0f1dedbf7c8c8c9fb8e583af\": container with ID starting with 57d0b32569a17bb01048eea9963047f15952754e0f1dedbf7c8c8c9fb8e583af not found: ID does not exist" containerID="57d0b32569a17bb01048eea9963047f15952754e0f1dedbf7c8c8c9fb8e583af" Oct 30 13:10:49.224647 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:49.224630 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57d0b32569a17bb01048eea9963047f15952754e0f1dedbf7c8c8c9fb8e583af"} err="failed to get container status \"57d0b32569a17bb01048eea9963047f15952754e0f1dedbf7c8c8c9fb8e583af\": rpc error: code = NotFound desc = could not find container \"57d0b32569a17bb01048eea9963047f15952754e0f1dedbf7c8c8c9fb8e583af\": container with ID starting with 57d0b32569a17bb01048eea9963047f15952754e0f1dedbf7c8c8c9fb8e583af not found: ID does not exist" Oct 30 13:10:49.224782 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:49.224653 5922 scope.go:122] "RemoveContainer" containerID="357040f68e61a06dd187d19aae4a0c6b89155204e8612c471965c0cd3d35c6db" Oct 30 13:10:49.224927 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:10:49.224906 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"357040f68e61a06dd187d19aae4a0c6b89155204e8612c471965c0cd3d35c6db\": container with ID starting with 357040f68e61a06dd187d19aae4a0c6b89155204e8612c471965c0cd3d35c6db not found: ID does not exist" containerID="357040f68e61a06dd187d19aae4a0c6b89155204e8612c471965c0cd3d35c6db" Oct 30 13:10:49.224966 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:49.224933 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"357040f68e61a06dd187d19aae4a0c6b89155204e8612c471965c0cd3d35c6db"} err="failed to get container status \"357040f68e61a06dd187d19aae4a0c6b89155204e8612c471965c0cd3d35c6db\": rpc error: code = NotFound desc = could not find container \"357040f68e61a06dd187d19aae4a0c6b89155204e8612c471965c0cd3d35c6db\": container with ID starting with 357040f68e61a06dd187d19aae4a0c6b89155204e8612c471965c0cd3d35c6db not found: ID does not exist" Oct 30 13:10:49.294859 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:49.294826 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c"] Oct 30 13:10:49.300271 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:49.300238 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-4a59cbc1-openshift-default-8b766555c-w6k5c"] Oct 30 13:10:49.600519 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:49.600433 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:49.669999 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:49.669963 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99d3f2ce-87ad-4849-8976-7c388312e1e7" path="/var/lib/kubelet/pods/99d3f2ce-87ad-4849-8976-7c388312e1e7/volumes" Oct 30 13:10:49.981725 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:49.980720 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-800c8ce2-ls4mh"] Oct 30 13:10:49.981725 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:49.981625 5922 cpu_manager.go:414] "containerMap: removing container" podUID="99d3f2ce-87ad-4849-8976-7c388312e1e7" containerName="istio-proxy" Oct 30 13:10:49.981725 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:49.981648 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99d3f2ce-87ad-4849-8976-7c388312e1e7" containerName="istio-proxy" Oct 30 13:10:49.982022 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:49.981871 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="99d3f2ce-87ad-4849-8976-7c388312e1e7" containerName="istio-proxy" Oct 30 13:10:49.982022 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:49.981888 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="99d3f2ce-87ad-4849-8976-7c388312e1e7" containerName="istio-proxy" Oct 30 13:10:49.988090 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:49.988068 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-800c8ce2-ls4mh" Oct 30 13:10:50.003104 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:50.003078 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-800c8ce2-ls4mh"] Oct 30 13:10:50.055225 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:50.055201 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0fd5fcba-688fdb5fb7-m2s7t" Oct 30 13:10:50.083237 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:50.083207 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4f503a4-2366-439a-9603-e2e49b8c823a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-800c8ce2-ls4mh\" (UID: \"d4f503a4-2366-439a-9603-e2e49b8c823a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-800c8ce2-ls4mh" Oct 30 13:10:50.083402 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:50.083275 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4f503a4-2366-439a-9603-e2e49b8c823a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-800c8ce2-ls4mh\" (UID: \"d4f503a4-2366-439a-9603-e2e49b8c823a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-800c8ce2-ls4mh" Oct 30 13:10:50.083402 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:50.083381 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2m8w\" (UniqueName: \"kubernetes.io/projected/d4f503a4-2366-439a-9603-e2e49b8c823a-kube-api-access-h2m8w\") pod \"maas-api-revoke-sub-800c8ce2-ls4mh\" (UID: \"d4f503a4-2366-439a-9603-e2e49b8c823a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-800c8ce2-ls4mh" Oct 30 13:10:50.184867 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:50.184837 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4f503a4-2366-439a-9603-e2e49b8c823a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-800c8ce2-ls4mh\" (UID: \"d4f503a4-2366-439a-9603-e2e49b8c823a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-800c8ce2-ls4mh" Oct 30 13:10:50.185039 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:50.184925 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4f503a4-2366-439a-9603-e2e49b8c823a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-800c8ce2-ls4mh\" (UID: \"d4f503a4-2366-439a-9603-e2e49b8c823a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-800c8ce2-ls4mh" Oct 30 13:10:50.185039 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:50.184980 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2m8w\" (UniqueName: \"kubernetes.io/projected/d4f503a4-2366-439a-9603-e2e49b8c823a-kube-api-access-h2m8w\") pod \"maas-api-revoke-sub-800c8ce2-ls4mh\" (UID: \"d4f503a4-2366-439a-9603-e2e49b8c823a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-800c8ce2-ls4mh" Oct 30 13:10:50.185623 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:50.185590 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4f503a4-2366-439a-9603-e2e49b8c823a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-800c8ce2-ls4mh\" (UID: \"d4f503a4-2366-439a-9603-e2e49b8c823a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-800c8ce2-ls4mh" Oct 30 13:10:50.187184 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:50.187161 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4f503a4-2366-439a-9603-e2e49b8c823a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-800c8ce2-ls4mh\" (UID: \"d4f503a4-2366-439a-9603-e2e49b8c823a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-800c8ce2-ls4mh" Oct 30 13:10:50.223042 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:50.223006 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2m8w\" (UniqueName: \"kubernetes.io/projected/d4f503a4-2366-439a-9603-e2e49b8c823a-kube-api-access-h2m8w\") pod \"maas-api-revoke-sub-800c8ce2-ls4mh\" (UID: \"d4f503a4-2366-439a-9603-e2e49b8c823a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-800c8ce2-ls4mh" Oct 30 13:10:50.311934 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:50.311899 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-800c8ce2-ls4mh" Oct 30 13:10:50.313835 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:50.313804 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-42ee9d-5645d7744d-d6f7m"] Oct 30 13:10:50.419631 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:50.418516 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a976c65-zdf47" Oct 30 13:10:50.474155 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:50.470504 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:50.489452 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:50.489417 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6b7cc9a-3532-4b96-b523-fb5eeb9f6258-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b7cc9a-3532-4b96-b523-fb5eeb9f6258-maas-api-xks-ca\") pod \"d6b7cc9a-3532-4b96-b523-fb5eeb9f6258\" (UID: \"d6b7cc9a-3532-4b96-b523-fb5eeb9f6258\") " Oct 30 13:10:50.489627 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:50.489506 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6b7cc9a-3532-4b96-b523-fb5eeb9f6258-kube-api-access-zvhxn\" (UniqueName: \"kubernetes.io/projected/d6b7cc9a-3532-4b96-b523-fb5eeb9f6258-kube-api-access-zvhxn\") pod \"d6b7cc9a-3532-4b96-b523-fb5eeb9f6258\" (UID: \"d6b7cc9a-3532-4b96-b523-fb5eeb9f6258\") " Oct 30 13:10:50.489627 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:50.489597 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6b7cc9a-3532-4b96-b523-fb5eeb9f6258-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b7cc9a-3532-4b96-b523-fb5eeb9f6258-maas-api-openshift-ca\") pod \"d6b7cc9a-3532-4b96-b523-fb5eeb9f6258\" (UID: \"d6b7cc9a-3532-4b96-b523-fb5eeb9f6258\") " Oct 30 13:10:50.490647 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:50.490579 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6b7cc9a-3532-4b96-b523-fb5eeb9f6258-maas-api-openshift-ca" pod "d6b7cc9a-3532-4b96-b523-fb5eeb9f6258" (UID: "d6b7cc9a-3532-4b96-b523-fb5eeb9f6258"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:10:50.492458 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:50.492438 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6b7cc9a-3532-4b96-b523-fb5eeb9f6258-kube-api-access-zvhxn" pod "d6b7cc9a-3532-4b96-b523-fb5eeb9f6258" (UID: "d6b7cc9a-3532-4b96-b523-fb5eeb9f6258"). InnerVolumeSpecName "kube-api-access-zvhxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:50.492729 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:50.492708 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6b7cc9a-3532-4b96-b523-fb5eeb9f6258-maas-api-xks-ca" pod "d6b7cc9a-3532-4b96-b523-fb5eeb9f6258" (UID: "d6b7cc9a-3532-4b96-b523-fb5eeb9f6258"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:10:50.587480 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:50.587324 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:50.591765 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:50.591733 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b7cc9a-3532-4b96-b523-fb5eeb9f6258-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:10:50.591765 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:50.591761 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvhxn\" (UniqueName: \"kubernetes.io/projected/d6b7cc9a-3532-4b96-b523-fb5eeb9f6258-kube-api-access-zvhxn\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:10:50.591968 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:50.591778 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b7cc9a-3532-4b96-b523-fb5eeb9f6258-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:10:50.683488 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:50.683456 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-800c8ce2-ls4mh"] Oct 30 13:10:50.684919 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:10:50.684881 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f503a4_2366_439a_9603_e2e49b8c823a.slice/crio-9abe9e84839862aa364b78a485eae1c542020976fda969159cfbfff7877b9e91 WatchSource:0}: Error finding container 9abe9e84839862aa364b78a485eae1c542020976fda969159cfbfff7877b9e91: Status 404 returned error can't find the container with id 9abe9e84839862aa364b78a485eae1c542020976fda969159cfbfff7877b9e91 Oct 30 13:10:51.091637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:51.090068 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-0fd5fcba-7b5c755894-dh5cw" Oct 30 13:10:51.240271 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:51.240244 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a976c65-zdf47" Oct 30 13:10:51.240736 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:51.240283 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a976c65-zdf47" event={"ID":"d6b7cc9a-3532-4b96-b523-fb5eeb9f6258","Type":"ContainerDied","Data":"1aea4e2e9538ef7eb33530a3348452c4e913dc59c61abe1274201415b374bc82"} Oct 30 13:10:51.240736 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:51.240329 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1aea4e2e9538ef7eb33530a3348452c4e913dc59c61abe1274201415b374bc82" Oct 30 13:10:51.242517 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:51.242490 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-800c8ce2-ls4mh" event={"ID":"d4f503a4-2366-439a-9603-e2e49b8c823a","Type":"ContainerStarted","Data":"e8bfc3c2833e78c45b43ea9dd8b219dcab439fc8196e0e3c966c7c5166b42f77"} Oct 30 13:10:51.242651 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:51.242541 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-800c8ce2-ls4mh" event={"ID":"d4f503a4-2366-439a-9603-e2e49b8c823a","Type":"ContainerStarted","Data":"9abe9e84839862aa364b78a485eae1c542020976fda969159cfbfff7877b9e91"} Oct 30 13:10:51.287867 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:51.287810 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-800c8ce2-ls4mh" podStartSLOduration=2.287797027 podStartE2EDuration="2.287797027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:51.28401748 +0000 UTC m=+1107.753433246" watchObservedRunningTime="2026-10-30 13:10:51.287797027 +0000 UTC m=+1107.757212773" Oct 30 13:10:51.609826 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:51.609732 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:51.718633 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:51.718578 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:52.134310 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:52.134267 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:52.254242 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:52.254210 5922 generic.go:356] "Generic (PLEG): container finished" podID="d4f503a4-2366-439a-9603-e2e49b8c823a" containerID="e8bfc3c2833e78c45b43ea9dd8b219dcab439fc8196e0e3c966c7c5166b42f77" exitCode=0 Oct 30 13:10:52.254655 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:52.254290 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-800c8ce2-ls4mh" event={"ID":"d4f503a4-2366-439a-9603-e2e49b8c823a","Type":"ContainerDied","Data":"e8bfc3c2833e78c45b43ea9dd8b219dcab439fc8196e0e3c966c7c5166b42f77"} Oct 30 13:10:53.451014 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:53.450988 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-800c8ce2-ls4mh" Oct 30 13:10:53.555121 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:10:53.555043 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:10:53.559641 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:53.559601 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4f503a4-2366-439a-9603-e2e49b8c823a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4f503a4-2366-439a-9603-e2e49b8c823a-maas-api-openshift-ca\") pod \"d4f503a4-2366-439a-9603-e2e49b8c823a\" (UID: \"d4f503a4-2366-439a-9603-e2e49b8c823a\") " Oct 30 13:10:53.559922 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:53.559904 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4f503a4-2366-439a-9603-e2e49b8c823a-kube-api-access-h2m8w\" (UniqueName: \"kubernetes.io/projected/d4f503a4-2366-439a-9603-e2e49b8c823a-kube-api-access-h2m8w\") pod \"d4f503a4-2366-439a-9603-e2e49b8c823a\" (UID: \"d4f503a4-2366-439a-9603-e2e49b8c823a\") " Oct 30 13:10:53.559966 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:53.559921 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4f503a4-2366-439a-9603-e2e49b8c823a-maas-api-openshift-ca" pod "d4f503a4-2366-439a-9603-e2e49b8c823a" (UID: "d4f503a4-2366-439a-9603-e2e49b8c823a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:10:53.560037 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:53.560024 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4f503a4-2366-439a-9603-e2e49b8c823a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4f503a4-2366-439a-9603-e2e49b8c823a-maas-api-xks-ca\") pod \"d4f503a4-2366-439a-9603-e2e49b8c823a\" (UID: \"d4f503a4-2366-439a-9603-e2e49b8c823a\") " Oct 30 13:10:53.560700 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:53.560676 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4f503a4-2366-439a-9603-e2e49b8c823a-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:10:53.561903 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:53.561874 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f503a4-2366-439a-9603-e2e49b8c823a-kube-api-access-h2m8w" pod "d4f503a4-2366-439a-9603-e2e49b8c823a" (UID: "d4f503a4-2366-439a-9603-e2e49b8c823a"). InnerVolumeSpecName "kube-api-access-h2m8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:53.562023 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:53.562004 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4f503a4-2366-439a-9603-e2e49b8c823a-maas-api-xks-ca" pod "d4f503a4-2366-439a-9603-e2e49b8c823a" (UID: "d4f503a4-2366-439a-9603-e2e49b8c823a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:10:53.662260 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:53.662239 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4f503a4-2366-439a-9603-e2e49b8c823a-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:10:53.662361 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:53.662262 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2m8w\" (UniqueName: \"kubernetes.io/projected/d4f503a4-2366-439a-9603-e2e49b8c823a-kube-api-access-h2m8w\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:10:53.974325 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:53.974288 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:54.277204 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:54.277114 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-800c8ce2-ls4mh" Oct 30 13:10:54.277204 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:54.277127 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-800c8ce2-ls4mh" event={"ID":"d4f503a4-2366-439a-9603-e2e49b8c823a","Type":"ContainerDied","Data":"9abe9e84839862aa364b78a485eae1c542020976fda969159cfbfff7877b9e91"} Oct 30 13:10:54.277204 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:54.277172 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9abe9e84839862aa364b78a485eae1c542020976fda969159cfbfff7877b9e91" Oct 30 13:10:54.958414 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:54.958380 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:55.205830 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:55.205789 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:55.314686 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:55.314548 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-42ee9d-5645d7744d-d6f7m" podUID="85a299c7-b978-4b75-8aed-fff5da4d32f7" containerName="payload-processing" containerID="cri-o://0d2bf6b0fa8808b75c204fd58adfd598458a476a1b1238b82760ac012441cfb9" gracePeriod=25 Oct 30 13:10:55.343664 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:55.343594 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-42ee9d-5645d7744d-d6f7m" podUID="85a299c7-b978-4b75-8aed-fff5da4d32f7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:10:55.370456 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:55.370425 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-253f7c37-9ngjf"] Oct 30 13:10:55.378653 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:55.378627 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-253f7c37-9ngjf"] Oct 30 13:10:55.655570 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:55.655487 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="564dfb19-1d58-44c9-9836-6c7d1bf80bad" path="/var/lib/kubelet/pods/564dfb19-1d58-44c9-9836-6c7d1bf80bad/volumes" Oct 30 13:10:55.979035 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:55.978998 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:56.932801 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:56.932739 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:57.051394 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:57.051356 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:58.028704 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:58.028667 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:58.187282 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:58.187240 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:58.530646 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:58.530585 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:58.649381 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:58.649343 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:58.770125 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:58.770095 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:59.679087 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:59.679051 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" podUID="a296df26-3a3b-40ef-a3ab-1866ecf7dcbc" containerName="main" containerID="cri-o://ec236ae48eaf54b6aa004cec816f861bef7139ccc86a08636af684361fb347c7" gracePeriod=45 Oct 30 13:10:59.701626 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:59.701589 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" podUID="dfd83cc8-6503-45a8-8260-c0edb8752471" containerName="main" containerID="cri-o://311c1aae849d63e5bfcd7772c98b907f696d432ea84d0c75175c8cc3d5836f54" gracePeriod=45 Oct 30 13:10:59.952078 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:10:59.951836 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" podUID="dfd83cc8-6503-45a8-8260-c0edb8752471" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": dial tcp 10.134.0.56:8000: connect: connection refused" Oct 30 13:11:00.115880 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.115852 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" Oct 30 13:11:00.119477 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.119454 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" Oct 30 13:11:00.282035 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.281962 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a296df26-3a3b-40ef-a3ab-1866ecf7dcbc-home\" (UniqueName: \"kubernetes.io/empty-dir/a296df26-3a3b-40ef-a3ab-1866ecf7dcbc-home\") pod \"a296df26-3a3b-40ef-a3ab-1866ecf7dcbc\" (UID: \"a296df26-3a3b-40ef-a3ab-1866ecf7dcbc\") " Oct 30 13:11:00.282035 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.281997 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a296df26-3a3b-40ef-a3ab-1866ecf7dcbc-tls-certs\" (UniqueName: \"kubernetes.io/secret/a296df26-3a3b-40ef-a3ab-1866ecf7dcbc-tls-certs\") pod \"a296df26-3a3b-40ef-a3ab-1866ecf7dcbc\" (UID: \"a296df26-3a3b-40ef-a3ab-1866ecf7dcbc\") " Oct 30 13:11:00.282251 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.282055 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a296df26-3a3b-40ef-a3ab-1866ecf7dcbc-kube-api-access-lbzjm\" (UniqueName: \"kubernetes.io/projected/a296df26-3a3b-40ef-a3ab-1866ecf7dcbc-kube-api-access-lbzjm\") pod \"a296df26-3a3b-40ef-a3ab-1866ecf7dcbc\" (UID: \"a296df26-3a3b-40ef-a3ab-1866ecf7dcbc\") " Oct 30 13:11:00.282251 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.282082 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfd83cc8-6503-45a8-8260-c0edb8752471-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfd83cc8-6503-45a8-8260-c0edb8752471-tmp-dir\") pod \"dfd83cc8-6503-45a8-8260-c0edb8752471\" (UID: \"dfd83cc8-6503-45a8-8260-c0edb8752471\") " Oct 30 13:11:00.282251 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.282140 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfd83cc8-6503-45a8-8260-c0edb8752471-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfd83cc8-6503-45a8-8260-c0edb8752471-dshm\") pod \"dfd83cc8-6503-45a8-8260-c0edb8752471\" (UID: \"dfd83cc8-6503-45a8-8260-c0edb8752471\") " Oct 30 13:11:00.282251 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.282182 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfd83cc8-6503-45a8-8260-c0edb8752471-tls-certs\" (UniqueName: \"kubernetes.io/secret/dfd83cc8-6503-45a8-8260-c0edb8752471-tls-certs\") pod \"dfd83cc8-6503-45a8-8260-c0edb8752471\" (UID: \"dfd83cc8-6503-45a8-8260-c0edb8752471\") " Oct 30 13:11:00.282251 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.282202 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a296df26-3a3b-40ef-a3ab-1866ecf7dcbc-home" pod "a296df26-3a3b-40ef-a3ab-1866ecf7dcbc" (UID: "a296df26-3a3b-40ef-a3ab-1866ecf7dcbc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:00.282251 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.282213 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a296df26-3a3b-40ef-a3ab-1866ecf7dcbc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a296df26-3a3b-40ef-a3ab-1866ecf7dcbc-tmp-dir\") pod \"a296df26-3a3b-40ef-a3ab-1866ecf7dcbc\" (UID: \"a296df26-3a3b-40ef-a3ab-1866ecf7dcbc\") " Oct 30 13:11:00.282555 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.282266 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfd83cc8-6503-45a8-8260-c0edb8752471-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfd83cc8-6503-45a8-8260-c0edb8752471-model-cache\") pod \"dfd83cc8-6503-45a8-8260-c0edb8752471\" (UID: \"dfd83cc8-6503-45a8-8260-c0edb8752471\") " Oct 30 13:11:00.282555 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.282435 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a296df26-3a3b-40ef-a3ab-1866ecf7dcbc-tmp-dir" pod "a296df26-3a3b-40ef-a3ab-1866ecf7dcbc" (UID: "a296df26-3a3b-40ef-a3ab-1866ecf7dcbc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:00.282555 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.282435 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd83cc8-6503-45a8-8260-c0edb8752471-tmp-dir" pod "dfd83cc8-6503-45a8-8260-c0edb8752471" (UID: "dfd83cc8-6503-45a8-8260-c0edb8752471"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:00.282555 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.282459 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfd83cc8-6503-45a8-8260-c0edb8752471-home\" (UniqueName: \"kubernetes.io/empty-dir/dfd83cc8-6503-45a8-8260-c0edb8752471-home\") pod \"dfd83cc8-6503-45a8-8260-c0edb8752471\" (UID: \"dfd83cc8-6503-45a8-8260-c0edb8752471\") " Oct 30 13:11:00.282555 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.282542 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a296df26-3a3b-40ef-a3ab-1866ecf7dcbc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a296df26-3a3b-40ef-a3ab-1866ecf7dcbc-model-cache\") pod \"a296df26-3a3b-40ef-a3ab-1866ecf7dcbc\" (UID: \"a296df26-3a3b-40ef-a3ab-1866ecf7dcbc\") " Oct 30 13:11:00.282837 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.282664 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd83cc8-6503-45a8-8260-c0edb8752471-model-cache" pod "dfd83cc8-6503-45a8-8260-c0edb8752471" (UID: "dfd83cc8-6503-45a8-8260-c0edb8752471"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:00.282837 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.282679 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd83cc8-6503-45a8-8260-c0edb8752471-home" pod "dfd83cc8-6503-45a8-8260-c0edb8752471" (UID: "dfd83cc8-6503-45a8-8260-c0edb8752471"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:00.282837 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.282706 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfd83cc8-6503-45a8-8260-c0edb8752471-kube-api-access-59crd\" (UniqueName: \"kubernetes.io/projected/dfd83cc8-6503-45a8-8260-c0edb8752471-kube-api-access-59crd\") pod \"dfd83cc8-6503-45a8-8260-c0edb8752471\" (UID: \"dfd83cc8-6503-45a8-8260-c0edb8752471\") " Oct 30 13:11:00.282837 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.282747 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a296df26-3a3b-40ef-a3ab-1866ecf7dcbc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a296df26-3a3b-40ef-a3ab-1866ecf7dcbc-dshm\") pod \"a296df26-3a3b-40ef-a3ab-1866ecf7dcbc\" (UID: \"a296df26-3a3b-40ef-a3ab-1866ecf7dcbc\") " Oct 30 13:11:00.283057 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.282875 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a296df26-3a3b-40ef-a3ab-1866ecf7dcbc-model-cache" pod "a296df26-3a3b-40ef-a3ab-1866ecf7dcbc" (UID: "a296df26-3a3b-40ef-a3ab-1866ecf7dcbc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:00.283498 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.283459 5922 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfd83cc8-6503-45a8-8260-c0edb8752471-tmp-dir\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:00.283498 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.283488 5922 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a296df26-3a3b-40ef-a3ab-1866ecf7dcbc-tmp-dir\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:00.283732 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.283504 5922 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfd83cc8-6503-45a8-8260-c0edb8752471-model-cache\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:00.283732 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.283518 5922 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfd83cc8-6503-45a8-8260-c0edb8752471-home\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:00.283732 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.283530 5922 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a296df26-3a3b-40ef-a3ab-1866ecf7dcbc-model-cache\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:00.283732 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.283542 5922 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a296df26-3a3b-40ef-a3ab-1866ecf7dcbc-home\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:00.284678 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.284653 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfd83cc8-6503-45a8-8260-c0edb8752471-tls-certs" pod "dfd83cc8-6503-45a8-8260-c0edb8752471" (UID: "dfd83cc8-6503-45a8-8260-c0edb8752471"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:11:00.284903 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.284873 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a296df26-3a3b-40ef-a3ab-1866ecf7dcbc-tls-certs" pod "a296df26-3a3b-40ef-a3ab-1866ecf7dcbc" (UID: "a296df26-3a3b-40ef-a3ab-1866ecf7dcbc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:11:00.284975 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.284958 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a296df26-3a3b-40ef-a3ab-1866ecf7dcbc-kube-api-access-lbzjm" pod "a296df26-3a3b-40ef-a3ab-1866ecf7dcbc" (UID: "a296df26-3a3b-40ef-a3ab-1866ecf7dcbc"). InnerVolumeSpecName "kube-api-access-lbzjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:00.285111 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.285093 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd83cc8-6503-45a8-8260-c0edb8752471-dshm" pod "dfd83cc8-6503-45a8-8260-c0edb8752471" (UID: "dfd83cc8-6503-45a8-8260-c0edb8752471"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:00.285205 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.285183 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfd83cc8-6503-45a8-8260-c0edb8752471-kube-api-access-59crd" pod "dfd83cc8-6503-45a8-8260-c0edb8752471" (UID: "dfd83cc8-6503-45a8-8260-c0edb8752471"). InnerVolumeSpecName "kube-api-access-59crd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:00.285267 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.285197 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a296df26-3a3b-40ef-a3ab-1866ecf7dcbc-dshm" pod "a296df26-3a3b-40ef-a3ab-1866ecf7dcbc" (UID: "a296df26-3a3b-40ef-a3ab-1866ecf7dcbc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:00.336464 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.336430 5922 generic.go:356] "Generic (PLEG): container finished" podID="dfd83cc8-6503-45a8-8260-c0edb8752471" containerID="311c1aae849d63e5bfcd7772c98b907f696d432ea84d0c75175c8cc3d5836f54" exitCode=0 Oct 30 13:11:00.336643 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.336481 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" Oct 30 13:11:00.336643 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.336512 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" event={"ID":"dfd83cc8-6503-45a8-8260-c0edb8752471","Type":"ContainerDied","Data":"311c1aae849d63e5bfcd7772c98b907f696d432ea84d0c75175c8cc3d5836f54"} Oct 30 13:11:00.336643 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.336559 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm" event={"ID":"dfd83cc8-6503-45a8-8260-c0edb8752471","Type":"ContainerDied","Data":"ce1832b714c4052e537b824d135a0906451acdb891980a959fc750e2f05abb10"} Oct 30 13:11:00.336643 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.336576 5922 scope.go:122] "RemoveContainer" containerID="311c1aae849d63e5bfcd7772c98b907f696d432ea84d0c75175c8cc3d5836f54" Oct 30 13:11:00.338406 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.338382 5922 generic.go:356] "Generic (PLEG): container finished" podID="a296df26-3a3b-40ef-a3ab-1866ecf7dcbc" containerID="ec236ae48eaf54b6aa004cec816f861bef7139ccc86a08636af684361fb347c7" exitCode=0 Oct 30 13:11:00.338527 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.338419 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" event={"ID":"a296df26-3a3b-40ef-a3ab-1866ecf7dcbc","Type":"ContainerDied","Data":"ec236ae48eaf54b6aa004cec816f861bef7139ccc86a08636af684361fb347c7"} Oct 30 13:11:00.338527 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.338434 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" Oct 30 13:11:00.338527 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.338461 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" event={"ID":"a296df26-3a3b-40ef-a3ab-1866ecf7dcbc","Type":"ContainerDied","Data":"8e44954240c2e15513fc436a61e8d7649d0148dd59f598e0cac55fd670e090c9"} Oct 30 13:11:00.348350 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.348328 5922 scope.go:122] "RemoveContainer" containerID="311c1aae849d63e5bfcd7772c98b907f696d432ea84d0c75175c8cc3d5836f54" Oct 30 13:11:00.348657 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:11:00.348635 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"311c1aae849d63e5bfcd7772c98b907f696d432ea84d0c75175c8cc3d5836f54\": container with ID starting with 311c1aae849d63e5bfcd7772c98b907f696d432ea84d0c75175c8cc3d5836f54 not found: ID does not exist" containerID="311c1aae849d63e5bfcd7772c98b907f696d432ea84d0c75175c8cc3d5836f54" Oct 30 13:11:00.348720 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.348663 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"311c1aae849d63e5bfcd7772c98b907f696d432ea84d0c75175c8cc3d5836f54"} err="failed to get container status \"311c1aae849d63e5bfcd7772c98b907f696d432ea84d0c75175c8cc3d5836f54\": rpc error: code = NotFound desc = could not find container \"311c1aae849d63e5bfcd7772c98b907f696d432ea84d0c75175c8cc3d5836f54\": container with ID starting with 311c1aae849d63e5bfcd7772c98b907f696d432ea84d0c75175c8cc3d5836f54 not found: ID does not exist" Oct 30 13:11:00.348720 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.348679 5922 scope.go:122] "RemoveContainer" containerID="ec236ae48eaf54b6aa004cec816f861bef7139ccc86a08636af684361fb347c7" Oct 30 13:11:00.359436 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.359417 5922 scope.go:122] "RemoveContainer" containerID="ec236ae48eaf54b6aa004cec816f861bef7139ccc86a08636af684361fb347c7" Oct 30 13:11:00.359693 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:11:00.359675 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec236ae48eaf54b6aa004cec816f861bef7139ccc86a08636af684361fb347c7\": container with ID starting with ec236ae48eaf54b6aa004cec816f861bef7139ccc86a08636af684361fb347c7 not found: ID does not exist" containerID="ec236ae48eaf54b6aa004cec816f861bef7139ccc86a08636af684361fb347c7" Oct 30 13:11:00.359754 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.359700 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec236ae48eaf54b6aa004cec816f861bef7139ccc86a08636af684361fb347c7"} err="failed to get container status \"ec236ae48eaf54b6aa004cec816f861bef7139ccc86a08636af684361fb347c7\": rpc error: code = NotFound desc = could not find container \"ec236ae48eaf54b6aa004cec816f861bef7139ccc86a08636af684361fb347c7\": container with ID starting with ec236ae48eaf54b6aa004cec816f861bef7139ccc86a08636af684361fb347c7 not found: ID does not exist" Oct 30 13:11:00.385162 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.385138 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59crd\" (UniqueName: \"kubernetes.io/projected/dfd83cc8-6503-45a8-8260-c0edb8752471-kube-api-access-59crd\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:00.385162 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.385161 5922 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a296df26-3a3b-40ef-a3ab-1866ecf7dcbc-dshm\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:00.385268 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.385172 5922 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a296df26-3a3b-40ef-a3ab-1866ecf7dcbc-tls-certs\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:00.385268 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.385181 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbzjm\" (UniqueName: \"kubernetes.io/projected/a296df26-3a3b-40ef-a3ab-1866ecf7dcbc-kube-api-access-lbzjm\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:00.385268 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.385189 5922 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfd83cc8-6503-45a8-8260-c0edb8752471-dshm\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:00.385268 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.385196 5922 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfd83cc8-6503-45a8-8260-c0edb8752471-tls-certs\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:00.393695 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.393674 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm"] Oct 30 13:11:00.398725 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.398704 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-42ee9d/premium-w0-42ee9d-kserve-b4df54c7c-4sskm"] Oct 30 13:11:00.435400 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.435379 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm"] Oct 30 13:11:00.441178 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.441154 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm"] Oct 30 13:11:00.683390 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.683309 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:00.927988 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:00.927949 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42ee9d/distinct-w0-42ee9d-kserve-f46fffc8d-r92sm" podUID="a296df26-3a3b-40ef-a3ab-1866ecf7dcbc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:01.647350 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:01.647317 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a296df26-3a3b-40ef-a3ab-1866ecf7dcbc" path="/var/lib/kubelet/pods/a296df26-3a3b-40ef-a3ab-1866ecf7dcbc/volumes" Oct 30 13:11:01.647901 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:01.647879 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfd83cc8-6503-45a8-8260-c0edb8752471" path="/var/lib/kubelet/pods/dfd83cc8-6503-45a8-8260-c0edb8752471/volumes" Oct 30 13:11:02.215846 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:02.215806 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:02.389085 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:02.389050 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3048f9-j6df9"] Oct 30 13:11:02.389888 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:02.389867 5922 cpu_manager.go:414] "containerMap: removing container" podUID="dfd83cc8-6503-45a8-8260-c0edb8752471" containerName="main" Oct 30 13:11:02.389888 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:02.389886 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfd83cc8-6503-45a8-8260-c0edb8752471" containerName="main" Oct 30 13:11:02.390017 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:02.389923 5922 cpu_manager.go:414] "containerMap: removing container" podUID="d4f503a4-2366-439a-9603-e2e49b8c823a" containerName="revoke-keys" Oct 30 13:11:02.390017 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:02.389929 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4f503a4-2366-439a-9603-e2e49b8c823a" containerName="revoke-keys" Oct 30 13:11:02.390017 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:02.389941 5922 cpu_manager.go:414] "containerMap: removing container" podUID="99d3f2ce-87ad-4849-8976-7c388312e1e7" containerName="istio-proxy" Oct 30 13:11:02.390017 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:02.389947 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99d3f2ce-87ad-4849-8976-7c388312e1e7" containerName="istio-proxy" Oct 30 13:11:02.390017 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:02.389956 5922 cpu_manager.go:414] "containerMap: removing container" podUID="a296df26-3a3b-40ef-a3ab-1866ecf7dcbc" containerName="main" Oct 30 13:11:02.390017 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:02.389961 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a296df26-3a3b-40ef-a3ab-1866ecf7dcbc" containerName="main" Oct 30 13:11:02.390017 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:02.389973 5922 cpu_manager.go:414] "containerMap: removing container" podUID="d6b7cc9a-3532-4b96-b523-fb5eeb9f6258" containerName="revoke-keys" Oct 30 13:11:02.390017 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:02.389978 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6b7cc9a-3532-4b96-b523-fb5eeb9f6258" containerName="revoke-keys" Oct 30 13:11:02.390255 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:02.390087 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfd83cc8-6503-45a8-8260-c0edb8752471" containerName="main" Oct 30 13:11:02.390255 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:02.390099 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4f503a4-2366-439a-9603-e2e49b8c823a" containerName="revoke-keys" Oct 30 13:11:02.390255 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:02.390107 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="a296df26-3a3b-40ef-a3ab-1866ecf7dcbc" containerName="main" Oct 30 13:11:02.390255 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:02.390114 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6b7cc9a-3532-4b96-b523-fb5eeb9f6258" containerName="revoke-keys" Oct 30 13:11:02.395509 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:02.395492 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3048f9-j6df9" Oct 30 13:11:02.412883 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:02.412857 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/757485a1-9f1f-472a-a000-0f4f6b2ab14e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-3048f9-j6df9\" (UID: \"757485a1-9f1f-472a-a000-0f4f6b2ab14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3048f9-j6df9" Oct 30 13:11:02.412993 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:02.412949 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv694\" (UniqueName: \"kubernetes.io/projected/757485a1-9f1f-472a-a000-0f4f6b2ab14e-kube-api-access-xv694\") pod \"maas-api-revoke-keys-e2e-worker-w3-3048f9-j6df9\" (UID: \"757485a1-9f1f-472a-a000-0f4f6b2ab14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3048f9-j6df9" Oct 30 13:11:02.413036 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:02.413008 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/757485a1-9f1f-472a-a000-0f4f6b2ab14e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-3048f9-j6df9\" (UID: \"757485a1-9f1f-472a-a000-0f4f6b2ab14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3048f9-j6df9" Oct 30 13:11:02.413956 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:02.413937 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3048f9-j6df9"] Oct 30 13:11:02.422452 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:02.420501 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:11:02.422452 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:02.420551 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:11:02.422452 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:02.420564 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:11:02.422809 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:02.422791 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:11:02.513966 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:02.513894 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/757485a1-9f1f-472a-a000-0f4f6b2ab14e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-3048f9-j6df9\" (UID: \"757485a1-9f1f-472a-a000-0f4f6b2ab14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3048f9-j6df9" Oct 30 13:11:02.514103 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:02.514087 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/757485a1-9f1f-472a-a000-0f4f6b2ab14e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-3048f9-j6df9\" (UID: \"757485a1-9f1f-472a-a000-0f4f6b2ab14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3048f9-j6df9" Oct 30 13:11:02.514164 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:02.514150 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv694\" (UniqueName: \"kubernetes.io/projected/757485a1-9f1f-472a-a000-0f4f6b2ab14e-kube-api-access-xv694\") pod \"maas-api-revoke-keys-e2e-worker-w3-3048f9-j6df9\" (UID: \"757485a1-9f1f-472a-a000-0f4f6b2ab14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3048f9-j6df9" Oct 30 13:11:02.514532 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:02.514498 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/757485a1-9f1f-472a-a000-0f4f6b2ab14e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-3048f9-j6df9\" (UID: \"757485a1-9f1f-472a-a000-0f4f6b2ab14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3048f9-j6df9" Oct 30 13:11:02.516482 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:02.516464 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/757485a1-9f1f-472a-a000-0f4f6b2ab14e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-3048f9-j6df9\" (UID: \"757485a1-9f1f-472a-a000-0f4f6b2ab14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3048f9-j6df9" Oct 30 13:11:02.541143 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:02.541111 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv694\" (UniqueName: \"kubernetes.io/projected/757485a1-9f1f-472a-a000-0f4f6b2ab14e-kube-api-access-xv694\") pod \"maas-api-revoke-keys-e2e-worker-w3-3048f9-j6df9\" (UID: \"757485a1-9f1f-472a-a000-0f4f6b2ab14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3048f9-j6df9" Oct 30 13:11:02.710713 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:02.710680 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3048f9-j6df9" Oct 30 13:11:02.749427 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:02.749391 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:03.024832 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:11:03.024762 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod757485a1_9f1f_472a_a000_0f4f6b2ab14e.slice/crio-82bafd8f3996e3fc748a31ee5cbda39a325a2b2fe7c3f3d2e958ad4b4d10d9e1 WatchSource:0}: Error finding container 82bafd8f3996e3fc748a31ee5cbda39a325a2b2fe7c3f3d2e958ad4b4d10d9e1: Status 404 returned error can't find the container with id 82bafd8f3996e3fc748a31ee5cbda39a325a2b2fe7c3f3d2e958ad4b4d10d9e1 Oct 30 13:11:03.025845 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:03.025816 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3048f9-j6df9"] Oct 30 13:11:03.371907 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:03.371820 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3048f9-j6df9" event={"ID":"757485a1-9f1f-472a-a000-0f4f6b2ab14e","Type":"ContainerStarted","Data":"88d0bb0714b430a483c165a29e6a401d62a2fbd0e515a34aa7e89c5bc94cfc92"} Oct 30 13:11:03.371907 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:03.371871 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3048f9-j6df9" event={"ID":"757485a1-9f1f-472a-a000-0f4f6b2ab14e","Type":"ContainerStarted","Data":"82bafd8f3996e3fc748a31ee5cbda39a325a2b2fe7c3f3d2e958ad4b4d10d9e1"} Oct 30 13:11:03.438136 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:03.438088 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3048f9-j6df9" podStartSLOduration=1.438075327 podStartE2EDuration="1.438075327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:03.436483368 +0000 UTC m=+1119.905899127" watchObservedRunningTime="2026-10-30 13:11:03.438075327 +0000 UTC m=+1119.907491072" Oct 30 13:11:04.382177 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:04.382139 5922 generic.go:356] "Generic (PLEG): container finished" podID="757485a1-9f1f-472a-a000-0f4f6b2ab14e" containerID="88d0bb0714b430a483c165a29e6a401d62a2fbd0e515a34aa7e89c5bc94cfc92" exitCode=0 Oct 30 13:11:04.382693 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:04.382216 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3048f9-j6df9" event={"ID":"757485a1-9f1f-472a-a000-0f4f6b2ab14e","Type":"ContainerDied","Data":"88d0bb0714b430a483c165a29e6a401d62a2fbd0e515a34aa7e89c5bc94cfc92"} Oct 30 13:11:04.643836 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:04.643760 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:05.031326 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:05.031290 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:05.343045 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:05.342958 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-42ee9d-5645d7744d-d6f7m" podUID="85a299c7-b978-4b75-8aed-fff5da4d32f7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:11:05.585207 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:05.585182 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3048f9-j6df9" Oct 30 13:11:05.661983 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:05.661913 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/757485a1-9f1f-472a-a000-0f4f6b2ab14e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/757485a1-9f1f-472a-a000-0f4f6b2ab14e-maas-api-xks-ca\") pod \"757485a1-9f1f-472a-a000-0f4f6b2ab14e\" (UID: \"757485a1-9f1f-472a-a000-0f4f6b2ab14e\") " Oct 30 13:11:05.662113 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:05.662013 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/757485a1-9f1f-472a-a000-0f4f6b2ab14e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/757485a1-9f1f-472a-a000-0f4f6b2ab14e-maas-api-openshift-ca\") pod \"757485a1-9f1f-472a-a000-0f4f6b2ab14e\" (UID: \"757485a1-9f1f-472a-a000-0f4f6b2ab14e\") " Oct 30 13:11:05.662113 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:05.662075 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/757485a1-9f1f-472a-a000-0f4f6b2ab14e-kube-api-access-xv694\" (UniqueName: \"kubernetes.io/projected/757485a1-9f1f-472a-a000-0f4f6b2ab14e-kube-api-access-xv694\") pod \"757485a1-9f1f-472a-a000-0f4f6b2ab14e\" (UID: \"757485a1-9f1f-472a-a000-0f4f6b2ab14e\") " Oct 30 13:11:05.662394 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:05.662364 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/757485a1-9f1f-472a-a000-0f4f6b2ab14e-maas-api-openshift-ca" pod "757485a1-9f1f-472a-a000-0f4f6b2ab14e" (UID: "757485a1-9f1f-472a-a000-0f4f6b2ab14e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:05.662949 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:05.662922 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/757485a1-9f1f-472a-a000-0f4f6b2ab14e-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:05.664105 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:05.664077 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/757485a1-9f1f-472a-a000-0f4f6b2ab14e-maas-api-xks-ca" pod "757485a1-9f1f-472a-a000-0f4f6b2ab14e" (UID: "757485a1-9f1f-472a-a000-0f4f6b2ab14e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:11:05.664273 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:05.664254 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/757485a1-9f1f-472a-a000-0f4f6b2ab14e-kube-api-access-xv694" pod "757485a1-9f1f-472a-a000-0f4f6b2ab14e" (UID: "757485a1-9f1f-472a-a000-0f4f6b2ab14e"). InnerVolumeSpecName "kube-api-access-xv694". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:05.764663 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:05.764638 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/757485a1-9f1f-472a-a000-0f4f6b2ab14e-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:05.764663 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:05.764663 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv694\" (UniqueName: \"kubernetes.io/projected/757485a1-9f1f-472a-a000-0f4f6b2ab14e-kube-api-access-xv694\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:06.401489 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:06.401453 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3048f9-j6df9" Oct 30 13:11:06.401668 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:06.401479 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3048f9-j6df9" event={"ID":"757485a1-9f1f-472a-a000-0f4f6b2ab14e","Type":"ContainerDied","Data":"82bafd8f3996e3fc748a31ee5cbda39a325a2b2fe7c3f3d2e958ad4b4d10d9e1"} Oct 30 13:11:06.401668 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:06.401560 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82bafd8f3996e3fc748a31ee5cbda39a325a2b2fe7c3f3d2e958ad4b4d10d9e1" Oct 30 13:11:06.559259 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:06.559227 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d1aa41c-tt4zt"] Oct 30 13:11:06.559915 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:06.559902 5922 cpu_manager.go:414] "containerMap: removing container" podUID="757485a1-9f1f-472a-a000-0f4f6b2ab14e" containerName="revoke-keys" Oct 30 13:11:06.559955 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:06.559917 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="757485a1-9f1f-472a-a000-0f4f6b2ab14e" containerName="revoke-keys" Oct 30 13:11:06.560035 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:06.560026 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="757485a1-9f1f-472a-a000-0f4f6b2ab14e" containerName="revoke-keys" Oct 30 13:11:06.563838 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:06.563820 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d1aa41c-tt4zt" Oct 30 13:11:06.579341 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:06.579311 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe3da6ff-bq57s"] Oct 30 13:11:06.584500 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:06.584478 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3da6ff-bq57s" Oct 30 13:11:06.585053 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:06.584997 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:11:06.585053 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:06.585006 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:11:06.585053 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:06.585038 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:11:06.586059 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:06.585476 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d1aa41c-tt4zt"] Oct 30 13:11:06.589879 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:06.589856 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:11:06.611145 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:06.611117 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe3da6ff-bq57s"] Oct 30 13:11:06.674307 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:06.674232 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/123b434c-7da6-49b6-9f2f-801c92f68c59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe3da6ff-bq57s\" (UID: \"123b434c-7da6-49b6-9f2f-801c92f68c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3da6ff-bq57s" Oct 30 13:11:06.674307 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:06.674273 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/123b434c-7da6-49b6-9f2f-801c92f68c59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe3da6ff-bq57s\" (UID: \"123b434c-7da6-49b6-9f2f-801c92f68c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3da6ff-bq57s" Oct 30 13:11:06.674494 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:06.674373 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ce4cf03-e6a2-442f-8065-9a3f541e84c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d1aa41c-tt4zt\" (UID: \"4ce4cf03-e6a2-442f-8065-9a3f541e84c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d1aa41c-tt4zt" Oct 30 13:11:06.674494 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:06.674412 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wddh7\" (UniqueName: \"kubernetes.io/projected/4ce4cf03-e6a2-442f-8065-9a3f541e84c1-kube-api-access-wddh7\") pod \"maas-api-revoke-sub-1d1aa41c-tt4zt\" (UID: \"4ce4cf03-e6a2-442f-8065-9a3f541e84c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d1aa41c-tt4zt" Oct 30 13:11:06.674566 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:06.674519 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxd7q\" (UniqueName: \"kubernetes.io/projected/123b434c-7da6-49b6-9f2f-801c92f68c59-kube-api-access-lxd7q\") pod \"maas-api-revoke-sub-fe3da6ff-bq57s\" (UID: \"123b434c-7da6-49b6-9f2f-801c92f68c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3da6ff-bq57s" Oct 30 13:11:06.674703 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:06.674690 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ce4cf03-e6a2-442f-8065-9a3f541e84c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d1aa41c-tt4zt\" (UID: \"4ce4cf03-e6a2-442f-8065-9a3f541e84c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d1aa41c-tt4zt" Oct 30 13:11:06.775868 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:06.775835 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/123b434c-7da6-49b6-9f2f-801c92f68c59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe3da6ff-bq57s\" (UID: \"123b434c-7da6-49b6-9f2f-801c92f68c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3da6ff-bq57s" Oct 30 13:11:06.776010 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:06.775903 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/123b434c-7da6-49b6-9f2f-801c92f68c59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe3da6ff-bq57s\" (UID: \"123b434c-7da6-49b6-9f2f-801c92f68c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3da6ff-bq57s" Oct 30 13:11:06.776010 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:06.775934 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ce4cf03-e6a2-442f-8065-9a3f541e84c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d1aa41c-tt4zt\" (UID: \"4ce4cf03-e6a2-442f-8065-9a3f541e84c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d1aa41c-tt4zt" Oct 30 13:11:06.776010 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:06.775966 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wddh7\" (UniqueName: \"kubernetes.io/projected/4ce4cf03-e6a2-442f-8065-9a3f541e84c1-kube-api-access-wddh7\") pod \"maas-api-revoke-sub-1d1aa41c-tt4zt\" (UID: \"4ce4cf03-e6a2-442f-8065-9a3f541e84c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d1aa41c-tt4zt" Oct 30 13:11:06.776159 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:06.776048 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxd7q\" (UniqueName: \"kubernetes.io/projected/123b434c-7da6-49b6-9f2f-801c92f68c59-kube-api-access-lxd7q\") pod \"maas-api-revoke-sub-fe3da6ff-bq57s\" (UID: \"123b434c-7da6-49b6-9f2f-801c92f68c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3da6ff-bq57s" Oct 30 13:11:06.776262 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:06.776215 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ce4cf03-e6a2-442f-8065-9a3f541e84c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d1aa41c-tt4zt\" (UID: \"4ce4cf03-e6a2-442f-8065-9a3f541e84c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d1aa41c-tt4zt" Oct 30 13:11:06.776750 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:06.776692 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/123b434c-7da6-49b6-9f2f-801c92f68c59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe3da6ff-bq57s\" (UID: \"123b434c-7da6-49b6-9f2f-801c92f68c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3da6ff-bq57s" Oct 30 13:11:06.777210 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:06.777187 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ce4cf03-e6a2-442f-8065-9a3f541e84c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d1aa41c-tt4zt\" (UID: \"4ce4cf03-e6a2-442f-8065-9a3f541e84c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d1aa41c-tt4zt" Oct 30 13:11:06.779005 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:06.778981 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/123b434c-7da6-49b6-9f2f-801c92f68c59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe3da6ff-bq57s\" (UID: \"123b434c-7da6-49b6-9f2f-801c92f68c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3da6ff-bq57s" Oct 30 13:11:06.779093 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:06.779042 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ce4cf03-e6a2-442f-8065-9a3f541e84c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d1aa41c-tt4zt\" (UID: \"4ce4cf03-e6a2-442f-8065-9a3f541e84c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d1aa41c-tt4zt" Oct 30 13:11:06.801366 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:06.801338 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxd7q\" (UniqueName: \"kubernetes.io/projected/123b434c-7da6-49b6-9f2f-801c92f68c59-kube-api-access-lxd7q\") pod \"maas-api-revoke-sub-fe3da6ff-bq57s\" (UID: \"123b434c-7da6-49b6-9f2f-801c92f68c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3da6ff-bq57s" Oct 30 13:11:06.801494 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:06.801479 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wddh7\" (UniqueName: \"kubernetes.io/projected/4ce4cf03-e6a2-442f-8065-9a3f541e84c1-kube-api-access-wddh7\") pod \"maas-api-revoke-sub-1d1aa41c-tt4zt\" (UID: \"4ce4cf03-e6a2-442f-8065-9a3f541e84c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d1aa41c-tt4zt" Oct 30 13:11:06.883684 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:06.883659 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d1aa41c-tt4zt" Oct 30 13:11:06.907631 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:06.907579 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3da6ff-bq57s" Oct 30 13:11:06.932501 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:06.932389 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:07.198998 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:07.198968 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d1aa41c-tt4zt"] Oct 30 13:11:07.199362 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:11:07.199339 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ce4cf03_e6a2_442f_8065_9a3f541e84c1.slice/crio-1f463188dd31644f9aa62ca9fbbc3562bce7b25a0bad06fa8dfabc846d64c8a8 WatchSource:0}: Error finding container 1f463188dd31644f9aa62ca9fbbc3562bce7b25a0bad06fa8dfabc846d64c8a8: Status 404 returned error can't find the container with id 1f463188dd31644f9aa62ca9fbbc3562bce7b25a0bad06fa8dfabc846d64c8a8 Oct 30 13:11:07.207864 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:07.207840 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp"] Oct 30 13:11:07.216695 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:07.216673 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j"] Oct 30 13:11:07.256750 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:11:07.256722 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod123b434c_7da6_49b6_9f2f_801c92f68c59.slice/crio-c6b15948d0e2961ae5574ab7d8ea5f822ed54defa5eca4b5372b77c5c6fb83fe WatchSource:0}: Error finding container c6b15948d0e2961ae5574ab7d8ea5f822ed54defa5eca4b5372b77c5c6fb83fe: Status 404 returned error can't find the container with id c6b15948d0e2961ae5574ab7d8ea5f822ed54defa5eca4b5372b77c5c6fb83fe Oct 30 13:11:07.257710 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:07.257689 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe3da6ff-bq57s"] Oct 30 13:11:07.336744 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:07.336698 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:07.411775 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:07.411744 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d1aa41c-tt4zt" event={"ID":"4ce4cf03-e6a2-442f-8065-9a3f541e84c1","Type":"ContainerStarted","Data":"1f463188dd31644f9aa62ca9fbbc3562bce7b25a0bad06fa8dfabc846d64c8a8"} Oct 30 13:11:07.413182 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:07.413159 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3da6ff-bq57s" event={"ID":"123b434c-7da6-49b6-9f2f-801c92f68c59","Type":"ContainerStarted","Data":"c6b15948d0e2961ae5574ab7d8ea5f822ed54defa5eca4b5372b77c5c6fb83fe"} Oct 30 13:11:08.423988 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:08.423900 5922 generic.go:356] "Generic (PLEG): container finished" podID="4ce4cf03-e6a2-442f-8065-9a3f541e84c1" containerID="4e808723219d80c13f4b09952f342e8b7ec69301b4132fd762032e770428bef1" exitCode=0 Oct 30 13:11:08.423988 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:08.423949 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d1aa41c-tt4zt" event={"ID":"4ce4cf03-e6a2-442f-8065-9a3f541e84c1","Type":"ContainerDied","Data":"4e808723219d80c13f4b09952f342e8b7ec69301b4132fd762032e770428bef1"} Oct 30 13:11:08.425908 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:08.425882 5922 generic.go:356] "Generic (PLEG): container finished" podID="123b434c-7da6-49b6-9f2f-801c92f68c59" containerID="53f2de8b3865d9ad99c0449d12a59589f74a2eab109176772be9948dea645e1e" exitCode=0 Oct 30 13:11:08.426017 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:08.425962 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3da6ff-bq57s" event={"ID":"123b434c-7da6-49b6-9f2f-801c92f68c59","Type":"ContainerDied","Data":"53f2de8b3865d9ad99c0449d12a59589f74a2eab109176772be9948dea645e1e"} Oct 30 13:11:08.892484 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:08.892447 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:09.009912 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:09.009877 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:09.619282 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:09.619257 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d1aa41c-tt4zt" Oct 30 13:11:09.647771 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:09.647748 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3da6ff-bq57s" Oct 30 13:11:09.726553 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:09.726521 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ce4cf03-e6a2-442f-8065-9a3f541e84c1-kube-api-access-wddh7\" (UniqueName: \"kubernetes.io/projected/4ce4cf03-e6a2-442f-8065-9a3f541e84c1-kube-api-access-wddh7\") pod \"4ce4cf03-e6a2-442f-8065-9a3f541e84c1\" (UID: \"4ce4cf03-e6a2-442f-8065-9a3f541e84c1\") " Oct 30 13:11:09.726754 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:09.726563 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ce4cf03-e6a2-442f-8065-9a3f541e84c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ce4cf03-e6a2-442f-8065-9a3f541e84c1-maas-api-xks-ca\") pod \"4ce4cf03-e6a2-442f-8065-9a3f541e84c1\" (UID: \"4ce4cf03-e6a2-442f-8065-9a3f541e84c1\") " Oct 30 13:11:09.726754 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:09.726593 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/123b434c-7da6-49b6-9f2f-801c92f68c59-kube-api-access-lxd7q\" (UniqueName: \"kubernetes.io/projected/123b434c-7da6-49b6-9f2f-801c92f68c59-kube-api-access-lxd7q\") pod \"123b434c-7da6-49b6-9f2f-801c92f68c59\" (UID: \"123b434c-7da6-49b6-9f2f-801c92f68c59\") " Oct 30 13:11:09.726754 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:09.726653 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/123b434c-7da6-49b6-9f2f-801c92f68c59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/123b434c-7da6-49b6-9f2f-801c92f68c59-maas-api-openshift-ca\") pod \"123b434c-7da6-49b6-9f2f-801c92f68c59\" (UID: \"123b434c-7da6-49b6-9f2f-801c92f68c59\") " Oct 30 13:11:09.726952 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:09.726799 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/123b434c-7da6-49b6-9f2f-801c92f68c59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/123b434c-7da6-49b6-9f2f-801c92f68c59-maas-api-xks-ca\") pod \"123b434c-7da6-49b6-9f2f-801c92f68c59\" (UID: \"123b434c-7da6-49b6-9f2f-801c92f68c59\") " Oct 30 13:11:09.727127 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:09.727094 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/123b434c-7da6-49b6-9f2f-801c92f68c59-maas-api-openshift-ca" pod "123b434c-7da6-49b6-9f2f-801c92f68c59" (UID: "123b434c-7da6-49b6-9f2f-801c92f68c59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:09.727290 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:09.727234 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ce4cf03-e6a2-442f-8065-9a3f541e84c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ce4cf03-e6a2-442f-8065-9a3f541e84c1-maas-api-openshift-ca\") pod \"4ce4cf03-e6a2-442f-8065-9a3f541e84c1\" (UID: \"4ce4cf03-e6a2-442f-8065-9a3f541e84c1\") " Oct 30 13:11:09.727653 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:09.727603 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ce4cf03-e6a2-442f-8065-9a3f541e84c1-maas-api-openshift-ca" pod "4ce4cf03-e6a2-442f-8065-9a3f541e84c1" (UID: "4ce4cf03-e6a2-442f-8065-9a3f541e84c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:09.728854 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:09.728836 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/123b434c-7da6-49b6-9f2f-801c92f68c59-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:09.729003 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:09.728986 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ce4cf03-e6a2-442f-8065-9a3f541e84c1-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:09.729316 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:09.729273 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ce4cf03-e6a2-442f-8065-9a3f541e84c1-kube-api-access-wddh7" pod "4ce4cf03-e6a2-442f-8065-9a3f541e84c1" (UID: "4ce4cf03-e6a2-442f-8065-9a3f541e84c1"). InnerVolumeSpecName "kube-api-access-wddh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:09.729862 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:09.729834 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/123b434c-7da6-49b6-9f2f-801c92f68c59-maas-api-xks-ca" pod "123b434c-7da6-49b6-9f2f-801c92f68c59" (UID: "123b434c-7da6-49b6-9f2f-801c92f68c59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:11:09.731137 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:09.731112 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/123b434c-7da6-49b6-9f2f-801c92f68c59-kube-api-access-lxd7q" pod "123b434c-7da6-49b6-9f2f-801c92f68c59" (UID: "123b434c-7da6-49b6-9f2f-801c92f68c59"). InnerVolumeSpecName "kube-api-access-lxd7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:09.731225 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:09.731120 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ce4cf03-e6a2-442f-8065-9a3f541e84c1-maas-api-xks-ca" pod "4ce4cf03-e6a2-442f-8065-9a3f541e84c1" (UID: "4ce4cf03-e6a2-442f-8065-9a3f541e84c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:11:09.830945 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:09.830922 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wddh7\" (UniqueName: \"kubernetes.io/projected/4ce4cf03-e6a2-442f-8065-9a3f541e84c1-kube-api-access-wddh7\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:09.830945 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:09.830944 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ce4cf03-e6a2-442f-8065-9a3f541e84c1-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:09.831105 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:09.830955 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxd7q\" (UniqueName: \"kubernetes.io/projected/123b434c-7da6-49b6-9f2f-801c92f68c59-kube-api-access-lxd7q\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:09.831105 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:09.830963 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/123b434c-7da6-49b6-9f2f-801c92f68c59-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:09.964941 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:09.964915 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:10.452143 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:10.452112 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3da6ff-bq57s" Oct 30 13:11:10.452972 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:10.452290 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3da6ff-bq57s" event={"ID":"123b434c-7da6-49b6-9f2f-801c92f68c59","Type":"ContainerDied","Data":"c6b15948d0e2961ae5574ab7d8ea5f822ed54defa5eca4b5372b77c5c6fb83fe"} Oct 30 13:11:10.452972 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:10.452358 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6b15948d0e2961ae5574ab7d8ea5f822ed54defa5eca4b5372b77c5c6fb83fe" Oct 30 13:11:10.454174 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:10.454141 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d1aa41c-tt4zt" event={"ID":"4ce4cf03-e6a2-442f-8065-9a3f541e84c1","Type":"ContainerDied","Data":"1f463188dd31644f9aa62ca9fbbc3562bce7b25a0bad06fa8dfabc846d64c8a8"} Oct 30 13:11:10.454174 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:10.454166 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d1aa41c-tt4zt" Oct 30 13:11:10.454367 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:10.454193 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f463188dd31644f9aa62ca9fbbc3562bce7b25a0bad06fa8dfabc846d64c8a8" Oct 30 13:11:10.773307 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:10.773227 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:10.905445 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:10.905410 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:10.968326 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:10.968285 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:11:12.947072 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:12.947035 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-3048f9-fc55696f5-j6t4d"] Oct 30 13:11:13.055839 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:13.055803 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:13.169653 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:13.169548 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:13.281443 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:13.281345 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:15.290181 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:15.290149 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:15.342646 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:11:15.342608 5922 logging.go:55] [core] [Channel #1739 SubChannel #1740]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.49:50052", ServerName: "10.134.0.49:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.49:50052: connect: connection refused" Oct 30 13:11:15.513661 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:15.513624 5922 generic.go:356] "Generic (PLEG): container finished" podID="85a299c7-b978-4b75-8aed-fff5da4d32f7" containerID="0d2bf6b0fa8808b75c204fd58adfd598458a476a1b1238b82760ac012441cfb9" exitCode=0 Oct 30 13:11:15.513829 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:15.513737 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-42ee9d-5645d7744d-d6f7m" event={"ID":"85a299c7-b978-4b75-8aed-fff5da4d32f7","Type":"ContainerDied","Data":"0d2bf6b0fa8808b75c204fd58adfd598458a476a1b1238b82760ac012441cfb9"} Oct 30 13:11:15.684374 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:15.684350 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-42ee9d-5645d7744d-d6f7m" Oct 30 13:11:15.776788 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:15.776757 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85a299c7-b978-4b75-8aed-fff5da4d32f7-kube-api-access-k8sc6\" (UniqueName: \"kubernetes.io/projected/85a299c7-b978-4b75-8aed-fff5da4d32f7-kube-api-access-k8sc6\") pod \"85a299c7-b978-4b75-8aed-fff5da4d32f7\" (UID: \"85a299c7-b978-4b75-8aed-fff5da4d32f7\") " Oct 30 13:11:15.776961 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:15.776923 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85a299c7-b978-4b75-8aed-fff5da4d32f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85a299c7-b978-4b75-8aed-fff5da4d32f7-plugins-config-volume\") pod \"85a299c7-b978-4b75-8aed-fff5da4d32f7\" (UID: \"85a299c7-b978-4b75-8aed-fff5da4d32f7\") " Oct 30 13:11:15.777275 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:15.777251 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85a299c7-b978-4b75-8aed-fff5da4d32f7-plugins-config-volume" pod "85a299c7-b978-4b75-8aed-fff5da4d32f7" (UID: "85a299c7-b978-4b75-8aed-fff5da4d32f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:15.777376 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:15.777336 5922 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85a299c7-b978-4b75-8aed-fff5da4d32f7-plugins-config-volume\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:15.778882 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:15.778856 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85a299c7-b978-4b75-8aed-fff5da4d32f7-kube-api-access-k8sc6" pod "85a299c7-b978-4b75-8aed-fff5da4d32f7" (UID: "85a299c7-b978-4b75-8aed-fff5da4d32f7"). InnerVolumeSpecName "kube-api-access-k8sc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:15.878219 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:15.878191 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8sc6\" (UniqueName: \"kubernetes.io/projected/85a299c7-b978-4b75-8aed-fff5da4d32f7-kube-api-access-k8sc6\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:16.342509 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:16.342470 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-42ee9d-5645d7744d-d6f7m" podUID="85a299c7-b978-4b75-8aed-fff5da4d32f7" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.49:50052\" within 1s: context deadline exceeded" Oct 30 13:11:16.342916 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:16.342546 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-42ee9d-5645d7744d-d6f7m" Oct 30 13:11:16.524488 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:16.524460 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-42ee9d-5645d7744d-d6f7m" Oct 30 13:11:16.524675 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:16.524499 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-42ee9d-5645d7744d-d6f7m" event={"ID":"85a299c7-b978-4b75-8aed-fff5da4d32f7","Type":"ContainerDied","Data":"b50f6a645c50d434c6c1dcb6f69e4e559e83f51a8e469e09091a36b262885c2b"} Oct 30 13:11:16.524675 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:16.524563 5922 scope.go:122] "RemoveContainer" containerID="0d2bf6b0fa8808b75c204fd58adfd598458a476a1b1238b82760ac012441cfb9" Oct 30 13:11:16.583781 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:16.583760 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-42ee9d-5645d7744d-d6f7m"] Oct 30 13:11:16.589304 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:16.589281 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-42ee9d-5645d7744d-d6f7m"] Oct 30 13:11:16.757820 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:16.757790 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:17.646235 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:17.646195 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85a299c7-b978-4b75-8aed-fff5da4d32f7" path="/var/lib/kubelet/pods/85a299c7-b978-4b75-8aed-fff5da4d32f7/volumes" Oct 30 13:11:17.947399 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:17.947363 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3048f9-fc55696f5-j6t4d" podUID="68a59f39-95e5-4e6c-b589-bf0cb560a12c" containerName="payload-pre-processing" containerID="cri-o://ff02d73c2f9c94e9703af8ddbb47c989cf3b32dfb6f5220da125202b5c940293" gracePeriod=25 Oct 30 13:11:18.246832 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:18.246750 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:19.238818 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:19.238783 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:21.037167 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:21.037133 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fea9f444-sxqcc"] Oct 30 13:11:21.049074 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:21.049047 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fea9f444-sxqcc"] Oct 30 13:11:21.206689 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:21.206653 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:21.646201 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:21.646170 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0df24ea-4b3c-42b3-9283-cecc6b2858ed" path="/var/lib/kubelet/pods/f0df24ea-4b3c-42b3-9283-cecc6b2858ed/volumes" Oct 30 13:11:22.013705 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.013675 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0fd5fcba-gqxfm"] Oct 30 13:11:22.014327 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.014312 5922 cpu_manager.go:414] "containerMap: removing container" podUID="4ce4cf03-e6a2-442f-8065-9a3f541e84c1" containerName="revoke-keys" Oct 30 13:11:22.014370 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.014330 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ce4cf03-e6a2-442f-8065-9a3f541e84c1" containerName="revoke-keys" Oct 30 13:11:22.014370 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.014342 5922 cpu_manager.go:414] "containerMap: removing container" podUID="85a299c7-b978-4b75-8aed-fff5da4d32f7" containerName="payload-processing" Oct 30 13:11:22.014370 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.014347 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85a299c7-b978-4b75-8aed-fff5da4d32f7" containerName="payload-processing" Oct 30 13:11:22.014468 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.014377 5922 cpu_manager.go:414] "containerMap: removing container" podUID="123b434c-7da6-49b6-9f2f-801c92f68c59" containerName="revoke-keys" Oct 30 13:11:22.014468 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.014383 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="123b434c-7da6-49b6-9f2f-801c92f68c59" containerName="revoke-keys" Oct 30 13:11:22.014529 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.014484 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ce4cf03-e6a2-442f-8065-9a3f541e84c1" containerName="revoke-keys" Oct 30 13:11:22.014529 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.014494 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="85a299c7-b978-4b75-8aed-fff5da4d32f7" containerName="payload-processing" Oct 30 13:11:22.014529 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.014504 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="123b434c-7da6-49b6-9f2f-801c92f68c59" containerName="revoke-keys" Oct 30 13:11:22.019800 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.019782 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0fd5fcba-gqxfm" Oct 30 13:11:22.040034 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.039998 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0fd5fcba-gqxfm"] Oct 30 13:11:22.048205 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.048181 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:11:22.048363 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.048341 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:11:22.048449 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.048293 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:11:22.048449 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.048259 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:11:22.056390 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.056365 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec623696-2a8c-4ddf-8efc-310a7d683722-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0fd5fcba-gqxfm\" (UID: \"ec623696-2a8c-4ddf-8efc-310a7d683722\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0fd5fcba-gqxfm" Oct 30 13:11:22.056496 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.056437 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-979k5\" (UniqueName: \"kubernetes.io/projected/ec623696-2a8c-4ddf-8efc-310a7d683722-kube-api-access-979k5\") pod \"maas-api-revoke-keys-e2e-migrate-0fd5fcba-gqxfm\" (UID: \"ec623696-2a8c-4ddf-8efc-310a7d683722\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0fd5fcba-gqxfm" Oct 30 13:11:22.056641 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.056585 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec623696-2a8c-4ddf-8efc-310a7d683722-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0fd5fcba-gqxfm\" (UID: \"ec623696-2a8c-4ddf-8efc-310a7d683722\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0fd5fcba-gqxfm" Oct 30 13:11:22.157457 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.157423 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec623696-2a8c-4ddf-8efc-310a7d683722-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0fd5fcba-gqxfm\" (UID: \"ec623696-2a8c-4ddf-8efc-310a7d683722\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0fd5fcba-gqxfm" Oct 30 13:11:22.157647 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.157476 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-979k5\" (UniqueName: \"kubernetes.io/projected/ec623696-2a8c-4ddf-8efc-310a7d683722-kube-api-access-979k5\") pod \"maas-api-revoke-keys-e2e-migrate-0fd5fcba-gqxfm\" (UID: \"ec623696-2a8c-4ddf-8efc-310a7d683722\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0fd5fcba-gqxfm" Oct 30 13:11:22.157647 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.157506 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec623696-2a8c-4ddf-8efc-310a7d683722-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0fd5fcba-gqxfm\" (UID: \"ec623696-2a8c-4ddf-8efc-310a7d683722\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0fd5fcba-gqxfm" Oct 30 13:11:22.158092 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.158068 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec623696-2a8c-4ddf-8efc-310a7d683722-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0fd5fcba-gqxfm\" (UID: \"ec623696-2a8c-4ddf-8efc-310a7d683722\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0fd5fcba-gqxfm" Oct 30 13:11:22.159841 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.159818 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec623696-2a8c-4ddf-8efc-310a7d683722-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0fd5fcba-gqxfm\" (UID: \"ec623696-2a8c-4ddf-8efc-310a7d683722\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0fd5fcba-gqxfm" Oct 30 13:11:22.181971 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.181948 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-979k5\" (UniqueName: \"kubernetes.io/projected/ec623696-2a8c-4ddf-8efc-310a7d683722-kube-api-access-979k5\") pod \"maas-api-revoke-keys-e2e-migrate-0fd5fcba-gqxfm\" (UID: \"ec623696-2a8c-4ddf-8efc-310a7d683722\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0fd5fcba-gqxfm" Oct 30 13:11:22.230021 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.229961 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" podUID="5f353928-8cb4-49a3-b999-1b90e6cb31bb" containerName="main" containerID="cri-o://a028881b68f42e00fc7c15e02e711bd3904046f6c22ec739ab99fe6a6dc8dbb7" gracePeriod=45 Oct 30 13:11:22.237724 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.237683 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" podUID="c0ea7fde-2ac5-432d-bb9b-534024186ac9" containerName="main" containerID="cri-o://a49b157703c046f271f6a406e5b85c923c3dc9a0a3f71a6fc6962a3f04cec30a" gracePeriod=45 Oct 30 13:11:22.336120 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.336095 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0fd5fcba-gqxfm" Oct 30 13:11:22.537098 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.537072 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" Oct 30 13:11:22.547477 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.547456 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" Oct 30 13:11:22.562479 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.562323 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f353928-8cb4-49a3-b999-1b90e6cb31bb-tls-certs\" (UniqueName: \"kubernetes.io/secret/5f353928-8cb4-49a3-b999-1b90e6cb31bb-tls-certs\") pod \"5f353928-8cb4-49a3-b999-1b90e6cb31bb\" (UID: \"5f353928-8cb4-49a3-b999-1b90e6cb31bb\") " Oct 30 13:11:22.562479 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.562372 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f353928-8cb4-49a3-b999-1b90e6cb31bb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f353928-8cb4-49a3-b999-1b90e6cb31bb-model-cache\") pod \"5f353928-8cb4-49a3-b999-1b90e6cb31bb\" (UID: \"5f353928-8cb4-49a3-b999-1b90e6cb31bb\") " Oct 30 13:11:22.562479 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.562460 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f353928-8cb4-49a3-b999-1b90e6cb31bb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f353928-8cb4-49a3-b999-1b90e6cb31bb-dshm\") pod \"5f353928-8cb4-49a3-b999-1b90e6cb31bb\" (UID: \"5f353928-8cb4-49a3-b999-1b90e6cb31bb\") " Oct 30 13:11:22.563253 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.562745 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f353928-8cb4-49a3-b999-1b90e6cb31bb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f353928-8cb4-49a3-b999-1b90e6cb31bb-tmp-dir\") pod \"5f353928-8cb4-49a3-b999-1b90e6cb31bb\" (UID: \"5f353928-8cb4-49a3-b999-1b90e6cb31bb\") " Oct 30 13:11:22.563253 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.562781 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f353928-8cb4-49a3-b999-1b90e6cb31bb-home\" (UniqueName: \"kubernetes.io/empty-dir/5f353928-8cb4-49a3-b999-1b90e6cb31bb-home\") pod \"5f353928-8cb4-49a3-b999-1b90e6cb31bb\" (UID: \"5f353928-8cb4-49a3-b999-1b90e6cb31bb\") " Oct 30 13:11:22.563253 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.562817 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f353928-8cb4-49a3-b999-1b90e6cb31bb-kube-api-access-665pc\" (UniqueName: \"kubernetes.io/projected/5f353928-8cb4-49a3-b999-1b90e6cb31bb-kube-api-access-665pc\") pod \"5f353928-8cb4-49a3-b999-1b90e6cb31bb\" (UID: \"5f353928-8cb4-49a3-b999-1b90e6cb31bb\") " Oct 30 13:11:22.563434 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.563344 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f353928-8cb4-49a3-b999-1b90e6cb31bb-model-cache" pod "5f353928-8cb4-49a3-b999-1b90e6cb31bb" (UID: "5f353928-8cb4-49a3-b999-1b90e6cb31bb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:22.563549 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.563530 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f353928-8cb4-49a3-b999-1b90e6cb31bb-home" pod "5f353928-8cb4-49a3-b999-1b90e6cb31bb" (UID: "5f353928-8cb4-49a3-b999-1b90e6cb31bb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:22.563651 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.563579 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f353928-8cb4-49a3-b999-1b90e6cb31bb-tmp-dir" pod "5f353928-8cb4-49a3-b999-1b90e6cb31bb" (UID: "5f353928-8cb4-49a3-b999-1b90e6cb31bb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:22.563651 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.563589 5922 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f353928-8cb4-49a3-b999-1b90e6cb31bb-model-cache\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:22.565916 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.565892 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f353928-8cb4-49a3-b999-1b90e6cb31bb-tls-certs" pod "5f353928-8cb4-49a3-b999-1b90e6cb31bb" (UID: "5f353928-8cb4-49a3-b999-1b90e6cb31bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:11:22.566071 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.566049 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f353928-8cb4-49a3-b999-1b90e6cb31bb-dshm" pod "5f353928-8cb4-49a3-b999-1b90e6cb31bb" (UID: "5f353928-8cb4-49a3-b999-1b90e6cb31bb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:22.566147 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.566078 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f353928-8cb4-49a3-b999-1b90e6cb31bb-kube-api-access-665pc" pod "5f353928-8cb4-49a3-b999-1b90e6cb31bb" (UID: "5f353928-8cb4-49a3-b999-1b90e6cb31bb"). InnerVolumeSpecName "kube-api-access-665pc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:22.585785 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.585752 5922 generic.go:356] "Generic (PLEG): container finished" podID="c0ea7fde-2ac5-432d-bb9b-534024186ac9" containerID="a49b157703c046f271f6a406e5b85c923c3dc9a0a3f71a6fc6962a3f04cec30a" exitCode=0 Oct 30 13:11:22.585894 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.585790 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" Oct 30 13:11:22.585894 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.585834 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" event={"ID":"c0ea7fde-2ac5-432d-bb9b-534024186ac9","Type":"ContainerDied","Data":"a49b157703c046f271f6a406e5b85c923c3dc9a0a3f71a6fc6962a3f04cec30a"} Oct 30 13:11:22.585894 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.585877 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j" event={"ID":"c0ea7fde-2ac5-432d-bb9b-534024186ac9","Type":"ContainerDied","Data":"84326493f218fbf8a55a33d5e321c499a00ac03a528793c784a87d7e19f9ad3b"} Oct 30 13:11:22.586081 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.585895 5922 scope.go:122] "RemoveContainer" containerID="a49b157703c046f271f6a406e5b85c923c3dc9a0a3f71a6fc6962a3f04cec30a" Oct 30 13:11:22.587661 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.587638 5922 generic.go:356] "Generic (PLEG): container finished" podID="5f353928-8cb4-49a3-b999-1b90e6cb31bb" containerID="a028881b68f42e00fc7c15e02e711bd3904046f6c22ec739ab99fe6a6dc8dbb7" exitCode=0 Oct 30 13:11:22.587761 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.587749 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" Oct 30 13:11:22.587826 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.587777 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" event={"ID":"5f353928-8cb4-49a3-b999-1b90e6cb31bb","Type":"ContainerDied","Data":"a028881b68f42e00fc7c15e02e711bd3904046f6c22ec739ab99fe6a6dc8dbb7"} Oct 30 13:11:22.587878 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.587822 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp" event={"ID":"5f353928-8cb4-49a3-b999-1b90e6cb31bb","Type":"ContainerDied","Data":"5b16afcd292ec72f4f11bbfe127239fffeec210ccc27b7065750172afa11ee49"} Oct 30 13:11:22.602913 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.602890 5922 scope.go:122] "RemoveContainer" containerID="a49b157703c046f271f6a406e5b85c923c3dc9a0a3f71a6fc6962a3f04cec30a" Oct 30 13:11:22.603354 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:11:22.603325 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a49b157703c046f271f6a406e5b85c923c3dc9a0a3f71a6fc6962a3f04cec30a\": container with ID starting with a49b157703c046f271f6a406e5b85c923c3dc9a0a3f71a6fc6962a3f04cec30a not found: ID does not exist" containerID="a49b157703c046f271f6a406e5b85c923c3dc9a0a3f71a6fc6962a3f04cec30a" Oct 30 13:11:22.603500 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.603369 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a49b157703c046f271f6a406e5b85c923c3dc9a0a3f71a6fc6962a3f04cec30a"} err="failed to get container status \"a49b157703c046f271f6a406e5b85c923c3dc9a0a3f71a6fc6962a3f04cec30a\": rpc error: code = NotFound desc = could not find container \"a49b157703c046f271f6a406e5b85c923c3dc9a0a3f71a6fc6962a3f04cec30a\": container with ID starting with a49b157703c046f271f6a406e5b85c923c3dc9a0a3f71a6fc6962a3f04cec30a not found: ID does not exist" Oct 30 13:11:22.603500 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.603395 5922 scope.go:122] "RemoveContainer" containerID="a028881b68f42e00fc7c15e02e711bd3904046f6c22ec739ab99fe6a6dc8dbb7" Oct 30 13:11:22.619981 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.619965 5922 scope.go:122] "RemoveContainer" containerID="a028881b68f42e00fc7c15e02e711bd3904046f6c22ec739ab99fe6a6dc8dbb7" Oct 30 13:11:22.620268 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:11:22.620240 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a028881b68f42e00fc7c15e02e711bd3904046f6c22ec739ab99fe6a6dc8dbb7\": container with ID starting with a028881b68f42e00fc7c15e02e711bd3904046f6c22ec739ab99fe6a6dc8dbb7 not found: ID does not exist" containerID="a028881b68f42e00fc7c15e02e711bd3904046f6c22ec739ab99fe6a6dc8dbb7" Oct 30 13:11:22.620441 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.620277 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a028881b68f42e00fc7c15e02e711bd3904046f6c22ec739ab99fe6a6dc8dbb7"} err="failed to get container status \"a028881b68f42e00fc7c15e02e711bd3904046f6c22ec739ab99fe6a6dc8dbb7\": rpc error: code = NotFound desc = could not find container \"a028881b68f42e00fc7c15e02e711bd3904046f6c22ec739ab99fe6a6dc8dbb7\": container with ID starting with a028881b68f42e00fc7c15e02e711bd3904046f6c22ec739ab99fe6a6dc8dbb7 not found: ID does not exist" Oct 30 13:11:22.668342 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.668315 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0fd5fcba-gqxfm"] Oct 30 13:11:22.669314 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.668914 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0ea7fde-2ac5-432d-bb9b-534024186ac9-kube-api-access-bf6g8\" (UniqueName: \"kubernetes.io/projected/c0ea7fde-2ac5-432d-bb9b-534024186ac9-kube-api-access-bf6g8\") pod \"c0ea7fde-2ac5-432d-bb9b-534024186ac9\" (UID: \"c0ea7fde-2ac5-432d-bb9b-534024186ac9\") " Oct 30 13:11:22.669503 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.669485 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0ea7fde-2ac5-432d-bb9b-534024186ac9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0ea7fde-2ac5-432d-bb9b-534024186ac9-tmp-dir\") pod \"c0ea7fde-2ac5-432d-bb9b-534024186ac9\" (UID: \"c0ea7fde-2ac5-432d-bb9b-534024186ac9\") " Oct 30 13:11:22.669811 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.669791 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0ea7fde-2ac5-432d-bb9b-534024186ac9-tmp-dir" pod "c0ea7fde-2ac5-432d-bb9b-534024186ac9" (UID: "c0ea7fde-2ac5-432d-bb9b-534024186ac9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:22.669927 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.669909 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0ea7fde-2ac5-432d-bb9b-534024186ac9-tls-certs\" (UniqueName: \"kubernetes.io/secret/c0ea7fde-2ac5-432d-bb9b-534024186ac9-tls-certs\") pod \"c0ea7fde-2ac5-432d-bb9b-534024186ac9\" (UID: \"c0ea7fde-2ac5-432d-bb9b-534024186ac9\") " Oct 30 13:11:22.670134 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.670117 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0ea7fde-2ac5-432d-bb9b-534024186ac9-home\" (UniqueName: \"kubernetes.io/empty-dir/c0ea7fde-2ac5-432d-bb9b-534024186ac9-home\") pod \"c0ea7fde-2ac5-432d-bb9b-534024186ac9\" (UID: \"c0ea7fde-2ac5-432d-bb9b-534024186ac9\") " Oct 30 13:11:22.670435 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.670418 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0ea7fde-2ac5-432d-bb9b-534024186ac9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0ea7fde-2ac5-432d-bb9b-534024186ac9-model-cache\") pod \"c0ea7fde-2ac5-432d-bb9b-534024186ac9\" (UID: \"c0ea7fde-2ac5-432d-bb9b-534024186ac9\") " Oct 30 13:11:22.670598 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.670582 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0ea7fde-2ac5-432d-bb9b-534024186ac9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0ea7fde-2ac5-432d-bb9b-534024186ac9-dshm\") pod \"c0ea7fde-2ac5-432d-bb9b-534024186ac9\" (UID: \"c0ea7fde-2ac5-432d-bb9b-534024186ac9\") " Oct 30 13:11:22.671320 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.671298 5922 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f353928-8cb4-49a3-b999-1b90e6cb31bb-tmp-dir\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:22.671442 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.671429 5922 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f353928-8cb4-49a3-b999-1b90e6cb31bb-home\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:22.671603 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.671589 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-665pc\" (UniqueName: \"kubernetes.io/projected/5f353928-8cb4-49a3-b999-1b90e6cb31bb-kube-api-access-665pc\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:22.671729 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.671713 5922 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f353928-8cb4-49a3-b999-1b90e6cb31bb-tls-certs\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:22.671830 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.671818 5922 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f353928-8cb4-49a3-b999-1b90e6cb31bb-dshm\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:22.671912 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.671901 5922 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0ea7fde-2ac5-432d-bb9b-534024186ac9-tmp-dir\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:22.671988 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.671380 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0ea7fde-2ac5-432d-bb9b-534024186ac9-model-cache" pod "c0ea7fde-2ac5-432d-bb9b-534024186ac9" (UID: "c0ea7fde-2ac5-432d-bb9b-534024186ac9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:22.672068 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.671544 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0ea7fde-2ac5-432d-bb9b-534024186ac9-home" pod "c0ea7fde-2ac5-432d-bb9b-534024186ac9" (UID: "c0ea7fde-2ac5-432d-bb9b-534024186ac9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:22.673378 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.672981 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0ea7fde-2ac5-432d-bb9b-534024186ac9-kube-api-access-bf6g8" pod "c0ea7fde-2ac5-432d-bb9b-534024186ac9" (UID: "c0ea7fde-2ac5-432d-bb9b-534024186ac9"). InnerVolumeSpecName "kube-api-access-bf6g8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:22.676764 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.676732 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0ea7fde-2ac5-432d-bb9b-534024186ac9-dshm" pod "c0ea7fde-2ac5-432d-bb9b-534024186ac9" (UID: "c0ea7fde-2ac5-432d-bb9b-534024186ac9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:22.676764 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.676747 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0ea7fde-2ac5-432d-bb9b-534024186ac9-tls-certs" pod "c0ea7fde-2ac5-432d-bb9b-534024186ac9" (UID: "c0ea7fde-2ac5-432d-bb9b-534024186ac9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:11:22.702552 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.702478 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp"] Oct 30 13:11:22.704807 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.704782 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-3048f9/premium-w3-3048f9-kserve-5dbc4b8c7-wscwp"] Oct 30 13:11:22.773974 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.773938 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf6g8\" (UniqueName: \"kubernetes.io/projected/c0ea7fde-2ac5-432d-bb9b-534024186ac9-kube-api-access-bf6g8\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:22.773974 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.773970 5922 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0ea7fde-2ac5-432d-bb9b-534024186ac9-tls-certs\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:22.774148 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.773988 5922 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0ea7fde-2ac5-432d-bb9b-534024186ac9-home\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:22.774148 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.773998 5922 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0ea7fde-2ac5-432d-bb9b-534024186ac9-model-cache\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:22.774148 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.774005 5922 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0ea7fde-2ac5-432d-bb9b-534024186ac9-dshm\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:22.938334 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.938304 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j"] Oct 30 13:11:22.944226 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:22.944198 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-3048f9/simulator-w3-3048f9-kserve-5665c6ccb8-zt49j"] Oct 30 13:11:23.342494 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:23.342459 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:23.454405 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:23.454367 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:23.633686 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:23.632556 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0fd5fcba-gqxfm" event={"ID":"ec623696-2a8c-4ddf-8efc-310a7d683722","Type":"ContainerStarted","Data":"255bfd3614b2c603399dcccd55d2e96858fa2b2c04fb1b1e9ba746c8db7a3574"} Oct 30 13:11:23.633686 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:23.632661 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0fd5fcba-gqxfm" event={"ID":"ec623696-2a8c-4ddf-8efc-310a7d683722","Type":"ContainerStarted","Data":"840141db3358012fb499d96b98ebb9c22edf3608e344c49ae0315c9d067454ca"} Oct 30 13:11:23.635019 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:11:23.634993 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:11:23.650205 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:23.650173 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f353928-8cb4-49a3-b999-1b90e6cb31bb" path="/var/lib/kubelet/pods/5f353928-8cb4-49a3-b999-1b90e6cb31bb/volumes" Oct 30 13:11:23.650844 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:23.650824 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0ea7fde-2ac5-432d-bb9b-534024186ac9" path="/var/lib/kubelet/pods/c0ea7fde-2ac5-432d-bb9b-534024186ac9/volumes" Oct 30 13:11:23.948443 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:23.948415 5922 scope.go:122] "RemoveContainer" containerID="68207c3d77ef68e743e90e957aeaf91efb4226aee63410b423af20c431974e74" Oct 30 13:11:23.960237 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:23.960215 5922 scope.go:122] "RemoveContainer" containerID="5534485af417ce5160e60dc319437d88d6b2784413c9beb2fa1f889e32a4f1fd" Oct 30 13:11:23.969869 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:23.969851 5922 scope.go:122] "RemoveContainer" containerID="20676957f431a8f0488edb5186e2fe83a70c667468543ce49f036a701fb03ab9" Oct 30 13:11:23.979129 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:23.979108 5922 scope.go:122] "RemoveContainer" containerID="fa5b76c7113cd9b24bf23b452f2b0d950ed4680c3ec67ca9f1c568d7a817c9cc" Oct 30 13:11:23.988733 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:23.988714 5922 scope.go:122] "RemoveContainer" containerID="8047d4d35ea048be67b7f96e4a0cd5a4f728e49882f631288f3c7a0abc6f81ce" Oct 30 13:11:23.997717 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:23.997700 5922 scope.go:122] "RemoveContainer" containerID="ebfbdcf2b28aad03e076b94d68e283ef2ac87cde6c6f50af28d2d3c0b50c9ed6" Oct 30 13:11:24.006876 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:24.006847 5922 scope.go:122] "RemoveContainer" containerID="eada85d9138e5d81244704ecad9ca1629c3afa876b31882c0d736fad205b6d9c" Oct 30 13:11:24.017394 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:24.017375 5922 scope.go:122] "RemoveContainer" containerID="44e6283939464347d11b827c761ff6fd91027a884eea611decf26f2a816d0a92" Oct 30 13:11:24.027559 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:24.027535 5922 scope.go:122] "RemoveContainer" containerID="71fcc72846b179a730312469ee4971897c17d850a99d0ff8d47e8c9e851d0b46" Oct 30 13:11:24.037566 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:24.037537 5922 scope.go:122] "RemoveContainer" containerID="ae4700fca92ebefbf08d774e920cdeb6a2941608312e97e402f9e64bee83e892" Oct 30 13:11:24.081150 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:24.081118 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:24.233913 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:24.233875 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:24.645088 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:24.645057 5922 generic.go:356] "Generic (PLEG): container finished" podID="ec623696-2a8c-4ddf-8efc-310a7d683722" containerID="255bfd3614b2c603399dcccd55d2e96858fa2b2c04fb1b1e9ba746c8db7a3574" exitCode=0 Oct 30 13:11:24.645489 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:24.645103 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0fd5fcba-gqxfm" event={"ID":"ec623696-2a8c-4ddf-8efc-310a7d683722","Type":"ContainerDied","Data":"255bfd3614b2c603399dcccd55d2e96858fa2b2c04fb1b1e9ba746c8db7a3574"} Oct 30 13:11:25.290412 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:25.290378 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f45f73ac-pvcfw"] Oct 30 13:11:25.291046 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:25.291031 5922 cpu_manager.go:414] "containerMap: removing container" podUID="5f353928-8cb4-49a3-b999-1b90e6cb31bb" containerName="main" Oct 30 13:11:25.291102 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:25.291047 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f353928-8cb4-49a3-b999-1b90e6cb31bb" containerName="main" Oct 30 13:11:25.291102 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:25.291060 5922 cpu_manager.go:414] "containerMap: removing container" podUID="c0ea7fde-2ac5-432d-bb9b-534024186ac9" containerName="main" Oct 30 13:11:25.291102 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:25.291065 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0ea7fde-2ac5-432d-bb9b-534024186ac9" containerName="main" Oct 30 13:11:25.291202 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:25.291192 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f353928-8cb4-49a3-b999-1b90e6cb31bb" containerName="main" Oct 30 13:11:25.291202 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:25.291201 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0ea7fde-2ac5-432d-bb9b-534024186ac9" containerName="main" Oct 30 13:11:25.298562 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:25.298542 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f45f73ac-pvcfw" Oct 30 13:11:25.317914 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:25.317891 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f45f73ac-pvcfw"] Oct 30 13:11:25.324596 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:25.324573 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f5ef83c-26bfl"] Oct 30 13:11:25.329091 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:25.329072 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5ef83c-26bfl" Oct 30 13:11:25.345316 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:25.345084 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f5ef83c-26bfl"] Oct 30 13:11:25.418712 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:25.418684 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq5v6\" (UniqueName: \"kubernetes.io/projected/8b8f7b30-787c-451b-a41f-a5973359c9d6-kube-api-access-xq5v6\") pod \"maas-api-revoke-sub-7f5ef83c-26bfl\" (UID: \"8b8f7b30-787c-451b-a41f-a5973359c9d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5ef83c-26bfl" Oct 30 13:11:25.418867 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:25.418802 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b8f7b30-787c-451b-a41f-a5973359c9d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f5ef83c-26bfl\" (UID: \"8b8f7b30-787c-451b-a41f-a5973359c9d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5ef83c-26bfl" Oct 30 13:11:25.418867 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:25.418857 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b8f7b30-787c-451b-a41f-a5973359c9d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f5ef83c-26bfl\" (UID: \"8b8f7b30-787c-451b-a41f-a5973359c9d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5ef83c-26bfl" Oct 30 13:11:25.419006 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:25.418982 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/175cab09-e4aa-424b-9eaa-01c9d32d2669-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f45f73ac-pvcfw\" (UID: \"175cab09-e4aa-424b-9eaa-01c9d32d2669\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f45f73ac-pvcfw" Oct 30 13:11:25.419147 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:25.419124 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sklhk\" (UniqueName: \"kubernetes.io/projected/175cab09-e4aa-424b-9eaa-01c9d32d2669-kube-api-access-sklhk\") pod \"maas-api-revoke-sub-f45f73ac-pvcfw\" (UID: \"175cab09-e4aa-424b-9eaa-01c9d32d2669\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f45f73ac-pvcfw" Oct 30 13:11:25.419230 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:25.419162 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/175cab09-e4aa-424b-9eaa-01c9d32d2669-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f45f73ac-pvcfw\" (UID: \"175cab09-e4aa-424b-9eaa-01c9d32d2669\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f45f73ac-pvcfw" Oct 30 13:11:25.486074 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:25.486045 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:25.519843 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:25.519818 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sklhk\" (UniqueName: \"kubernetes.io/projected/175cab09-e4aa-424b-9eaa-01c9d32d2669-kube-api-access-sklhk\") pod \"maas-api-revoke-sub-f45f73ac-pvcfw\" (UID: \"175cab09-e4aa-424b-9eaa-01c9d32d2669\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f45f73ac-pvcfw" Oct 30 13:11:25.519942 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:25.519850 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/175cab09-e4aa-424b-9eaa-01c9d32d2669-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f45f73ac-pvcfw\" (UID: \"175cab09-e4aa-424b-9eaa-01c9d32d2669\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f45f73ac-pvcfw" Oct 30 13:11:25.519942 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:25.519920 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq5v6\" (UniqueName: \"kubernetes.io/projected/8b8f7b30-787c-451b-a41f-a5973359c9d6-kube-api-access-xq5v6\") pod \"maas-api-revoke-sub-7f5ef83c-26bfl\" (UID: \"8b8f7b30-787c-451b-a41f-a5973359c9d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5ef83c-26bfl" Oct 30 13:11:25.520028 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:25.520019 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b8f7b30-787c-451b-a41f-a5973359c9d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f5ef83c-26bfl\" (UID: \"8b8f7b30-787c-451b-a41f-a5973359c9d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5ef83c-26bfl" Oct 30 13:11:25.520075 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:25.520065 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b8f7b30-787c-451b-a41f-a5973359c9d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f5ef83c-26bfl\" (UID: \"8b8f7b30-787c-451b-a41f-a5973359c9d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5ef83c-26bfl" Oct 30 13:11:25.520241 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:25.520219 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/175cab09-e4aa-424b-9eaa-01c9d32d2669-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f45f73ac-pvcfw\" (UID: \"175cab09-e4aa-424b-9eaa-01c9d32d2669\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f45f73ac-pvcfw" Oct 30 13:11:25.520753 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:25.520732 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b8f7b30-787c-451b-a41f-a5973359c9d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f5ef83c-26bfl\" (UID: \"8b8f7b30-787c-451b-a41f-a5973359c9d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5ef83c-26bfl" Oct 30 13:11:25.520884 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:25.520864 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/175cab09-e4aa-424b-9eaa-01c9d32d2669-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f45f73ac-pvcfw\" (UID: \"175cab09-e4aa-424b-9eaa-01c9d32d2669\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f45f73ac-pvcfw" Oct 30 13:11:25.522281 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:25.522256 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/175cab09-e4aa-424b-9eaa-01c9d32d2669-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f45f73ac-pvcfw\" (UID: \"175cab09-e4aa-424b-9eaa-01c9d32d2669\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f45f73ac-pvcfw" Oct 30 13:11:25.522398 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:25.522383 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b8f7b30-787c-451b-a41f-a5973359c9d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f5ef83c-26bfl\" (UID: \"8b8f7b30-787c-451b-a41f-a5973359c9d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5ef83c-26bfl" Oct 30 13:11:25.544398 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:25.544350 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq5v6\" (UniqueName: \"kubernetes.io/projected/8b8f7b30-787c-451b-a41f-a5973359c9d6-kube-api-access-xq5v6\") pod \"maas-api-revoke-sub-7f5ef83c-26bfl\" (UID: \"8b8f7b30-787c-451b-a41f-a5973359c9d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5ef83c-26bfl" Oct 30 13:11:25.544510 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:25.544478 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sklhk\" (UniqueName: \"kubernetes.io/projected/175cab09-e4aa-424b-9eaa-01c9d32d2669-kube-api-access-sklhk\") pod \"maas-api-revoke-sub-f45f73ac-pvcfw\" (UID: \"175cab09-e4aa-424b-9eaa-01c9d32d2669\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f45f73ac-pvcfw" Oct 30 13:11:25.614984 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:25.614951 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f45f73ac-pvcfw" Oct 30 13:11:25.651034 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:25.651008 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5ef83c-26bfl" Oct 30 13:11:25.783506 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:25.783468 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:25.868909 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:25.868867 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0fd5fcba-gqxfm" Oct 30 13:11:25.932582 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:25.932544 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f45f73ac-pvcfw"] Oct 30 13:11:25.933247 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:25.933220 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec623696-2a8c-4ddf-8efc-310a7d683722-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec623696-2a8c-4ddf-8efc-310a7d683722-maas-api-xks-ca\") pod \"ec623696-2a8c-4ddf-8efc-310a7d683722\" (UID: \"ec623696-2a8c-4ddf-8efc-310a7d683722\") " Oct 30 13:11:25.933436 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:25.933418 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec623696-2a8c-4ddf-8efc-310a7d683722-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec623696-2a8c-4ddf-8efc-310a7d683722-maas-api-openshift-ca\") pod \"ec623696-2a8c-4ddf-8efc-310a7d683722\" (UID: \"ec623696-2a8c-4ddf-8efc-310a7d683722\") " Oct 30 13:11:25.933585 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:25.933573 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec623696-2a8c-4ddf-8efc-310a7d683722-kube-api-access-979k5\" (UniqueName: \"kubernetes.io/projected/ec623696-2a8c-4ddf-8efc-310a7d683722-kube-api-access-979k5\") pod \"ec623696-2a8c-4ddf-8efc-310a7d683722\" (UID: \"ec623696-2a8c-4ddf-8efc-310a7d683722\") " Oct 30 13:11:25.935488 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:25.935424 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec623696-2a8c-4ddf-8efc-310a7d683722-maas-api-openshift-ca" pod "ec623696-2a8c-4ddf-8efc-310a7d683722" (UID: "ec623696-2a8c-4ddf-8efc-310a7d683722"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:25.936151 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:25.936105 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec623696-2a8c-4ddf-8efc-310a7d683722-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:25.940922 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:25.940814 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec623696-2a8c-4ddf-8efc-310a7d683722-kube-api-access-979k5" pod "ec623696-2a8c-4ddf-8efc-310a7d683722" (UID: "ec623696-2a8c-4ddf-8efc-310a7d683722"). InnerVolumeSpecName "kube-api-access-979k5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:25.951758 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:25.951734 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec623696-2a8c-4ddf-8efc-310a7d683722-maas-api-xks-ca" pod "ec623696-2a8c-4ddf-8efc-310a7d683722" (UID: "ec623696-2a8c-4ddf-8efc-310a7d683722"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:11:25.982410 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:25.982382 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f5ef83c-26bfl"] Oct 30 13:11:25.983030 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:11:25.983007 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b8f7b30_787c_451b_a41f_a5973359c9d6.slice/crio-1b4de6a2d10ca9e07285ae2699b42f1dc7bab04109303dc1d465c164f6a12164 WatchSource:0}: Error finding container 1b4de6a2d10ca9e07285ae2699b42f1dc7bab04109303dc1d465c164f6a12164: Status 404 returned error can't find the container with id 1b4de6a2d10ca9e07285ae2699b42f1dc7bab04109303dc1d465c164f6a12164 Oct 30 13:11:26.037173 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:26.037144 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec623696-2a8c-4ddf-8efc-310a7d683722-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:26.037263 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:26.037175 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-979k5\" (UniqueName: \"kubernetes.io/projected/ec623696-2a8c-4ddf-8efc-310a7d683722-kube-api-access-979k5\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:26.663069 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:26.663035 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f45f73ac-pvcfw" event={"ID":"175cab09-e4aa-424b-9eaa-01c9d32d2669","Type":"ContainerStarted","Data":"73b134027f26a17722b5ee94dcaeb7d4c8becfd5d136644bdd15c373f28d035f"} Oct 30 13:11:26.663457 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:26.663093 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f45f73ac-pvcfw" event={"ID":"175cab09-e4aa-424b-9eaa-01c9d32d2669","Type":"ContainerStarted","Data":"169c247f96b00df8d6a6aba1e4561a8bbc478147e8ad3194b979ed700546fe2c"} Oct 30 13:11:26.665007 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:26.664976 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5ef83c-26bfl" event={"ID":"8b8f7b30-787c-451b-a41f-a5973359c9d6","Type":"ContainerStarted","Data":"5c97d255df6aee6196e2feb25d70e4bedc1d130440fb12c1963533329d3e42ff"} Oct 30 13:11:26.665135 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:26.665027 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5ef83c-26bfl" event={"ID":"8b8f7b30-787c-451b-a41f-a5973359c9d6","Type":"ContainerStarted","Data":"1b4de6a2d10ca9e07285ae2699b42f1dc7bab04109303dc1d465c164f6a12164"} Oct 30 13:11:26.666661 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:26.666639 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0fd5fcba-gqxfm" Oct 30 13:11:26.666765 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:26.666637 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0fd5fcba-gqxfm" event={"ID":"ec623696-2a8c-4ddf-8efc-310a7d683722","Type":"ContainerDied","Data":"840141db3358012fb499d96b98ebb9c22edf3608e344c49ae0315c9d067454ca"} Oct 30 13:11:26.666765 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:26.666757 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="840141db3358012fb499d96b98ebb9c22edf3608e344c49ae0315c9d067454ca" Oct 30 13:11:26.707790 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:26.707751 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f45f73ac-pvcfw" podStartSLOduration=1.7077391629999998 podStartE2EDuration="1.707739163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:26.705678872 +0000 UTC m=+1143.175094617" watchObservedRunningTime="2026-10-30 13:11:26.707739163 +0000 UTC m=+1143.177154907" Oct 30 13:11:26.752596 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:26.752557 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5ef83c-26bfl" podStartSLOduration=1.752546019 podStartE2EDuration="1.752546019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:26.748767522 +0000 UTC m=+1143.218183247" watchObservedRunningTime="2026-10-30 13:11:26.752546019 +0000 UTC m=+1143.221961764" Oct 30 13:11:27.046396 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:11:27.046364 5922 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b8f7b30_787c_451b_a41f_a5973359c9d6.slice/crio-conmon-5c97d255df6aee6196e2feb25d70e4bedc1d130440fb12c1963533329d3e42ff.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod175cab09_e4aa_424b_9eaa_01c9d32d2669.slice/crio-73b134027f26a17722b5ee94dcaeb7d4c8becfd5d136644bdd15c373f28d035f.scope\": RecentStats: unable to find data in memory cache]" Oct 30 13:11:27.256471 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:27.256434 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3048f9-fc55696f5-j6t4d" podUID="68a59f39-95e5-4e6c-b589-bf0cb560a12c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:11:27.503146 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:27.503111 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:27.677345 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:27.677229 5922 generic.go:356] "Generic (PLEG): container finished" podID="8b8f7b30-787c-451b-a41f-a5973359c9d6" containerID="5c97d255df6aee6196e2feb25d70e4bedc1d130440fb12c1963533329d3e42ff" exitCode=0 Oct 30 13:11:27.678030 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:27.677349 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5ef83c-26bfl" event={"ID":"8b8f7b30-787c-451b-a41f-a5973359c9d6","Type":"ContainerDied","Data":"5c97d255df6aee6196e2feb25d70e4bedc1d130440fb12c1963533329d3e42ff"} Oct 30 13:11:27.679714 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:27.679688 5922 generic.go:356] "Generic (PLEG): container finished" podID="175cab09-e4aa-424b-9eaa-01c9d32d2669" containerID="73b134027f26a17722b5ee94dcaeb7d4c8becfd5d136644bdd15c373f28d035f" exitCode=0 Oct 30 13:11:27.679879 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:27.679852 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f45f73ac-pvcfw" event={"ID":"175cab09-e4aa-424b-9eaa-01c9d32d2669","Type":"ContainerDied","Data":"73b134027f26a17722b5ee94dcaeb7d4c8becfd5d136644bdd15c373f28d035f"} Oct 30 13:11:28.296631 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:28.294355 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:28.542050 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:28.542017 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:28.607700 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:28.607595 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42ee9d-6trfh"] Oct 30 13:11:28.616296 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:28.616258 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-42ee9d-6trfh"] Oct 30 13:11:28.795907 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:28.795841 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:28.917343 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:28.917314 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5ef83c-26bfl" Oct 30 13:11:28.921821 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:28.921801 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f45f73ac-pvcfw" Oct 30 13:11:29.068807 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:29.068771 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:29.088139 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:29.088112 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/175cab09-e4aa-424b-9eaa-01c9d32d2669-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/175cab09-e4aa-424b-9eaa-01c9d32d2669-maas-api-xks-ca\") pod \"175cab09-e4aa-424b-9eaa-01c9d32d2669\" (UID: \"175cab09-e4aa-424b-9eaa-01c9d32d2669\") " Oct 30 13:11:29.088251 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:29.088150 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b8f7b30-787c-451b-a41f-a5973359c9d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b8f7b30-787c-451b-a41f-a5973359c9d6-maas-api-xks-ca\") pod \"8b8f7b30-787c-451b-a41f-a5973359c9d6\" (UID: \"8b8f7b30-787c-451b-a41f-a5973359c9d6\") " Oct 30 13:11:29.088251 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:29.088217 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/175cab09-e4aa-424b-9eaa-01c9d32d2669-kube-api-access-sklhk\" (UniqueName: \"kubernetes.io/projected/175cab09-e4aa-424b-9eaa-01c9d32d2669-kube-api-access-sklhk\") pod \"175cab09-e4aa-424b-9eaa-01c9d32d2669\" (UID: \"175cab09-e4aa-424b-9eaa-01c9d32d2669\") " Oct 30 13:11:29.088340 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:29.088308 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b8f7b30-787c-451b-a41f-a5973359c9d6-kube-api-access-xq5v6\" (UniqueName: \"kubernetes.io/projected/8b8f7b30-787c-451b-a41f-a5973359c9d6-kube-api-access-xq5v6\") pod \"8b8f7b30-787c-451b-a41f-a5973359c9d6\" (UID: \"8b8f7b30-787c-451b-a41f-a5973359c9d6\") " Oct 30 13:11:29.088375 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:29.088347 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b8f7b30-787c-451b-a41f-a5973359c9d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b8f7b30-787c-451b-a41f-a5973359c9d6-maas-api-openshift-ca\") pod \"8b8f7b30-787c-451b-a41f-a5973359c9d6\" (UID: \"8b8f7b30-787c-451b-a41f-a5973359c9d6\") " Oct 30 13:11:29.088566 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:29.088527 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/175cab09-e4aa-424b-9eaa-01c9d32d2669-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/175cab09-e4aa-424b-9eaa-01c9d32d2669-maas-api-openshift-ca\") pod \"175cab09-e4aa-424b-9eaa-01c9d32d2669\" (UID: \"175cab09-e4aa-424b-9eaa-01c9d32d2669\") " Oct 30 13:11:29.088787 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:29.088746 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b8f7b30-787c-451b-a41f-a5973359c9d6-maas-api-openshift-ca" pod "8b8f7b30-787c-451b-a41f-a5973359c9d6" (UID: "8b8f7b30-787c-451b-a41f-a5973359c9d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:29.089157 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:29.089036 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/175cab09-e4aa-424b-9eaa-01c9d32d2669-maas-api-openshift-ca" pod "175cab09-e4aa-424b-9eaa-01c9d32d2669" (UID: "175cab09-e4aa-424b-9eaa-01c9d32d2669"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:29.089380 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:29.089360 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b8f7b30-787c-451b-a41f-a5973359c9d6-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:29.089433 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:29.089385 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/175cab09-e4aa-424b-9eaa-01c9d32d2669-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:29.090523 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:29.090499 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/175cab09-e4aa-424b-9eaa-01c9d32d2669-maas-api-xks-ca" pod "175cab09-e4aa-424b-9eaa-01c9d32d2669" (UID: "175cab09-e4aa-424b-9eaa-01c9d32d2669"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:11:29.090523 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:29.090516 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b8f7b30-787c-451b-a41f-a5973359c9d6-kube-api-access-xq5v6" pod "8b8f7b30-787c-451b-a41f-a5973359c9d6" (UID: "8b8f7b30-787c-451b-a41f-a5973359c9d6"). InnerVolumeSpecName "kube-api-access-xq5v6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:29.090793 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:29.090774 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b8f7b30-787c-451b-a41f-a5973359c9d6-maas-api-xks-ca" pod "8b8f7b30-787c-451b-a41f-a5973359c9d6" (UID: "8b8f7b30-787c-451b-a41f-a5973359c9d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:11:29.090864 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:29.090848 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/175cab09-e4aa-424b-9eaa-01c9d32d2669-kube-api-access-sklhk" pod "175cab09-e4aa-424b-9eaa-01c9d32d2669" (UID: "175cab09-e4aa-424b-9eaa-01c9d32d2669"). InnerVolumeSpecName "kube-api-access-sklhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:29.190460 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:29.190394 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/175cab09-e4aa-424b-9eaa-01c9d32d2669-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:29.190460 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:29.190422 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b8f7b30-787c-451b-a41f-a5973359c9d6-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:29.190460 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:29.190435 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sklhk\" (UniqueName: \"kubernetes.io/projected/175cab09-e4aa-424b-9eaa-01c9d32d2669-kube-api-access-sklhk\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:29.190460 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:29.190449 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq5v6\" (UniqueName: \"kubernetes.io/projected/8b8f7b30-787c-451b-a41f-a5973359c9d6-kube-api-access-xq5v6\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:29.646973 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:29.646934 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4" path="/var/lib/kubelet/pods/acaa09b7-dfe3-474b-b0dd-2198bd2ab7d4/volumes" Oct 30 13:11:29.696707 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:29.696680 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f45f73ac-pvcfw" Oct 30 13:11:29.696819 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:29.696725 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f45f73ac-pvcfw" event={"ID":"175cab09-e4aa-424b-9eaa-01c9d32d2669","Type":"ContainerDied","Data":"169c247f96b00df8d6a6aba1e4561a8bbc478147e8ad3194b979ed700546fe2c"} Oct 30 13:11:29.696819 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:29.696770 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="169c247f96b00df8d6a6aba1e4561a8bbc478147e8ad3194b979ed700546fe2c" Oct 30 13:11:29.698248 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:29.698230 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5ef83c-26bfl" Oct 30 13:11:29.698385 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:29.698276 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5ef83c-26bfl" event={"ID":"8b8f7b30-787c-451b-a41f-a5973359c9d6","Type":"ContainerDied","Data":"1b4de6a2d10ca9e07285ae2699b42f1dc7bab04109303dc1d465c164f6a12164"} Oct 30 13:11:29.698385 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:29.698311 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b4de6a2d10ca9e07285ae2699b42f1dc7bab04109303dc1d465c164f6a12164" Oct 30 13:11:30.852882 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:30.852845 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:32.479314 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:32.479272 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0fd5fcba-7b5c755894-dh5cw"] Oct 30 13:11:32.516591 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:32.516558 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0fd5fcba-688fdb5fb7-m2s7t"] Oct 30 13:11:32.643317 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:32.643280 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:32.778026 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:32.777935 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:33.058668 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:33.058574 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:33.225635 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:33.224449 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:33.424594 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:33.424498 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:34.614517 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:34.614476 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:36.572751 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:36.572711 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:36.775962 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:36.775928 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:36.891488 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:36.891386 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:37.148020 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:37.147938 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q"] Oct 30 13:11:37.148254 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:37.148219 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" podUID="983e3a6f-7d5b-49d1-ab1f-560b1f59c935" containerName="istio-proxy" containerID="cri-o://7f729fa6f134d11c0dacf3d27d042882c0fd0df9dbfd29853a12b457ecd06dff" gracePeriod=30 Oct 30 13:11:37.161036 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:37.161005 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l"] Oct 30 13:11:37.161246 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:37.161215 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" podUID="9126994e-0a2a-4558-a255-bc3f01359322" containerName="istio-proxy" containerID="cri-o://f853776dc96b270a425160f55cd9ec498be7ed2caee6a8614d173fbda5a57667" gracePeriod=30 Oct 30 13:11:37.167940 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:37.167918 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk"] Oct 30 13:11:37.168146 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:37.168126 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" podUID="9f0665fa-cce6-4a67-97c7-88982d036ac8" containerName="istio-proxy" containerID="cri-o://02e0c93506f7ac881565bd42ac37a7ba949eb12d9e29e0795dd0340f8c79a603" gracePeriod=30 Oct 30 13:11:37.175847 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:37.175822 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:37.257654 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:37.257579 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3048f9-fc55696f5-j6t4d" podUID="68a59f39-95e5-4e6c-b589-bf0cb560a12c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:11:37.479951 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:37.479903 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-0fd5fcba-7b5c755894-dh5cw" podUID="8ead3af3-ce7e-40db-988e-d2bcd10fa01f" containerName="payload-processing" containerID="cri-o://ff770e3fee9d14b64762865721f17a4a98e6fe0e658b040aff28317c4c998776" gracePeriod=25 Oct 30 13:11:37.517505 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:37.517462 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0fd5fcba-688fdb5fb7-m2s7t" podUID="4cd060f1-18db-4577-b842-09a746874192" containerName="payload-pre-processing" containerID="cri-o://9950d3b2c01caa7558fa7ba5f65dc5c10fd781dd9ae65d5ad52c8f2197f6bb47" gracePeriod=25 Oct 30 13:11:37.769741 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:37.769702 5922 generic.go:356] "Generic (PLEG): container finished" podID="8ead3af3-ce7e-40db-988e-d2bcd10fa01f" containerID="ff770e3fee9d14b64762865721f17a4a98e6fe0e658b040aff28317c4c998776" exitCode=0 Oct 30 13:11:37.770164 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:37.769914 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-0fd5fcba-7b5c755894-dh5cw" event={"ID":"8ead3af3-ce7e-40db-988e-d2bcd10fa01f","Type":"ContainerDied","Data":"ff770e3fee9d14b64762865721f17a4a98e6fe0e658b040aff28317c4c998776"} Oct 30 13:11:37.772186 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:37.772159 5922 generic.go:356] "Generic (PLEG): container finished" podID="4cd060f1-18db-4577-b842-09a746874192" containerID="9950d3b2c01caa7558fa7ba5f65dc5c10fd781dd9ae65d5ad52c8f2197f6bb47" exitCode=0 Oct 30 13:11:37.772305 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:37.772199 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0fd5fcba-688fdb5fb7-m2s7t" event={"ID":"4cd060f1-18db-4577-b842-09a746874192","Type":"ContainerDied","Data":"9950d3b2c01caa7558fa7ba5f65dc5c10fd781dd9ae65d5ad52c8f2197f6bb47"} Oct 30 13:11:37.806738 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:37.806712 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-0fd5fcba-7b5c755894-dh5cw" Oct 30 13:11:37.815137 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:37.815113 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ead3af3-ce7e-40db-988e-d2bcd10fa01f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ead3af3-ce7e-40db-988e-d2bcd10fa01f-plugins-config-volume\") pod \"8ead3af3-ce7e-40db-988e-d2bcd10fa01f\" (UID: \"8ead3af3-ce7e-40db-988e-d2bcd10fa01f\") " Oct 30 13:11:37.815137 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:37.815146 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ead3af3-ce7e-40db-988e-d2bcd10fa01f-kube-api-access-nlxtq\" (UniqueName: \"kubernetes.io/projected/8ead3af3-ce7e-40db-988e-d2bcd10fa01f-kube-api-access-nlxtq\") pod \"8ead3af3-ce7e-40db-988e-d2bcd10fa01f\" (UID: \"8ead3af3-ce7e-40db-988e-d2bcd10fa01f\") " Oct 30 13:11:37.815452 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:37.815431 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ead3af3-ce7e-40db-988e-d2bcd10fa01f-plugins-config-volume" pod "8ead3af3-ce7e-40db-988e-d2bcd10fa01f" (UID: "8ead3af3-ce7e-40db-988e-d2bcd10fa01f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:37.815773 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:37.815748 5922 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ead3af3-ce7e-40db-988e-d2bcd10fa01f-plugins-config-volume\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:37.817316 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:37.817295 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ead3af3-ce7e-40db-988e-d2bcd10fa01f-kube-api-access-nlxtq" pod "8ead3af3-ce7e-40db-988e-d2bcd10fa01f" (UID: "8ead3af3-ce7e-40db-988e-d2bcd10fa01f"). InnerVolumeSpecName "kube-api-access-nlxtq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:37.836335 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:37.836314 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0fd5fcba-688fdb5fb7-m2s7t" Oct 30 13:11:37.916974 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:37.916934 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cd060f1-18db-4577-b842-09a746874192-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd060f1-18db-4577-b842-09a746874192-plugins-config-volume\") pod \"4cd060f1-18db-4577-b842-09a746874192\" (UID: \"4cd060f1-18db-4577-b842-09a746874192\") " Oct 30 13:11:37.917171 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:37.917071 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cd060f1-18db-4577-b842-09a746874192-kube-api-access-58dgf\" (UniqueName: \"kubernetes.io/projected/4cd060f1-18db-4577-b842-09a746874192-kube-api-access-58dgf\") pod \"4cd060f1-18db-4577-b842-09a746874192\" (UID: \"4cd060f1-18db-4577-b842-09a746874192\") " Oct 30 13:11:37.917374 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:37.917346 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cd060f1-18db-4577-b842-09a746874192-plugins-config-volume" pod "4cd060f1-18db-4577-b842-09a746874192" (UID: "4cd060f1-18db-4577-b842-09a746874192"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:37.917624 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:37.917592 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlxtq\" (UniqueName: \"kubernetes.io/projected/8ead3af3-ce7e-40db-988e-d2bcd10fa01f-kube-api-access-nlxtq\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:37.917692 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:37.917624 5922 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd060f1-18db-4577-b842-09a746874192-plugins-config-volume\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:37.919234 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:37.919211 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cd060f1-18db-4577-b842-09a746874192-kube-api-access-58dgf" pod "4cd060f1-18db-4577-b842-09a746874192" (UID: "4cd060f1-18db-4577-b842-09a746874192"). InnerVolumeSpecName "kube-api-access-58dgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:38.018584 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:38.018527 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58dgf\" (UniqueName: \"kubernetes.io/projected/4cd060f1-18db-4577-b842-09a746874192-kube-api-access-58dgf\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:38.197677 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:38.197652 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3048f9-fc55696f5-j6t4d" Oct 30 13:11:38.220700 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:38.220676 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68a59f39-95e5-4e6c-b589-bf0cb560a12c-kube-api-access-bfr42\" (UniqueName: \"kubernetes.io/projected/68a59f39-95e5-4e6c-b589-bf0cb560a12c-kube-api-access-bfr42\") pod \"68a59f39-95e5-4e6c-b589-bf0cb560a12c\" (UID: \"68a59f39-95e5-4e6c-b589-bf0cb560a12c\") " Oct 30 13:11:38.220798 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:38.220733 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68a59f39-95e5-4e6c-b589-bf0cb560a12c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68a59f39-95e5-4e6c-b589-bf0cb560a12c-plugins-config-volume\") pod \"68a59f39-95e5-4e6c-b589-bf0cb560a12c\" (UID: \"68a59f39-95e5-4e6c-b589-bf0cb560a12c\") " Oct 30 13:11:38.221028 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:38.221008 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68a59f39-95e5-4e6c-b589-bf0cb560a12c-plugins-config-volume" pod "68a59f39-95e5-4e6c-b589-bf0cb560a12c" (UID: "68a59f39-95e5-4e6c-b589-bf0cb560a12c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:38.221275 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:38.221258 5922 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68a59f39-95e5-4e6c-b589-bf0cb560a12c-plugins-config-volume\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:38.222728 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:38.222711 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68a59f39-95e5-4e6c-b589-bf0cb560a12c-kube-api-access-bfr42" pod "68a59f39-95e5-4e6c-b589-bf0cb560a12c" (UID: "68a59f39-95e5-4e6c-b589-bf0cb560a12c"). InnerVolumeSpecName "kube-api-access-bfr42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:38.329533 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:38.329446 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfr42\" (UniqueName: \"kubernetes.io/projected/68a59f39-95e5-4e6c-b589-bf0cb560a12c-kube-api-access-bfr42\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:38.510845 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:38.510808 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:38.781423 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:38.781390 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0fd5fcba-688fdb5fb7-m2s7t" Oct 30 13:11:38.781423 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:38.781393 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0fd5fcba-688fdb5fb7-m2s7t" event={"ID":"4cd060f1-18db-4577-b842-09a746874192","Type":"ContainerDied","Data":"a1f2b6c853abcf96323035fbd196edf96c42716f1340e2b1e1c93823ff748044"} Oct 30 13:11:38.781961 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:38.781456 5922 scope.go:122] "RemoveContainer" containerID="9950d3b2c01caa7558fa7ba5f65dc5c10fd781dd9ae65d5ad52c8f2197f6bb47" Oct 30 13:11:38.783082 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:38.783058 5922 generic.go:356] "Generic (PLEG): container finished" podID="68a59f39-95e5-4e6c-b589-bf0cb560a12c" containerID="ff02d73c2f9c94e9703af8ddbb47c989cf3b32dfb6f5220da125202b5c940293" exitCode=0 Oct 30 13:11:38.783195 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:38.783125 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3048f9-fc55696f5-j6t4d" Oct 30 13:11:38.783195 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:38.783134 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3048f9-fc55696f5-j6t4d" event={"ID":"68a59f39-95e5-4e6c-b589-bf0cb560a12c","Type":"ContainerDied","Data":"ff02d73c2f9c94e9703af8ddbb47c989cf3b32dfb6f5220da125202b5c940293"} Oct 30 13:11:38.783195 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:38.783190 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3048f9-fc55696f5-j6t4d" event={"ID":"68a59f39-95e5-4e6c-b589-bf0cb560a12c","Type":"ContainerDied","Data":"c46aefdc6d111ed4ecd575dff499ce76c506ba7b6bac1ba399d5830fd2b97b7a"} Oct 30 13:11:38.785869 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:38.785845 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-0fd5fcba-7b5c755894-dh5cw" event={"ID":"8ead3af3-ce7e-40db-988e-d2bcd10fa01f","Type":"ContainerDied","Data":"da02b96f70fec787640dc9efa3916169f7e132b0eef814d5b1a691fe6ad0905e"} Oct 30 13:11:38.785987 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:38.785883 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-0fd5fcba-7b5c755894-dh5cw" Oct 30 13:11:38.795655 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:38.795633 5922 scope.go:122] "RemoveContainer" containerID="ff02d73c2f9c94e9703af8ddbb47c989cf3b32dfb6f5220da125202b5c940293" Oct 30 13:11:38.806252 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:38.806236 5922 scope.go:122] "RemoveContainer" containerID="ff02d73c2f9c94e9703af8ddbb47c989cf3b32dfb6f5220da125202b5c940293" Oct 30 13:11:38.806526 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:11:38.806502 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff02d73c2f9c94e9703af8ddbb47c989cf3b32dfb6f5220da125202b5c940293\": container with ID starting with ff02d73c2f9c94e9703af8ddbb47c989cf3b32dfb6f5220da125202b5c940293 not found: ID does not exist" containerID="ff02d73c2f9c94e9703af8ddbb47c989cf3b32dfb6f5220da125202b5c940293" Oct 30 13:11:38.806590 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:38.806537 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff02d73c2f9c94e9703af8ddbb47c989cf3b32dfb6f5220da125202b5c940293"} err="failed to get container status \"ff02d73c2f9c94e9703af8ddbb47c989cf3b32dfb6f5220da125202b5c940293\": rpc error: code = NotFound desc = could not find container \"ff02d73c2f9c94e9703af8ddbb47c989cf3b32dfb6f5220da125202b5c940293\": container with ID starting with ff02d73c2f9c94e9703af8ddbb47c989cf3b32dfb6f5220da125202b5c940293 not found: ID does not exist" Oct 30 13:11:38.806590 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:38.806563 5922 scope.go:122] "RemoveContainer" containerID="ff770e3fee9d14b64762865721f17a4a98e6fe0e658b040aff28317c4c998776" Oct 30 13:11:38.848014 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:38.847992 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0fd5fcba-7b5c755894-dh5cw"] Oct 30 13:11:38.855766 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:38.855746 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0fd5fcba-7b5c755894-dh5cw"] Oct 30 13:11:38.893338 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:38.893309 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0fd5fcba-688fdb5fb7-m2s7t"] Oct 30 13:11:38.901001 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:38.900977 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0fd5fcba-688fdb5fb7-m2s7t"] Oct 30 13:11:38.937038 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:38.937012 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-3048f9-fc55696f5-j6t4d"] Oct 30 13:11:38.942986 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:38.942964 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-3048f9-fc55696f5-j6t4d"] Oct 30 13:11:39.040238 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:39.040144 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3465b539-kjn9s"] Oct 30 13:11:39.046910 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:39.046874 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3465b539-kjn9s"] Oct 30 13:11:39.647705 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:39.647672 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cd060f1-18db-4577-b842-09a746874192" path="/var/lib/kubelet/pods/4cd060f1-18db-4577-b842-09a746874192/volumes" Oct 30 13:11:39.648182 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:39.648165 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68a59f39-95e5-4e6c-b589-bf0cb560a12c" path="/var/lib/kubelet/pods/68a59f39-95e5-4e6c-b589-bf0cb560a12c/volumes" Oct 30 13:11:39.648582 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:39.648567 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bcabc28-e50c-4e82-bd30-79a43685ae14" path="/var/lib/kubelet/pods/8bcabc28-e50c-4e82-bd30-79a43685ae14/volumes" Oct 30 13:11:39.649275 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:39.649257 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ead3af3-ce7e-40db-988e-d2bcd10fa01f" path="/var/lib/kubelet/pods/8ead3af3-ce7e-40db-988e-d2bcd10fa01f/volumes" Oct 30 13:11:40.130785 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:40.130750 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:40.557177 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:40.557138 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:40.785642 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:40.784237 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:40.940813 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:40.940720 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:40.985064 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:40.985028 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3048f9-j6df9"] Oct 30 13:11:40.997605 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:40.997579 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3048f9-j6df9"] Oct 30 13:11:41.058789 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:41.058661 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:41.458630 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:41.458589 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:41.645989 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:41.645959 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="757485a1-9f1f-472a-a000-0f4f6b2ab14e" path="/var/lib/kubelet/pods/757485a1-9f1f-472a-a000-0f4f6b2ab14e/volumes" Oct 30 13:11:42.454871 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.454846 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" Oct 30 13:11:42.495254 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.495230 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" Oct 30 13:11:42.514945 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.514916 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-credential-socket\") pod \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\" (UID: \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\") " Oct 30 13:11:42.515099 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.515033 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-workload-socket\") pod \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\" (UID: \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\") " Oct 30 13:11:42.515169 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.515135 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-istio-envoy\") pod \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\" (UID: \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\") " Oct 30 13:11:42.515227 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.515174 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-istio-podinfo\") pod \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\" (UID: \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\") " Oct 30 13:11:42.515328 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.515292 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-istio-token\" (UniqueName: \"kubernetes.io/projected/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-istio-token\") pod \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\" (UID: \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\") " Oct 30 13:11:42.515388 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.515349 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-workload-certs\") pod \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\" (UID: \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\") " Oct 30 13:11:42.515647 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.515608 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-workload-certs" pod "983e3a6f-7d5b-49d1-ab1f-560b1f59c935" (UID: "983e3a6f-7d5b-49d1-ab1f-560b1f59c935"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:42.515766 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.515678 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-istio-data\") pod \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\" (UID: \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\") " Oct 30 13:11:42.515766 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.515741 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-istiod-ca-cert\") pod \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\" (UID: \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\") " Oct 30 13:11:42.515923 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.515785 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-kube-api-access-sbk57\" (UniqueName: \"kubernetes.io/projected/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-kube-api-access-sbk57\") pod \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\" (UID: \"983e3a6f-7d5b-49d1-ab1f-560b1f59c935\") " Oct 30 13:11:42.516392 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.516366 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-istio-data" pod "983e3a6f-7d5b-49d1-ab1f-560b1f59c935" (UID: "983e3a6f-7d5b-49d1-ab1f-560b1f59c935"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:42.516471 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.516418 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-workload-socket" pod "983e3a6f-7d5b-49d1-ab1f-560b1f59c935" (UID: "983e3a6f-7d5b-49d1-ab1f-560b1f59c935"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:42.516579 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.516480 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-credential-socket" pod "983e3a6f-7d5b-49d1-ab1f-560b1f59c935" (UID: "983e3a6f-7d5b-49d1-ab1f-560b1f59c935"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:42.516803 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.516673 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-istiod-ca-cert" pod "983e3a6f-7d5b-49d1-ab1f-560b1f59c935" (UID: "983e3a6f-7d5b-49d1-ab1f-560b1f59c935"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:42.517056 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.517037 5922 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-workload-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.517056 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.517055 5922 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-workload-certs\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.517233 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.517075 5922 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-istio-data\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.517233 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.517086 5922 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-istiod-ca-cert\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.517233 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.517099 5922 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-credential-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.518425 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.518354 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-istio-envoy" pod "983e3a6f-7d5b-49d1-ab1f-560b1f59c935" (UID: "983e3a6f-7d5b-49d1-ab1f-560b1f59c935"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:42.518941 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.518903 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-istio-podinfo" pod "983e3a6f-7d5b-49d1-ab1f-560b1f59c935" (UID: "983e3a6f-7d5b-49d1-ab1f-560b1f59c935"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:11:42.519474 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.519439 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-istio-token" pod "983e3a6f-7d5b-49d1-ab1f-560b1f59c935" (UID: "983e3a6f-7d5b-49d1-ab1f-560b1f59c935"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:42.519565 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.519521 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-kube-api-access-sbk57" pod "983e3a6f-7d5b-49d1-ab1f-560b1f59c935" (UID: "983e3a6f-7d5b-49d1-ab1f-560b1f59c935"). InnerVolumeSpecName "kube-api-access-sbk57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:42.536349 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.536322 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" Oct 30 13:11:42.617973 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.617934 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f0665fa-cce6-4a67-97c7-88982d036ac8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f0665fa-cce6-4a67-97c7-88982d036ac8-credential-socket\") pod \"9f0665fa-cce6-4a67-97c7-88982d036ac8\" (UID: \"9f0665fa-cce6-4a67-97c7-88982d036ac8\") " Oct 30 13:11:42.618153 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.618004 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9126994e-0a2a-4558-a255-bc3f01359322-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9126994e-0a2a-4558-a255-bc3f01359322-credential-socket\") pod \"9126994e-0a2a-4558-a255-bc3f01359322\" (UID: \"9126994e-0a2a-4558-a255-bc3f01359322\") " Oct 30 13:11:42.618153 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.618051 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f0665fa-cce6-4a67-97c7-88982d036ac8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f0665fa-cce6-4a67-97c7-88982d036ac8-istio-data\") pod \"9f0665fa-cce6-4a67-97c7-88982d036ac8\" (UID: \"9f0665fa-cce6-4a67-97c7-88982d036ac8\") " Oct 30 13:11:42.618153 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.618081 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9126994e-0a2a-4558-a255-bc3f01359322-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9126994e-0a2a-4558-a255-bc3f01359322-workload-socket\") pod \"9126994e-0a2a-4558-a255-bc3f01359322\" (UID: \"9126994e-0a2a-4558-a255-bc3f01359322\") " Oct 30 13:11:42.618311 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.618174 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9126994e-0a2a-4558-a255-bc3f01359322-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9126994e-0a2a-4558-a255-bc3f01359322-istio-data\") pod \"9126994e-0a2a-4558-a255-bc3f01359322\" (UID: \"9126994e-0a2a-4558-a255-bc3f01359322\") " Oct 30 13:11:42.618311 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.618278 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9126994e-0a2a-4558-a255-bc3f01359322-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9126994e-0a2a-4558-a255-bc3f01359322-istio-podinfo\") pod \"9126994e-0a2a-4558-a255-bc3f01359322\" (UID: \"9126994e-0a2a-4558-a255-bc3f01359322\") " Oct 30 13:11:42.618405 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.618307 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f0665fa-cce6-4a67-97c7-88982d036ac8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f0665fa-cce6-4a67-97c7-88982d036ac8-istio-podinfo\") pod \"9f0665fa-cce6-4a67-97c7-88982d036ac8\" (UID: \"9f0665fa-cce6-4a67-97c7-88982d036ac8\") " Oct 30 13:11:42.618405 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.618346 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9126994e-0a2a-4558-a255-bc3f01359322-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9126994e-0a2a-4558-a255-bc3f01359322-istio-envoy\") pod \"9126994e-0a2a-4558-a255-bc3f01359322\" (UID: \"9126994e-0a2a-4558-a255-bc3f01359322\") " Oct 30 13:11:42.618405 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.618386 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9126994e-0a2a-4558-a255-bc3f01359322-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9126994e-0a2a-4558-a255-bc3f01359322-istiod-ca-cert\") pod \"9126994e-0a2a-4558-a255-bc3f01359322\" (UID: \"9126994e-0a2a-4558-a255-bc3f01359322\") " Oct 30 13:11:42.618574 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.618421 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9126994e-0a2a-4558-a255-bc3f01359322-istio-token\" (UniqueName: \"kubernetes.io/projected/9126994e-0a2a-4558-a255-bc3f01359322-istio-token\") pod \"9126994e-0a2a-4558-a255-bc3f01359322\" (UID: \"9126994e-0a2a-4558-a255-bc3f01359322\") " Oct 30 13:11:42.618574 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.618453 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f0665fa-cce6-4a67-97c7-88982d036ac8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f0665fa-cce6-4a67-97c7-88982d036ac8-workload-socket\") pod \"9f0665fa-cce6-4a67-97c7-88982d036ac8\" (UID: \"9f0665fa-cce6-4a67-97c7-88982d036ac8\") " Oct 30 13:11:42.618574 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.618506 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f0665fa-cce6-4a67-97c7-88982d036ac8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f0665fa-cce6-4a67-97c7-88982d036ac8-workload-certs\") pod \"9f0665fa-cce6-4a67-97c7-88982d036ac8\" (UID: \"9f0665fa-cce6-4a67-97c7-88982d036ac8\") " Oct 30 13:11:42.618759 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.618585 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f0665fa-cce6-4a67-97c7-88982d036ac8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f0665fa-cce6-4a67-97c7-88982d036ac8-istiod-ca-cert\") pod \"9f0665fa-cce6-4a67-97c7-88982d036ac8\" (UID: \"9f0665fa-cce6-4a67-97c7-88982d036ac8\") " Oct 30 13:11:42.618759 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.618642 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f0665fa-cce6-4a67-97c7-88982d036ac8-istio-token\" (UniqueName: \"kubernetes.io/projected/9f0665fa-cce6-4a67-97c7-88982d036ac8-istio-token\") pod \"9f0665fa-cce6-4a67-97c7-88982d036ac8\" (UID: \"9f0665fa-cce6-4a67-97c7-88982d036ac8\") " Oct 30 13:11:42.618759 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.618676 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9126994e-0a2a-4558-a255-bc3f01359322-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9126994e-0a2a-4558-a255-bc3f01359322-workload-certs\") pod \"9126994e-0a2a-4558-a255-bc3f01359322\" (UID: \"9126994e-0a2a-4558-a255-bc3f01359322\") " Oct 30 13:11:42.618759 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.618706 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9126994e-0a2a-4558-a255-bc3f01359322-kube-api-access-lcwkq\" (UniqueName: \"kubernetes.io/projected/9126994e-0a2a-4558-a255-bc3f01359322-kube-api-access-lcwkq\") pod \"9126994e-0a2a-4558-a255-bc3f01359322\" (UID: \"9126994e-0a2a-4558-a255-bc3f01359322\") " Oct 30 13:11:42.618759 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.618746 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f0665fa-cce6-4a67-97c7-88982d036ac8-kube-api-access-8kt7b\" (UniqueName: \"kubernetes.io/projected/9f0665fa-cce6-4a67-97c7-88982d036ac8-kube-api-access-8kt7b\") pod \"9f0665fa-cce6-4a67-97c7-88982d036ac8\" (UID: \"9f0665fa-cce6-4a67-97c7-88982d036ac8\") " Oct 30 13:11:42.619019 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.618777 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f0665fa-cce6-4a67-97c7-88982d036ac8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f0665fa-cce6-4a67-97c7-88982d036ac8-istio-envoy\") pod \"9f0665fa-cce6-4a67-97c7-88982d036ac8\" (UID: \"9f0665fa-cce6-4a67-97c7-88982d036ac8\") " Oct 30 13:11:42.619584 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.619559 5922 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-istio-token\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.619725 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.619589 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbk57\" (UniqueName: \"kubernetes.io/projected/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-kube-api-access-sbk57\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.619725 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.619604 5922 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-istio-envoy\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.619725 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.619633 5922 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/983e3a6f-7d5b-49d1-ab1f-560b1f59c935-istio-podinfo\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.621743 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.620734 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9126994e-0a2a-4558-a255-bc3f01359322-istiod-ca-cert" pod "9126994e-0a2a-4558-a255-bc3f01359322" (UID: "9126994e-0a2a-4558-a255-bc3f01359322"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:42.621743 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.621006 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f0665fa-cce6-4a67-97c7-88982d036ac8-credential-socket" pod "9f0665fa-cce6-4a67-97c7-88982d036ac8" (UID: "9f0665fa-cce6-4a67-97c7-88982d036ac8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:42.621743 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.621163 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9126994e-0a2a-4558-a255-bc3f01359322-credential-socket" pod "9126994e-0a2a-4558-a255-bc3f01359322" (UID: "9126994e-0a2a-4558-a255-bc3f01359322"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:42.621743 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.621280 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f0665fa-cce6-4a67-97c7-88982d036ac8-istio-data" pod "9f0665fa-cce6-4a67-97c7-88982d036ac8" (UID: "9f0665fa-cce6-4a67-97c7-88982d036ac8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:42.621743 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.621404 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9126994e-0a2a-4558-a255-bc3f01359322-workload-socket" pod "9126994e-0a2a-4558-a255-bc3f01359322" (UID: "9126994e-0a2a-4558-a255-bc3f01359322"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:42.621743 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.621510 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9126994e-0a2a-4558-a255-bc3f01359322-istio-data" pod "9126994e-0a2a-4558-a255-bc3f01359322" (UID: "9126994e-0a2a-4558-a255-bc3f01359322"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:42.623520 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.623489 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f0665fa-cce6-4a67-97c7-88982d036ac8-istio-envoy" pod "9f0665fa-cce6-4a67-97c7-88982d036ac8" (UID: "9f0665fa-cce6-4a67-97c7-88982d036ac8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:42.624009 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.623818 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f0665fa-cce6-4a67-97c7-88982d036ac8-workload-socket" pod "9f0665fa-cce6-4a67-97c7-88982d036ac8" (UID: "9f0665fa-cce6-4a67-97c7-88982d036ac8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:42.624358 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.624177 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9126994e-0a2a-4558-a255-bc3f01359322-istio-podinfo" pod "9126994e-0a2a-4558-a255-bc3f01359322" (UID: "9126994e-0a2a-4558-a255-bc3f01359322"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:11:42.624487 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.624338 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f0665fa-cce6-4a67-97c7-88982d036ac8-istiod-ca-cert" pod "9f0665fa-cce6-4a67-97c7-88982d036ac8" (UID: "9f0665fa-cce6-4a67-97c7-88982d036ac8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:42.624964 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.624600 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f0665fa-cce6-4a67-97c7-88982d036ac8-workload-certs" pod "9f0665fa-cce6-4a67-97c7-88982d036ac8" (UID: "9f0665fa-cce6-4a67-97c7-88982d036ac8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:42.624964 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.624937 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9126994e-0a2a-4558-a255-bc3f01359322-workload-certs" pod "9126994e-0a2a-4558-a255-bc3f01359322" (UID: "9126994e-0a2a-4558-a255-bc3f01359322"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:42.625512 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.625356 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9126994e-0a2a-4558-a255-bc3f01359322-istio-envoy" pod "9126994e-0a2a-4558-a255-bc3f01359322" (UID: "9126994e-0a2a-4558-a255-bc3f01359322"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:42.626544 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.626520 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f0665fa-cce6-4a67-97c7-88982d036ac8-istio-podinfo" pod "9f0665fa-cce6-4a67-97c7-88982d036ac8" (UID: "9f0665fa-cce6-4a67-97c7-88982d036ac8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:11:42.626867 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.626789 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9126994e-0a2a-4558-a255-bc3f01359322-istio-token" pod "9126994e-0a2a-4558-a255-bc3f01359322" (UID: "9126994e-0a2a-4558-a255-bc3f01359322"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:42.626867 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.626848 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9126994e-0a2a-4558-a255-bc3f01359322-kube-api-access-lcwkq" pod "9126994e-0a2a-4558-a255-bc3f01359322" (UID: "9126994e-0a2a-4558-a255-bc3f01359322"). InnerVolumeSpecName "kube-api-access-lcwkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:42.627960 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.627936 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f0665fa-cce6-4a67-97c7-88982d036ac8-kube-api-access-8kt7b" pod "9f0665fa-cce6-4a67-97c7-88982d036ac8" (UID: "9f0665fa-cce6-4a67-97c7-88982d036ac8"). InnerVolumeSpecName "kube-api-access-8kt7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:42.628060 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.628043 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f0665fa-cce6-4a67-97c7-88982d036ac8-istio-token" pod "9f0665fa-cce6-4a67-97c7-88982d036ac8" (UID: "9f0665fa-cce6-4a67-97c7-88982d036ac8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:42.721332 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.721288 5922 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9126994e-0a2a-4558-a255-bc3f01359322-istio-data\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.721332 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.721329 5922 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9126994e-0a2a-4558-a255-bc3f01359322-istio-podinfo\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.721565 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.721343 5922 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f0665fa-cce6-4a67-97c7-88982d036ac8-istio-podinfo\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.721565 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.721356 5922 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9126994e-0a2a-4558-a255-bc3f01359322-istio-envoy\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.721565 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.721369 5922 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9126994e-0a2a-4558-a255-bc3f01359322-istiod-ca-cert\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.721565 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.721381 5922 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9126994e-0a2a-4558-a255-bc3f01359322-istio-token\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.721565 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.721394 5922 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f0665fa-cce6-4a67-97c7-88982d036ac8-workload-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.721565 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.721407 5922 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f0665fa-cce6-4a67-97c7-88982d036ac8-workload-certs\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.721565 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.721422 5922 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f0665fa-cce6-4a67-97c7-88982d036ac8-istiod-ca-cert\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.721565 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.721434 5922 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f0665fa-cce6-4a67-97c7-88982d036ac8-istio-token\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.721565 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.721446 5922 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9126994e-0a2a-4558-a255-bc3f01359322-workload-certs\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.721565 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.721460 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcwkq\" (UniqueName: \"kubernetes.io/projected/9126994e-0a2a-4558-a255-bc3f01359322-kube-api-access-lcwkq\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.721565 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.721473 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kt7b\" (UniqueName: \"kubernetes.io/projected/9f0665fa-cce6-4a67-97c7-88982d036ac8-kube-api-access-8kt7b\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.721565 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.721487 5922 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f0665fa-cce6-4a67-97c7-88982d036ac8-istio-envoy\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.721565 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.721500 5922 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f0665fa-cce6-4a67-97c7-88982d036ac8-credential-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.721565 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.721513 5922 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9126994e-0a2a-4558-a255-bc3f01359322-credential-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.721565 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.721527 5922 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f0665fa-cce6-4a67-97c7-88982d036ac8-istio-data\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.721565 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.721540 5922 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9126994e-0a2a-4558-a255-bc3f01359322-workload-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.847261 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.847232 5922 generic.go:356] "Generic (PLEG): container finished" podID="9126994e-0a2a-4558-a255-bc3f01359322" containerID="f853776dc96b270a425160f55cd9ec498be7ed2caee6a8614d173fbda5a57667" exitCode=0 Oct 30 13:11:42.847452 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.847301 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" Oct 30 13:11:42.847452 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.847320 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" event={"ID":"9126994e-0a2a-4558-a255-bc3f01359322","Type":"ContainerDied","Data":"f853776dc96b270a425160f55cd9ec498be7ed2caee6a8614d173fbda5a57667"} Oct 30 13:11:42.847452 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.847381 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l" event={"ID":"9126994e-0a2a-4558-a255-bc3f01359322","Type":"ContainerDied","Data":"b4de3670c3ce9b89e890d9a9a928da79054f6549253735296e2b131aa2c37d2b"} Oct 30 13:11:42.847452 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.847412 5922 scope.go:122] "RemoveContainer" containerID="f853776dc96b270a425160f55cd9ec498be7ed2caee6a8614d173fbda5a57667" Oct 30 13:11:42.848869 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.848844 5922 generic.go:356] "Generic (PLEG): container finished" podID="983e3a6f-7d5b-49d1-ab1f-560b1f59c935" containerID="7f729fa6f134d11c0dacf3d27d042882c0fd0df9dbfd29853a12b457ecd06dff" exitCode=0 Oct 30 13:11:42.849006 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.848893 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" Oct 30 13:11:42.849006 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.848931 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" event={"ID":"983e3a6f-7d5b-49d1-ab1f-560b1f59c935","Type":"ContainerDied","Data":"7f729fa6f134d11c0dacf3d27d042882c0fd0df9dbfd29853a12b457ecd06dff"} Oct 30 13:11:42.849006 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.848979 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q" event={"ID":"983e3a6f-7d5b-49d1-ab1f-560b1f59c935","Type":"ContainerDied","Data":"91b29d19a1ceeae8d187a7d2570adeb8bcfbe7ef47b5770d917b766b95afaff8"} Oct 30 13:11:42.850349 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.850326 5922 generic.go:356] "Generic (PLEG): container finished" podID="9f0665fa-cce6-4a67-97c7-88982d036ac8" containerID="02e0c93506f7ac881565bd42ac37a7ba949eb12d9e29e0795dd0340f8c79a603" exitCode=0 Oct 30 13:11:42.850459 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.850362 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" event={"ID":"9f0665fa-cce6-4a67-97c7-88982d036ac8","Type":"ContainerDied","Data":"02e0c93506f7ac881565bd42ac37a7ba949eb12d9e29e0795dd0340f8c79a603"} Oct 30 13:11:42.850459 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.850372 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" Oct 30 13:11:42.850459 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.850400 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk" event={"ID":"9f0665fa-cce6-4a67-97c7-88982d036ac8","Type":"ContainerDied","Data":"343bcb54b492d93b9735f8283d21f5a38ebb026b5c14d7b8941f04eb533a4962"} Oct 30 13:11:42.862074 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.862055 5922 scope.go:122] "RemoveContainer" containerID="f853776dc96b270a425160f55cd9ec498be7ed2caee6a8614d173fbda5a57667" Oct 30 13:11:42.862438 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:11:42.862414 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f853776dc96b270a425160f55cd9ec498be7ed2caee6a8614d173fbda5a57667\": container with ID starting with f853776dc96b270a425160f55cd9ec498be7ed2caee6a8614d173fbda5a57667 not found: ID does not exist" containerID="f853776dc96b270a425160f55cd9ec498be7ed2caee6a8614d173fbda5a57667" Oct 30 13:11:42.862533 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.862447 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f853776dc96b270a425160f55cd9ec498be7ed2caee6a8614d173fbda5a57667"} err="failed to get container status \"f853776dc96b270a425160f55cd9ec498be7ed2caee6a8614d173fbda5a57667\": rpc error: code = NotFound desc = could not find container \"f853776dc96b270a425160f55cd9ec498be7ed2caee6a8614d173fbda5a57667\": container with ID starting with f853776dc96b270a425160f55cd9ec498be7ed2caee6a8614d173fbda5a57667 not found: ID does not exist" Oct 30 13:11:42.862533 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.862471 5922 scope.go:122] "RemoveContainer" containerID="7f729fa6f134d11c0dacf3d27d042882c0fd0df9dbfd29853a12b457ecd06dff" Oct 30 13:11:42.873935 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.873910 5922 scope.go:122] "RemoveContainer" containerID="7f729fa6f134d11c0dacf3d27d042882c0fd0df9dbfd29853a12b457ecd06dff" Oct 30 13:11:42.874204 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:11:42.874181 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f729fa6f134d11c0dacf3d27d042882c0fd0df9dbfd29853a12b457ecd06dff\": container with ID starting with 7f729fa6f134d11c0dacf3d27d042882c0fd0df9dbfd29853a12b457ecd06dff not found: ID does not exist" containerID="7f729fa6f134d11c0dacf3d27d042882c0fd0df9dbfd29853a12b457ecd06dff" Oct 30 13:11:42.874297 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.874210 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f729fa6f134d11c0dacf3d27d042882c0fd0df9dbfd29853a12b457ecd06dff"} err="failed to get container status \"7f729fa6f134d11c0dacf3d27d042882c0fd0df9dbfd29853a12b457ecd06dff\": rpc error: code = NotFound desc = could not find container \"7f729fa6f134d11c0dacf3d27d042882c0fd0df9dbfd29853a12b457ecd06dff\": container with ID starting with 7f729fa6f134d11c0dacf3d27d042882c0fd0df9dbfd29853a12b457ecd06dff not found: ID does not exist" Oct 30 13:11:42.874297 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.874226 5922 scope.go:122] "RemoveContainer" containerID="02e0c93506f7ac881565bd42ac37a7ba949eb12d9e29e0795dd0340f8c79a603" Oct 30 13:11:42.885135 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.885117 5922 scope.go:122] "RemoveContainer" containerID="02e0c93506f7ac881565bd42ac37a7ba949eb12d9e29e0795dd0340f8c79a603" Oct 30 13:11:42.885389 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:11:42.885371 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02e0c93506f7ac881565bd42ac37a7ba949eb12d9e29e0795dd0340f8c79a603\": container with ID starting with 02e0c93506f7ac881565bd42ac37a7ba949eb12d9e29e0795dd0340f8c79a603 not found: ID does not exist" containerID="02e0c93506f7ac881565bd42ac37a7ba949eb12d9e29e0795dd0340f8c79a603" Oct 30 13:11:42.885446 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.885395 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02e0c93506f7ac881565bd42ac37a7ba949eb12d9e29e0795dd0340f8c79a603"} err="failed to get container status \"02e0c93506f7ac881565bd42ac37a7ba949eb12d9e29e0795dd0340f8c79a603\": rpc error: code = NotFound desc = could not find container \"02e0c93506f7ac881565bd42ac37a7ba949eb12d9e29e0795dd0340f8c79a603\": container with ID starting with 02e0c93506f7ac881565bd42ac37a7ba949eb12d9e29e0795dd0340f8c79a603 not found: ID does not exist" Oct 30 13:11:42.908329 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.908307 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l"] Oct 30 13:11:42.912966 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.912948 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-4dc8l"] Oct 30 13:11:42.982114 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.982030 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q"] Oct 30 13:11:42.989048 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:42.989023 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-czf9q"] Oct 30 13:11:43.029348 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:43.029325 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk"] Oct 30 13:11:43.037563 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:43.037542 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-cqqnk"] Oct 30 13:11:43.054345 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:43.054322 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1b3044b-8p7s2"] Oct 30 13:11:43.058111 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:43.058092 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1b3044b-8p7s2"] Oct 30 13:11:43.283280 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:43.283152 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:43.645996 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:43.645919 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="692066d9-7507-4e4f-95aa-60b4fef097cb" path="/var/lib/kubelet/pods/692066d9-7507-4e4f-95aa-60b4fef097cb/volumes" Oct 30 13:11:43.646471 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:43.646453 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9126994e-0a2a-4558-a255-bc3f01359322" path="/var/lib/kubelet/pods/9126994e-0a2a-4558-a255-bc3f01359322/volumes" Oct 30 13:11:43.646977 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:43.646961 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="983e3a6f-7d5b-49d1-ab1f-560b1f59c935" path="/var/lib/kubelet/pods/983e3a6f-7d5b-49d1-ab1f-560b1f59c935/volumes" Oct 30 13:11:43.647871 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:43.647849 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f0665fa-cce6-4a67-97c7-88982d036ac8" path="/var/lib/kubelet/pods/9f0665fa-cce6-4a67-97c7-88982d036ac8/volumes" Oct 30 13:11:44.869790 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:44.869751 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:45.408011 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:45.407959 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:45.532319 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:45.532284 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0fd5fcba-gqxfm"] Oct 30 13:11:45.532506 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:45.532340 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:45.545346 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:45.545308 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0fd5fcba-gqxfm"] Oct 30 13:11:45.646119 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:45.646088 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec623696-2a8c-4ddf-8efc-310a7d683722" path="/var/lib/kubelet/pods/ec623696-2a8c-4ddf-8efc-310a7d683722/volumes" Oct 30 13:11:45.750923 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:45.750891 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:46.184685 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:46.181788 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:46.297116 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:46.297085 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:46.649090 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:46.649058 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:46.810092 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:46.810061 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-484607-vs47h"] Oct 30 13:11:46.810945 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:46.810926 5922 cpu_manager.go:414] "containerMap: removing container" podUID="175cab09-e4aa-424b-9eaa-01c9d32d2669" containerName="revoke-keys" Oct 30 13:11:46.811002 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:46.810948 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="175cab09-e4aa-424b-9eaa-01c9d32d2669" containerName="revoke-keys" Oct 30 13:11:46.811002 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:46.810979 5922 cpu_manager.go:414] "containerMap: removing container" podUID="ec623696-2a8c-4ddf-8efc-310a7d683722" containerName="revoke-keys" Oct 30 13:11:46.811002 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:46.810988 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec623696-2a8c-4ddf-8efc-310a7d683722" containerName="revoke-keys" Oct 30 13:11:46.811002 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:46.810998 5922 cpu_manager.go:414] "containerMap: removing container" podUID="9126994e-0a2a-4558-a255-bc3f01359322" containerName="istio-proxy" Oct 30 13:11:46.811129 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:46.811007 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9126994e-0a2a-4558-a255-bc3f01359322" containerName="istio-proxy" Oct 30 13:11:46.811129 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:46.811019 5922 cpu_manager.go:414] "containerMap: removing container" podUID="9f0665fa-cce6-4a67-97c7-88982d036ac8" containerName="istio-proxy" Oct 30 13:11:46.811129 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:46.811027 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f0665fa-cce6-4a67-97c7-88982d036ac8" containerName="istio-proxy" Oct 30 13:11:46.811129 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:46.811037 5922 cpu_manager.go:414] "containerMap: removing container" podUID="68a59f39-95e5-4e6c-b589-bf0cb560a12c" containerName="payload-pre-processing" Oct 30 13:11:46.811129 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:46.811044 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68a59f39-95e5-4e6c-b589-bf0cb560a12c" containerName="payload-pre-processing" Oct 30 13:11:46.811129 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:46.811058 5922 cpu_manager.go:414] "containerMap: removing container" podUID="8ead3af3-ce7e-40db-988e-d2bcd10fa01f" containerName="payload-processing" Oct 30 13:11:46.811129 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:46.811067 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ead3af3-ce7e-40db-988e-d2bcd10fa01f" containerName="payload-processing" Oct 30 13:11:46.811129 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:46.811101 5922 cpu_manager.go:414] "containerMap: removing container" podUID="983e3a6f-7d5b-49d1-ab1f-560b1f59c935" containerName="istio-proxy" Oct 30 13:11:46.811129 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:46.811108 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="983e3a6f-7d5b-49d1-ab1f-560b1f59c935" containerName="istio-proxy" Oct 30 13:11:46.811129 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:46.811118 5922 cpu_manager.go:414] "containerMap: removing container" podUID="4cd060f1-18db-4577-b842-09a746874192" containerName="payload-pre-processing" Oct 30 13:11:46.811129 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:46.811125 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cd060f1-18db-4577-b842-09a746874192" containerName="payload-pre-processing" Oct 30 13:11:46.811575 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:46.811143 5922 cpu_manager.go:414] "containerMap: removing container" podUID="8b8f7b30-787c-451b-a41f-a5973359c9d6" containerName="revoke-keys" Oct 30 13:11:46.811575 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:46.811150 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b8f7b30-787c-451b-a41f-a5973359c9d6" containerName="revoke-keys" Oct 30 13:11:46.811575 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:46.811299 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="9126994e-0a2a-4558-a255-bc3f01359322" containerName="istio-proxy" Oct 30 13:11:46.811575 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:46.811318 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f0665fa-cce6-4a67-97c7-88982d036ac8" containerName="istio-proxy" Oct 30 13:11:46.811575 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:46.811327 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="68a59f39-95e5-4e6c-b589-bf0cb560a12c" containerName="payload-pre-processing" Oct 30 13:11:46.811575 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:46.811338 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ead3af3-ce7e-40db-988e-d2bcd10fa01f" containerName="payload-processing" Oct 30 13:11:46.811575 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:46.811354 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="983e3a6f-7d5b-49d1-ab1f-560b1f59c935" containerName="istio-proxy" Oct 30 13:11:46.811575 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:46.811364 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cd060f1-18db-4577-b842-09a746874192" containerName="payload-pre-processing" Oct 30 13:11:46.811575 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:46.811378 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b8f7b30-787c-451b-a41f-a5973359c9d6" containerName="revoke-keys" Oct 30 13:11:46.811575 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:46.811391 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="175cab09-e4aa-424b-9eaa-01c9d32d2669" containerName="revoke-keys" Oct 30 13:11:46.811575 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:46.811404 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec623696-2a8c-4ddf-8efc-310a7d683722" containerName="revoke-keys" Oct 30 13:11:46.817022 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:46.817003 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-484607-vs47h" Oct 30 13:11:46.834699 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:46.834676 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-484607-vs47h"] Oct 30 13:11:46.841172 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:46.841153 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:11:46.841342 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:46.841195 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:11:46.841342 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:46.841243 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:11:46.844689 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:46.844671 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:11:46.996233 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:46.996208 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd0296bd-3eee-4fa4-896b-c6e1585a77c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-484607-vs47h\" (UID: \"bd0296bd-3eee-4fa4-896b-c6e1585a77c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-484607-vs47h" Oct 30 13:11:46.996393 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:46.996248 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85lg5\" (UniqueName: \"kubernetes.io/projected/bd0296bd-3eee-4fa4-896b-c6e1585a77c3-kube-api-access-85lg5\") pod \"maas-api-revoke-keys-e2e-worker-w6-484607-vs47h\" (UID: \"bd0296bd-3eee-4fa4-896b-c6e1585a77c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-484607-vs47h" Oct 30 13:11:46.996443 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:46.996392 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd0296bd-3eee-4fa4-896b-c6e1585a77c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-484607-vs47h\" (UID: \"bd0296bd-3eee-4fa4-896b-c6e1585a77c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-484607-vs47h" Oct 30 13:11:47.097486 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:47.097458 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd0296bd-3eee-4fa4-896b-c6e1585a77c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-484607-vs47h\" (UID: \"bd0296bd-3eee-4fa4-896b-c6e1585a77c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-484607-vs47h" Oct 30 13:11:47.097684 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:47.097499 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85lg5\" (UniqueName: \"kubernetes.io/projected/bd0296bd-3eee-4fa4-896b-c6e1585a77c3-kube-api-access-85lg5\") pod \"maas-api-revoke-keys-e2e-worker-w6-484607-vs47h\" (UID: \"bd0296bd-3eee-4fa4-896b-c6e1585a77c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-484607-vs47h" Oct 30 13:11:47.097684 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:47.097532 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd0296bd-3eee-4fa4-896b-c6e1585a77c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-484607-vs47h\" (UID: \"bd0296bd-3eee-4fa4-896b-c6e1585a77c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-484607-vs47h" Oct 30 13:11:47.098184 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:47.098158 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd0296bd-3eee-4fa4-896b-c6e1585a77c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-484607-vs47h\" (UID: \"bd0296bd-3eee-4fa4-896b-c6e1585a77c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-484607-vs47h" Oct 30 13:11:47.099708 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:47.099687 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd0296bd-3eee-4fa4-896b-c6e1585a77c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-484607-vs47h\" (UID: \"bd0296bd-3eee-4fa4-896b-c6e1585a77c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-484607-vs47h" Oct 30 13:11:47.154998 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:47.154973 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85lg5\" (UniqueName: \"kubernetes.io/projected/bd0296bd-3eee-4fa4-896b-c6e1585a77c3-kube-api-access-85lg5\") pod \"maas-api-revoke-keys-e2e-worker-w6-484607-vs47h\" (UID: \"bd0296bd-3eee-4fa4-896b-c6e1585a77c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-484607-vs47h" Oct 30 13:11:47.432470 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:47.432389 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-484607-vs47h" Oct 30 13:11:47.748737 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:47.748597 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-484607-vs47h"] Oct 30 13:11:47.748837 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:11:47.748650 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd0296bd_3eee_4fa4_896b_c6e1585a77c3.slice/crio-51ad3613238d2dbdc93a0196e8b28fa7360216b3644f6a107eadfebc46fc8ac4 WatchSource:0}: Error finding container 51ad3613238d2dbdc93a0196e8b28fa7360216b3644f6a107eadfebc46fc8ac4: Status 404 returned error can't find the container with id 51ad3613238d2dbdc93a0196e8b28fa7360216b3644f6a107eadfebc46fc8ac4 Oct 30 13:11:47.923336 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:47.923303 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-484607-vs47h" event={"ID":"bd0296bd-3eee-4fa4-896b-c6e1585a77c3","Type":"ContainerStarted","Data":"51ad3613238d2dbdc93a0196e8b28fa7360216b3644f6a107eadfebc46fc8ac4"} Oct 30 13:11:48.008154 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:48.008079 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:48.152332 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:48.152291 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:48.944150 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:48.944114 5922 generic.go:356] "Generic (PLEG): container finished" podID="bd0296bd-3eee-4fa4-896b-c6e1585a77c3" containerID="fb539cc171ca84c72ad57fd9b50f01fb7b6e3ec2710a9dcb7ccff06728420efe" exitCode=0 Oct 30 13:11:48.944581 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:48.944192 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-484607-vs47h" event={"ID":"bd0296bd-3eee-4fa4-896b-c6e1585a77c3","Type":"ContainerDied","Data":"fb539cc171ca84c72ad57fd9b50f01fb7b6e3ec2710a9dcb7ccff06728420efe"} Oct 30 13:11:49.889093 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:49.889056 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:50.026962 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:50.026867 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:50.252333 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:50.252296 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-484607-vs47h" Oct 30 13:11:50.343740 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:50.343704 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd0296bd-3eee-4fa4-896b-c6e1585a77c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd0296bd-3eee-4fa4-896b-c6e1585a77c3-maas-api-xks-ca\") pod \"bd0296bd-3eee-4fa4-896b-c6e1585a77c3\" (UID: \"bd0296bd-3eee-4fa4-896b-c6e1585a77c3\") " Oct 30 13:11:50.344080 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:50.344056 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd0296bd-3eee-4fa4-896b-c6e1585a77c3-kube-api-access-85lg5\" (UniqueName: \"kubernetes.io/projected/bd0296bd-3eee-4fa4-896b-c6e1585a77c3-kube-api-access-85lg5\") pod \"bd0296bd-3eee-4fa4-896b-c6e1585a77c3\" (UID: \"bd0296bd-3eee-4fa4-896b-c6e1585a77c3\") " Oct 30 13:11:50.344211 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:50.344196 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd0296bd-3eee-4fa4-896b-c6e1585a77c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd0296bd-3eee-4fa4-896b-c6e1585a77c3-maas-api-openshift-ca\") pod \"bd0296bd-3eee-4fa4-896b-c6e1585a77c3\" (UID: \"bd0296bd-3eee-4fa4-896b-c6e1585a77c3\") " Oct 30 13:11:50.344880 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:50.344849 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd0296bd-3eee-4fa4-896b-c6e1585a77c3-maas-api-openshift-ca" pod "bd0296bd-3eee-4fa4-896b-c6e1585a77c3" (UID: "bd0296bd-3eee-4fa4-896b-c6e1585a77c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:50.350196 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:50.350160 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd0296bd-3eee-4fa4-896b-c6e1585a77c3-maas-api-xks-ca" pod "bd0296bd-3eee-4fa4-896b-c6e1585a77c3" (UID: "bd0296bd-3eee-4fa4-896b-c6e1585a77c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:11:50.352281 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:50.352248 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd0296bd-3eee-4fa4-896b-c6e1585a77c3-kube-api-access-85lg5" pod "bd0296bd-3eee-4fa4-896b-c6e1585a77c3" (UID: "bd0296bd-3eee-4fa4-896b-c6e1585a77c3"). InnerVolumeSpecName "kube-api-access-85lg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:50.370894 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:50.370865 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:50.447512 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:50.446663 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd0296bd-3eee-4fa4-896b-c6e1585a77c3-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:50.447512 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:50.446704 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85lg5\" (UniqueName: \"kubernetes.io/projected/bd0296bd-3eee-4fa4-896b-c6e1585a77c3-kube-api-access-85lg5\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:50.447512 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:50.446723 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd0296bd-3eee-4fa4-896b-c6e1585a77c3-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:50.498974 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:50.498887 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:50.821176 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:50.821018 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:50.944719 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:50.944672 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m"] Oct 30 13:11:50.996641 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:50.995722 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" podUID="49ffe88e-df16-4759-9a79-68dc934427fb" containerName="istio-proxy" containerID="cri-o://3c0071616a73c6daf5ec02ef82d3eab5952db36646dac9d434f90f1e42480b75" gracePeriod=30 Oct 30 13:11:51.117692 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:51.117270 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-484607-vs47h" event={"ID":"bd0296bd-3eee-4fa4-896b-c6e1585a77c3","Type":"ContainerDied","Data":"51ad3613238d2dbdc93a0196e8b28fa7360216b3644f6a107eadfebc46fc8ac4"} Oct 30 13:11:51.117692 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:51.117331 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51ad3613238d2dbdc93a0196e8b28fa7360216b3644f6a107eadfebc46fc8ac4" Oct 30 13:11:51.117692 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:51.117492 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-484607-vs47h" Oct 30 13:11:51.174725 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:51.174692 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b"] Oct 30 13:11:51.226819 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:51.220900 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" podUID="cecdab1d-a927-4d7a-b270-ba7908c88d52" containerName="istio-proxy" containerID="cri-o://3e878132d030716b2a26484e794a3815e7d2f9d535210bc5c095d4472e0dda0d" gracePeriod=30 Oct 30 13:11:51.259497 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:51.259466 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4"] Oct 30 13:11:51.259952 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:51.259918 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" podUID="4e546672-c25a-4243-a2aa-b2f23dbcae95" containerName="istio-proxy" containerID="cri-o://8f8eada104aa41b86740bb6cd2e794ed937407fbb2817ee38261b9e543b43004" gracePeriod=30 Oct 30 13:11:51.439951 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:11:51.439784 5922 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd0296bd_3eee_4fa4_896b_c6e1585a77c3.slice\": RecentStats: unable to find data in memory cache]" Oct 30 13:11:51.721671 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:51.721638 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7"] Oct 30 13:11:51.753481 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:51.753454 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5"] Oct 30 13:11:51.882127 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:51.882093 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:52.379025 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:52.378987 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:53.063221 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:53.063190 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63cdf4c0-px7v6"] Oct 30 13:11:53.074648 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:53.074605 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63cdf4c0-px7v6"] Oct 30 13:11:53.553058 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:11:53.553024 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:11:53.645580 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:53.645549 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ed1ede4-062b-4b91-b25c-8f583e120427" path="/var/lib/kubelet/pods/6ed1ede4-062b-4b91-b25c-8f583e120427/volumes" Oct 30 13:11:54.478197 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:54.478162 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:56.279372 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.279335 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:56.459562 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.459505 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:56.781758 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.781662 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:56.797370 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.797311 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" Oct 30 13:11:56.813735 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.813709 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49ffe88e-df16-4759-9a79-68dc934427fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49ffe88e-df16-4759-9a79-68dc934427fb-workload-certs\") pod \"49ffe88e-df16-4759-9a79-68dc934427fb\" (UID: \"49ffe88e-df16-4759-9a79-68dc934427fb\") " Oct 30 13:11:56.813843 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.813777 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49ffe88e-df16-4759-9a79-68dc934427fb-kube-api-access-g48cf\" (UniqueName: \"kubernetes.io/projected/49ffe88e-df16-4759-9a79-68dc934427fb-kube-api-access-g48cf\") pod \"49ffe88e-df16-4759-9a79-68dc934427fb\" (UID: \"49ffe88e-df16-4759-9a79-68dc934427fb\") " Oct 30 13:11:56.813843 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.813811 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49ffe88e-df16-4759-9a79-68dc934427fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ffe88e-df16-4759-9a79-68dc934427fb-workload-socket\") pod \"49ffe88e-df16-4759-9a79-68dc934427fb\" (UID: \"49ffe88e-df16-4759-9a79-68dc934427fb\") " Oct 30 13:11:56.813843 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.813838 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49ffe88e-df16-4759-9a79-68dc934427fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49ffe88e-df16-4759-9a79-68dc934427fb-istio-data\") pod \"49ffe88e-df16-4759-9a79-68dc934427fb\" (UID: \"49ffe88e-df16-4759-9a79-68dc934427fb\") " Oct 30 13:11:56.814016 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.813884 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49ffe88e-df16-4759-9a79-68dc934427fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ffe88e-df16-4759-9a79-68dc934427fb-credential-socket\") pod \"49ffe88e-df16-4759-9a79-68dc934427fb\" (UID: \"49ffe88e-df16-4759-9a79-68dc934427fb\") " Oct 30 13:11:56.814016 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.813946 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49ffe88e-df16-4759-9a79-68dc934427fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49ffe88e-df16-4759-9a79-68dc934427fb-istiod-ca-cert\") pod \"49ffe88e-df16-4759-9a79-68dc934427fb\" (UID: \"49ffe88e-df16-4759-9a79-68dc934427fb\") " Oct 30 13:11:56.814016 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.813992 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49ffe88e-df16-4759-9a79-68dc934427fb-istio-token\" (UniqueName: \"kubernetes.io/projected/49ffe88e-df16-4759-9a79-68dc934427fb-istio-token\") pod \"49ffe88e-df16-4759-9a79-68dc934427fb\" (UID: \"49ffe88e-df16-4759-9a79-68dc934427fb\") " Oct 30 13:11:56.814168 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.814108 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49ffe88e-df16-4759-9a79-68dc934427fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49ffe88e-df16-4759-9a79-68dc934427fb-istio-envoy\") pod \"49ffe88e-df16-4759-9a79-68dc934427fb\" (UID: \"49ffe88e-df16-4759-9a79-68dc934427fb\") " Oct 30 13:11:56.814294 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.814279 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49ffe88e-df16-4759-9a79-68dc934427fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49ffe88e-df16-4759-9a79-68dc934427fb-istio-podinfo\") pod \"49ffe88e-df16-4759-9a79-68dc934427fb\" (UID: \"49ffe88e-df16-4759-9a79-68dc934427fb\") " Oct 30 13:11:56.814346 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.814299 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ffe88e-df16-4759-9a79-68dc934427fb-workload-socket" pod "49ffe88e-df16-4759-9a79-68dc934427fb" (UID: "49ffe88e-df16-4759-9a79-68dc934427fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:56.814563 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.814540 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ffe88e-df16-4759-9a79-68dc934427fb-workload-certs" pod "49ffe88e-df16-4759-9a79-68dc934427fb" (UID: "49ffe88e-df16-4759-9a79-68dc934427fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:56.814916 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.814893 5922 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ffe88e-df16-4759-9a79-68dc934427fb-workload-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:56.815018 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.814920 5922 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49ffe88e-df16-4759-9a79-68dc934427fb-workload-certs\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:56.815862 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.815819 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49ffe88e-df16-4759-9a79-68dc934427fb-istiod-ca-cert" pod "49ffe88e-df16-4759-9a79-68dc934427fb" (UID: "49ffe88e-df16-4759-9a79-68dc934427fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:56.816087 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.816065 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ffe88e-df16-4759-9a79-68dc934427fb-istio-data" pod "49ffe88e-df16-4759-9a79-68dc934427fb" (UID: "49ffe88e-df16-4759-9a79-68dc934427fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:56.816265 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.816245 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ffe88e-df16-4759-9a79-68dc934427fb-credential-socket" pod "49ffe88e-df16-4759-9a79-68dc934427fb" (UID: "49ffe88e-df16-4759-9a79-68dc934427fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:56.818976 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.818952 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49ffe88e-df16-4759-9a79-68dc934427fb-istio-podinfo" pod "49ffe88e-df16-4759-9a79-68dc934427fb" (UID: "49ffe88e-df16-4759-9a79-68dc934427fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:11:56.820504 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.820364 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49ffe88e-df16-4759-9a79-68dc934427fb-istio-token" pod "49ffe88e-df16-4759-9a79-68dc934427fb" (UID: "49ffe88e-df16-4759-9a79-68dc934427fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:56.821231 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.821206 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49ffe88e-df16-4759-9a79-68dc934427fb-kube-api-access-g48cf" pod "49ffe88e-df16-4759-9a79-68dc934427fb" (UID: "49ffe88e-df16-4759-9a79-68dc934427fb"). InnerVolumeSpecName "kube-api-access-g48cf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:56.822910 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.822881 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ffe88e-df16-4759-9a79-68dc934427fb-istio-envoy" pod "49ffe88e-df16-4759-9a79-68dc934427fb" (UID: "49ffe88e-df16-4759-9a79-68dc934427fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:56.827946 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.827928 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" Oct 30 13:11:56.859569 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.859549 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" Oct 30 13:11:56.915800 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.915771 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cecdab1d-a927-4d7a-b270-ba7908c88d52-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecdab1d-a927-4d7a-b270-ba7908c88d52-credential-socket\") pod \"cecdab1d-a927-4d7a-b270-ba7908c88d52\" (UID: \"cecdab1d-a927-4d7a-b270-ba7908c88d52\") " Oct 30 13:11:56.915946 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.915815 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e546672-c25a-4243-a2aa-b2f23dbcae95-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e546672-c25a-4243-a2aa-b2f23dbcae95-istio-podinfo\") pod \"4e546672-c25a-4243-a2aa-b2f23dbcae95\" (UID: \"4e546672-c25a-4243-a2aa-b2f23dbcae95\") " Oct 30 13:11:56.915946 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.915841 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cecdab1d-a927-4d7a-b270-ba7908c88d52-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cecdab1d-a927-4d7a-b270-ba7908c88d52-istio-podinfo\") pod \"cecdab1d-a927-4d7a-b270-ba7908c88d52\" (UID: \"cecdab1d-a927-4d7a-b270-ba7908c88d52\") " Oct 30 13:11:56.915946 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.915881 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e546672-c25a-4243-a2aa-b2f23dbcae95-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e546672-c25a-4243-a2aa-b2f23dbcae95-istiod-ca-cert\") pod \"4e546672-c25a-4243-a2aa-b2f23dbcae95\" (UID: \"4e546672-c25a-4243-a2aa-b2f23dbcae95\") " Oct 30 13:11:56.915946 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.915910 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cecdab1d-a927-4d7a-b270-ba7908c88d52-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cecdab1d-a927-4d7a-b270-ba7908c88d52-istiod-ca-cert\") pod \"cecdab1d-a927-4d7a-b270-ba7908c88d52\" (UID: \"cecdab1d-a927-4d7a-b270-ba7908c88d52\") " Oct 30 13:11:56.915946 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.915927 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e546672-c25a-4243-a2aa-b2f23dbcae95-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e546672-c25a-4243-a2aa-b2f23dbcae95-workload-certs\") pod \"4e546672-c25a-4243-a2aa-b2f23dbcae95\" (UID: \"4e546672-c25a-4243-a2aa-b2f23dbcae95\") " Oct 30 13:11:56.916240 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.915957 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cecdab1d-a927-4d7a-b270-ba7908c88d52-kube-api-access-vbb5n\" (UniqueName: \"kubernetes.io/projected/cecdab1d-a927-4d7a-b270-ba7908c88d52-kube-api-access-vbb5n\") pod \"cecdab1d-a927-4d7a-b270-ba7908c88d52\" (UID: \"cecdab1d-a927-4d7a-b270-ba7908c88d52\") " Oct 30 13:11:56.916240 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.916006 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e546672-c25a-4243-a2aa-b2f23dbcae95-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e546672-c25a-4243-a2aa-b2f23dbcae95-workload-socket\") pod \"4e546672-c25a-4243-a2aa-b2f23dbcae95\" (UID: \"4e546672-c25a-4243-a2aa-b2f23dbcae95\") " Oct 30 13:11:56.916240 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.916055 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e546672-c25a-4243-a2aa-b2f23dbcae95-istio-token\" (UniqueName: \"kubernetes.io/projected/4e546672-c25a-4243-a2aa-b2f23dbcae95-istio-token\") pod \"4e546672-c25a-4243-a2aa-b2f23dbcae95\" (UID: \"4e546672-c25a-4243-a2aa-b2f23dbcae95\") " Oct 30 13:11:56.916240 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.916071 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cecdab1d-a927-4d7a-b270-ba7908c88d52-credential-socket" pod "cecdab1d-a927-4d7a-b270-ba7908c88d52" (UID: "cecdab1d-a927-4d7a-b270-ba7908c88d52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:56.916240 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.916087 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cecdab1d-a927-4d7a-b270-ba7908c88d52-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecdab1d-a927-4d7a-b270-ba7908c88d52-workload-socket\") pod \"cecdab1d-a927-4d7a-b270-ba7908c88d52\" (UID: \"cecdab1d-a927-4d7a-b270-ba7908c88d52\") " Oct 30 13:11:56.916240 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.916137 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cecdab1d-a927-4d7a-b270-ba7908c88d52-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cecdab1d-a927-4d7a-b270-ba7908c88d52-workload-certs\") pod \"cecdab1d-a927-4d7a-b270-ba7908c88d52\" (UID: \"cecdab1d-a927-4d7a-b270-ba7908c88d52\") " Oct 30 13:11:56.916539 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.916277 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e546672-c25a-4243-a2aa-b2f23dbcae95-workload-socket" pod "4e546672-c25a-4243-a2aa-b2f23dbcae95" (UID: "4e546672-c25a-4243-a2aa-b2f23dbcae95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:56.916539 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.916354 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e546672-c25a-4243-a2aa-b2f23dbcae95-istiod-ca-cert" pod "4e546672-c25a-4243-a2aa-b2f23dbcae95" (UID: "4e546672-c25a-4243-a2aa-b2f23dbcae95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:56.916539 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.916388 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cecdab1d-a927-4d7a-b270-ba7908c88d52-workload-certs" pod "cecdab1d-a927-4d7a-b270-ba7908c88d52" (UID: "cecdab1d-a927-4d7a-b270-ba7908c88d52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:56.916539 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.916417 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cecdab1d-a927-4d7a-b270-ba7908c88d52-istiod-ca-cert" pod "cecdab1d-a927-4d7a-b270-ba7908c88d52" (UID: "cecdab1d-a927-4d7a-b270-ba7908c88d52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:56.916539 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.916434 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e546672-c25a-4243-a2aa-b2f23dbcae95-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e546672-c25a-4243-a2aa-b2f23dbcae95-istio-data\") pod \"4e546672-c25a-4243-a2aa-b2f23dbcae95\" (UID: \"4e546672-c25a-4243-a2aa-b2f23dbcae95\") " Oct 30 13:11:56.916539 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.916494 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e546672-c25a-4243-a2aa-b2f23dbcae95-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e546672-c25a-4243-a2aa-b2f23dbcae95-istio-envoy\") pod \"4e546672-c25a-4243-a2aa-b2f23dbcae95\" (UID: \"4e546672-c25a-4243-a2aa-b2f23dbcae95\") " Oct 30 13:11:56.916893 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.916598 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e546672-c25a-4243-a2aa-b2f23dbcae95-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e546672-c25a-4243-a2aa-b2f23dbcae95-credential-socket\") pod \"4e546672-c25a-4243-a2aa-b2f23dbcae95\" (UID: \"4e546672-c25a-4243-a2aa-b2f23dbcae95\") " Oct 30 13:11:56.916893 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.916700 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cecdab1d-a927-4d7a-b270-ba7908c88d52-istio-token\" (UniqueName: \"kubernetes.io/projected/cecdab1d-a927-4d7a-b270-ba7908c88d52-istio-token\") pod \"cecdab1d-a927-4d7a-b270-ba7908c88d52\" (UID: \"cecdab1d-a927-4d7a-b270-ba7908c88d52\") " Oct 30 13:11:56.916893 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.916750 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e546672-c25a-4243-a2aa-b2f23dbcae95-kube-api-access-l7r9q\" (UniqueName: \"kubernetes.io/projected/4e546672-c25a-4243-a2aa-b2f23dbcae95-kube-api-access-l7r9q\") pod \"4e546672-c25a-4243-a2aa-b2f23dbcae95\" (UID: \"4e546672-c25a-4243-a2aa-b2f23dbcae95\") " Oct 30 13:11:56.916893 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.916630 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cecdab1d-a927-4d7a-b270-ba7908c88d52-workload-socket" pod "cecdab1d-a927-4d7a-b270-ba7908c88d52" (UID: "cecdab1d-a927-4d7a-b270-ba7908c88d52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:56.916893 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.916760 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e546672-c25a-4243-a2aa-b2f23dbcae95-istio-data" pod "4e546672-c25a-4243-a2aa-b2f23dbcae95" (UID: "4e546672-c25a-4243-a2aa-b2f23dbcae95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:56.916893 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.916799 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cecdab1d-a927-4d7a-b270-ba7908c88d52-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cecdab1d-a927-4d7a-b270-ba7908c88d52-istio-data\") pod \"cecdab1d-a927-4d7a-b270-ba7908c88d52\" (UID: \"cecdab1d-a927-4d7a-b270-ba7908c88d52\") " Oct 30 13:11:56.916893 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.916853 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cecdab1d-a927-4d7a-b270-ba7908c88d52-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cecdab1d-a927-4d7a-b270-ba7908c88d52-istio-envoy\") pod \"cecdab1d-a927-4d7a-b270-ba7908c88d52\" (UID: \"cecdab1d-a927-4d7a-b270-ba7908c88d52\") " Oct 30 13:11:56.916893 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.916854 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e546672-c25a-4243-a2aa-b2f23dbcae95-workload-certs" pod "4e546672-c25a-4243-a2aa-b2f23dbcae95" (UID: "4e546672-c25a-4243-a2aa-b2f23dbcae95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:56.917300 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.917006 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e546672-c25a-4243-a2aa-b2f23dbcae95-credential-socket" pod "4e546672-c25a-4243-a2aa-b2f23dbcae95" (UID: "4e546672-c25a-4243-a2aa-b2f23dbcae95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:56.917663 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.917643 5922 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecdab1d-a927-4d7a-b270-ba7908c88d52-credential-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:56.917768 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.917668 5922 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e546672-c25a-4243-a2aa-b2f23dbcae95-istiod-ca-cert\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:56.917768 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.917683 5922 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cecdab1d-a927-4d7a-b270-ba7908c88d52-istiod-ca-cert\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:56.917768 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.917696 5922 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e546672-c25a-4243-a2aa-b2f23dbcae95-workload-certs\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:56.917768 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.917710 5922 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e546672-c25a-4243-a2aa-b2f23dbcae95-workload-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:56.917768 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.917722 5922 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49ffe88e-df16-4759-9a79-68dc934427fb-istio-podinfo\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:56.917768 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.917735 5922 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecdab1d-a927-4d7a-b270-ba7908c88d52-workload-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:56.917768 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.917748 5922 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cecdab1d-a927-4d7a-b270-ba7908c88d52-workload-certs\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:56.917768 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.917762 5922 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e546672-c25a-4243-a2aa-b2f23dbcae95-istio-data\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:56.918182 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.917777 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g48cf\" (UniqueName: \"kubernetes.io/projected/49ffe88e-df16-4759-9a79-68dc934427fb-kube-api-access-g48cf\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:56.918182 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.917790 5922 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49ffe88e-df16-4759-9a79-68dc934427fb-istio-data\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:56.918182 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.917803 5922 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ffe88e-df16-4759-9a79-68dc934427fb-credential-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:56.918182 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.917816 5922 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e546672-c25a-4243-a2aa-b2f23dbcae95-credential-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:56.918182 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.917828 5922 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49ffe88e-df16-4759-9a79-68dc934427fb-istiod-ca-cert\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:56.918182 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.917841 5922 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49ffe88e-df16-4759-9a79-68dc934427fb-istio-token\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:56.918182 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.917853 5922 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49ffe88e-df16-4759-9a79-68dc934427fb-istio-envoy\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:56.918579 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.918560 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e546672-c25a-4243-a2aa-b2f23dbcae95-istio-podinfo" pod "4e546672-c25a-4243-a2aa-b2f23dbcae95" (UID: "4e546672-c25a-4243-a2aa-b2f23dbcae95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:11:56.918885 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.918866 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cecdab1d-a927-4d7a-b270-ba7908c88d52-istio-data" pod "cecdab1d-a927-4d7a-b270-ba7908c88d52" (UID: "cecdab1d-a927-4d7a-b270-ba7908c88d52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:56.919260 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.919235 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cecdab1d-a927-4d7a-b270-ba7908c88d52-kube-api-access-vbb5n" pod "cecdab1d-a927-4d7a-b270-ba7908c88d52" (UID: "cecdab1d-a927-4d7a-b270-ba7908c88d52"). InnerVolumeSpecName "kube-api-access-vbb5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:56.919412 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.919392 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e546672-c25a-4243-a2aa-b2f23dbcae95-istio-envoy" pod "4e546672-c25a-4243-a2aa-b2f23dbcae95" (UID: "4e546672-c25a-4243-a2aa-b2f23dbcae95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:56.919470 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.919417 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cecdab1d-a927-4d7a-b270-ba7908c88d52-istio-token" pod "cecdab1d-a927-4d7a-b270-ba7908c88d52" (UID: "cecdab1d-a927-4d7a-b270-ba7908c88d52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:56.920121 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.920095 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cecdab1d-a927-4d7a-b270-ba7908c88d52-istio-envoy" pod "cecdab1d-a927-4d7a-b270-ba7908c88d52" (UID: "cecdab1d-a927-4d7a-b270-ba7908c88d52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:56.920862 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.920835 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cecdab1d-a927-4d7a-b270-ba7908c88d52-istio-podinfo" pod "cecdab1d-a927-4d7a-b270-ba7908c88d52" (UID: "cecdab1d-a927-4d7a-b270-ba7908c88d52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:11:56.921203 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.921182 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e546672-c25a-4243-a2aa-b2f23dbcae95-istio-token" pod "4e546672-c25a-4243-a2aa-b2f23dbcae95" (UID: "4e546672-c25a-4243-a2aa-b2f23dbcae95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:56.921669 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.921648 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e546672-c25a-4243-a2aa-b2f23dbcae95-kube-api-access-l7r9q" pod "4e546672-c25a-4243-a2aa-b2f23dbcae95" (UID: "4e546672-c25a-4243-a2aa-b2f23dbcae95"). InnerVolumeSpecName "kube-api-access-l7r9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:56.984886 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:56.984830 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:57.019165 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.019137 5922 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cecdab1d-a927-4d7a-b270-ba7908c88d52-istio-envoy\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:57.019165 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.019162 5922 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e546672-c25a-4243-a2aa-b2f23dbcae95-istio-podinfo\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:57.019165 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.019171 5922 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cecdab1d-a927-4d7a-b270-ba7908c88d52-istio-podinfo\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:57.019356 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.019179 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbb5n\" (UniqueName: \"kubernetes.io/projected/cecdab1d-a927-4d7a-b270-ba7908c88d52-kube-api-access-vbb5n\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:57.019356 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.019191 5922 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e546672-c25a-4243-a2aa-b2f23dbcae95-istio-token\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:57.019356 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.019199 5922 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e546672-c25a-4243-a2aa-b2f23dbcae95-istio-envoy\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:57.019356 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.019207 5922 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cecdab1d-a927-4d7a-b270-ba7908c88d52-istio-token\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:57.019356 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.019215 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7r9q\" (UniqueName: \"kubernetes.io/projected/4e546672-c25a-4243-a2aa-b2f23dbcae95-kube-api-access-l7r9q\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:57.019356 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.019223 5922 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cecdab1d-a927-4d7a-b270-ba7908c88d52-istio-data\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:11:57.057749 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.057671 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3adc8c0-5hkn5"] Oct 30 13:11:57.065045 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.065020 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3adc8c0-5hkn5"] Oct 30 13:11:57.171782 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.171747 5922 generic.go:356] "Generic (PLEG): container finished" podID="cecdab1d-a927-4d7a-b270-ba7908c88d52" containerID="3e878132d030716b2a26484e794a3815e7d2f9d535210bc5c095d4472e0dda0d" exitCode=0 Oct 30 13:11:57.171974 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.171780 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" event={"ID":"cecdab1d-a927-4d7a-b270-ba7908c88d52","Type":"ContainerDied","Data":"3e878132d030716b2a26484e794a3815e7d2f9d535210bc5c095d4472e0dda0d"} Oct 30 13:11:57.171974 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.171808 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" Oct 30 13:11:57.171974 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.171849 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b" event={"ID":"cecdab1d-a927-4d7a-b270-ba7908c88d52","Type":"ContainerDied","Data":"4ffef3b1a64685463f41f47001fec79cc78d2bd36a2f6d0c2d2cdf2235938383"} Oct 30 13:11:57.171974 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.171879 5922 scope.go:122] "RemoveContainer" containerID="3e878132d030716b2a26484e794a3815e7d2f9d535210bc5c095d4472e0dda0d" Oct 30 13:11:57.178382 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.178061 5922 generic.go:356] "Generic (PLEG): container finished" podID="4e546672-c25a-4243-a2aa-b2f23dbcae95" containerID="8f8eada104aa41b86740bb6cd2e794ed937407fbb2817ee38261b9e543b43004" exitCode=0 Oct 30 13:11:57.178382 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.178138 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" Oct 30 13:11:57.178382 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.178148 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" event={"ID":"4e546672-c25a-4243-a2aa-b2f23dbcae95","Type":"ContainerDied","Data":"8f8eada104aa41b86740bb6cd2e794ed937407fbb2817ee38261b9e543b43004"} Oct 30 13:11:57.178382 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.178302 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4" event={"ID":"4e546672-c25a-4243-a2aa-b2f23dbcae95","Type":"ContainerDied","Data":"e1b8217f5115a8740b970ea6adf37dcaadc010041cb2e652bb863f3256364da1"} Oct 30 13:11:57.180254 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.180227 5922 generic.go:356] "Generic (PLEG): container finished" podID="49ffe88e-df16-4759-9a79-68dc934427fb" containerID="3c0071616a73c6daf5ec02ef82d3eab5952db36646dac9d434f90f1e42480b75" exitCode=0 Oct 30 13:11:57.180385 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.180292 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" Oct 30 13:11:57.180597 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.180292 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" event={"ID":"49ffe88e-df16-4759-9a79-68dc934427fb","Type":"ContainerDied","Data":"3c0071616a73c6daf5ec02ef82d3eab5952db36646dac9d434f90f1e42480b75"} Oct 30 13:11:57.180597 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.180561 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m" event={"ID":"49ffe88e-df16-4759-9a79-68dc934427fb","Type":"ContainerDied","Data":"12e203e156059cbcafb5fcfe0ffe267b673449809530585840b33fb410e508ea"} Oct 30 13:11:57.191590 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.191568 5922 scope.go:122] "RemoveContainer" containerID="39fe7c325a40de5d43ff7303c8776ba7b385000c66965a74ea286879561694a6" Oct 30 13:11:57.205211 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.205192 5922 scope.go:122] "RemoveContainer" containerID="3e878132d030716b2a26484e794a3815e7d2f9d535210bc5c095d4472e0dda0d" Oct 30 13:11:57.205496 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:11:57.205473 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e878132d030716b2a26484e794a3815e7d2f9d535210bc5c095d4472e0dda0d\": container with ID starting with 3e878132d030716b2a26484e794a3815e7d2f9d535210bc5c095d4472e0dda0d not found: ID does not exist" containerID="3e878132d030716b2a26484e794a3815e7d2f9d535210bc5c095d4472e0dda0d" Oct 30 13:11:57.205590 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.205509 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e878132d030716b2a26484e794a3815e7d2f9d535210bc5c095d4472e0dda0d"} err="failed to get container status \"3e878132d030716b2a26484e794a3815e7d2f9d535210bc5c095d4472e0dda0d\": rpc error: code = NotFound desc = could not find container \"3e878132d030716b2a26484e794a3815e7d2f9d535210bc5c095d4472e0dda0d\": container with ID starting with 3e878132d030716b2a26484e794a3815e7d2f9d535210bc5c095d4472e0dda0d not found: ID does not exist" Oct 30 13:11:57.205590 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.205529 5922 scope.go:122] "RemoveContainer" containerID="39fe7c325a40de5d43ff7303c8776ba7b385000c66965a74ea286879561694a6" Oct 30 13:11:57.205820 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:11:57.205799 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39fe7c325a40de5d43ff7303c8776ba7b385000c66965a74ea286879561694a6\": container with ID starting with 39fe7c325a40de5d43ff7303c8776ba7b385000c66965a74ea286879561694a6 not found: ID does not exist" containerID="39fe7c325a40de5d43ff7303c8776ba7b385000c66965a74ea286879561694a6" Oct 30 13:11:57.205920 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.205822 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39fe7c325a40de5d43ff7303c8776ba7b385000c66965a74ea286879561694a6"} err="failed to get container status \"39fe7c325a40de5d43ff7303c8776ba7b385000c66965a74ea286879561694a6\": rpc error: code = NotFound desc = could not find container \"39fe7c325a40de5d43ff7303c8776ba7b385000c66965a74ea286879561694a6\": container with ID starting with 39fe7c325a40de5d43ff7303c8776ba7b385000c66965a74ea286879561694a6 not found: ID does not exist" Oct 30 13:11:57.205920 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.205839 5922 scope.go:122] "RemoveContainer" containerID="8f8eada104aa41b86740bb6cd2e794ed937407fbb2817ee38261b9e543b43004" Oct 30 13:11:57.215866 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.215842 5922 scope.go:122] "RemoveContainer" containerID="8f8eada104aa41b86740bb6cd2e794ed937407fbb2817ee38261b9e543b43004" Oct 30 13:11:57.216145 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:11:57.216127 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f8eada104aa41b86740bb6cd2e794ed937407fbb2817ee38261b9e543b43004\": container with ID starting with 8f8eada104aa41b86740bb6cd2e794ed937407fbb2817ee38261b9e543b43004 not found: ID does not exist" containerID="8f8eada104aa41b86740bb6cd2e794ed937407fbb2817ee38261b9e543b43004" Oct 30 13:11:57.216216 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.216154 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f8eada104aa41b86740bb6cd2e794ed937407fbb2817ee38261b9e543b43004"} err="failed to get container status \"8f8eada104aa41b86740bb6cd2e794ed937407fbb2817ee38261b9e543b43004\": rpc error: code = NotFound desc = could not find container \"8f8eada104aa41b86740bb6cd2e794ed937407fbb2817ee38261b9e543b43004\": container with ID starting with 8f8eada104aa41b86740bb6cd2e794ed937407fbb2817ee38261b9e543b43004 not found: ID does not exist" Oct 30 13:11:57.216216 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.216175 5922 scope.go:122] "RemoveContainer" containerID="3c0071616a73c6daf5ec02ef82d3eab5952db36646dac9d434f90f1e42480b75" Oct 30 13:11:57.226042 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.226011 5922 scope.go:122] "RemoveContainer" containerID="3c0071616a73c6daf5ec02ef82d3eab5952db36646dac9d434f90f1e42480b75" Oct 30 13:11:57.226369 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:11:57.226336 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c0071616a73c6daf5ec02ef82d3eab5952db36646dac9d434f90f1e42480b75\": container with ID starting with 3c0071616a73c6daf5ec02ef82d3eab5952db36646dac9d434f90f1e42480b75 not found: ID does not exist" containerID="3c0071616a73c6daf5ec02ef82d3eab5952db36646dac9d434f90f1e42480b75" Oct 30 13:11:57.226459 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.226374 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c0071616a73c6daf5ec02ef82d3eab5952db36646dac9d434f90f1e42480b75"} err="failed to get container status \"3c0071616a73c6daf5ec02ef82d3eab5952db36646dac9d434f90f1e42480b75\": rpc error: code = NotFound desc = could not find container \"3c0071616a73c6daf5ec02ef82d3eab5952db36646dac9d434f90f1e42480b75\": container with ID starting with 3c0071616a73c6daf5ec02ef82d3eab5952db36646dac9d434f90f1e42480b75 not found: ID does not exist" Oct 30 13:11:57.249313 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.249283 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b"] Oct 30 13:11:57.253845 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.253822 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-jtn7b"] Oct 30 13:11:57.307729 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.307674 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m"] Oct 30 13:11:57.312069 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.311993 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-484607-5684dfc6c6-8z79j"] Oct 30 13:11:57.315742 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.315720 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-klz8m"] Oct 30 13:11:57.346789 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.346763 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-484607-6d646d7567-hd7j6"] Oct 30 13:11:57.351816 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.351792 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4"] Oct 30 13:11:57.365605 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.365580 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-mb2j4"] Oct 30 13:11:57.471272 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.471237 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:57.603786 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.603701 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:57.649230 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.649183 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646" path="/var/lib/kubelet/pods/1dfb2b3b-76f0-4bcb-b0f9-eb39ae32b646/volumes" Oct 30 13:11:57.649890 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.649862 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49ffe88e-df16-4759-9a79-68dc934427fb" path="/var/lib/kubelet/pods/49ffe88e-df16-4759-9a79-68dc934427fb/volumes" Oct 30 13:11:57.650374 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.650357 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e546672-c25a-4243-a2aa-b2f23dbcae95" path="/var/lib/kubelet/pods/4e546672-c25a-4243-a2aa-b2f23dbcae95/volumes" Oct 30 13:11:57.651423 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:57.651391 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cecdab1d-a927-4d7a-b270-ba7908c88d52" path="/var/lib/kubelet/pods/cecdab1d-a927-4d7a-b270-ba7908c88d52/volumes" Oct 30 13:11:58.333320 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.333287 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88b5cc45-jzt8s"] Oct 30 13:11:58.334088 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.334066 5922 cpu_manager.go:414] "containerMap: removing container" podUID="cecdab1d-a927-4d7a-b270-ba7908c88d52" containerName="istio-proxy" Oct 30 13:11:58.334198 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.334089 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cecdab1d-a927-4d7a-b270-ba7908c88d52" containerName="istio-proxy" Oct 30 13:11:58.334198 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.334112 5922 cpu_manager.go:414] "containerMap: removing container" podUID="cecdab1d-a927-4d7a-b270-ba7908c88d52" containerName="istio-proxy" Oct 30 13:11:58.334198 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.334119 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cecdab1d-a927-4d7a-b270-ba7908c88d52" containerName="istio-proxy" Oct 30 13:11:58.334198 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.334144 5922 cpu_manager.go:414] "containerMap: removing container" podUID="bd0296bd-3eee-4fa4-896b-c6e1585a77c3" containerName="revoke-keys" Oct 30 13:11:58.334198 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.334152 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd0296bd-3eee-4fa4-896b-c6e1585a77c3" containerName="revoke-keys" Oct 30 13:11:58.334198 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.334169 5922 cpu_manager.go:414] "containerMap: removing container" podUID="4e546672-c25a-4243-a2aa-b2f23dbcae95" containerName="istio-proxy" Oct 30 13:11:58.334198 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.334177 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e546672-c25a-4243-a2aa-b2f23dbcae95" containerName="istio-proxy" Oct 30 13:11:58.334198 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.334194 5922 cpu_manager.go:414] "containerMap: removing container" podUID="49ffe88e-df16-4759-9a79-68dc934427fb" containerName="istio-proxy" Oct 30 13:11:58.334571 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.334202 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49ffe88e-df16-4759-9a79-68dc934427fb" containerName="istio-proxy" Oct 30 13:11:58.334571 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.334357 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd0296bd-3eee-4fa4-896b-c6e1585a77c3" containerName="revoke-keys" Oct 30 13:11:58.334571 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.334374 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e546672-c25a-4243-a2aa-b2f23dbcae95" containerName="istio-proxy" Oct 30 13:11:58.334571 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.334386 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="49ffe88e-df16-4759-9a79-68dc934427fb" containerName="istio-proxy" Oct 30 13:11:58.334571 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.334403 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="cecdab1d-a927-4d7a-b270-ba7908c88d52" containerName="istio-proxy" Oct 30 13:11:58.334571 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.334413 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="cecdab1d-a927-4d7a-b270-ba7908c88d52" containerName="istio-proxy" Oct 30 13:11:58.340043 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.340021 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b5cc45-jzt8s" Oct 30 13:11:58.347825 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.347802 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5da9845-7bwfw"] Oct 30 13:11:58.359818 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.359786 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:11:58.359982 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.359963 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:11:58.363896 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.363868 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da9845-7bwfw" Oct 30 13:11:58.367292 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.367266 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88b5cc45-jzt8s"] Oct 30 13:11:58.370339 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.369139 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:11:58.370339 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.369211 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:11:58.388502 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.388481 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5da9845-7bwfw"] Oct 30 13:11:58.438682 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.438646 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5da9845-7bwfw\" (UID: \"76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da9845-7bwfw" Oct 30 13:11:58.438798 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.438762 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5da9845-7bwfw\" (UID: \"76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da9845-7bwfw" Oct 30 13:11:58.438836 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.438817 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d02e9835-5ada-40e8-ba81-13f9f8fe6541-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88b5cc45-jzt8s\" (UID: \"d02e9835-5ada-40e8-ba81-13f9f8fe6541\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b5cc45-jzt8s" Oct 30 13:11:58.438889 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.438834 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d02e9835-5ada-40e8-ba81-13f9f8fe6541-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88b5cc45-jzt8s\" (UID: \"d02e9835-5ada-40e8-ba81-13f9f8fe6541\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b5cc45-jzt8s" Oct 30 13:11:58.438889 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.438863 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74m8s\" (UniqueName: \"kubernetes.io/projected/d02e9835-5ada-40e8-ba81-13f9f8fe6541-kube-api-access-74m8s\") pod \"maas-api-revoke-sub-88b5cc45-jzt8s\" (UID: \"d02e9835-5ada-40e8-ba81-13f9f8fe6541\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b5cc45-jzt8s" Oct 30 13:11:58.438958 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.438944 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc9s2\" (UniqueName: \"kubernetes.io/projected/76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07-kube-api-access-dc9s2\") pod \"maas-api-revoke-sub-e5da9845-7bwfw\" (UID: \"76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da9845-7bwfw" Oct 30 13:11:58.540136 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.540106 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5da9845-7bwfw\" (UID: \"76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da9845-7bwfw" Oct 30 13:11:58.540267 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.540176 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5da9845-7bwfw\" (UID: \"76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da9845-7bwfw" Oct 30 13:11:58.540267 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.540238 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d02e9835-5ada-40e8-ba81-13f9f8fe6541-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88b5cc45-jzt8s\" (UID: \"d02e9835-5ada-40e8-ba81-13f9f8fe6541\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b5cc45-jzt8s" Oct 30 13:11:58.540345 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.540263 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d02e9835-5ada-40e8-ba81-13f9f8fe6541-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88b5cc45-jzt8s\" (UID: \"d02e9835-5ada-40e8-ba81-13f9f8fe6541\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b5cc45-jzt8s" Oct 30 13:11:58.540345 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.540295 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74m8s\" (UniqueName: \"kubernetes.io/projected/d02e9835-5ada-40e8-ba81-13f9f8fe6541-kube-api-access-74m8s\") pod \"maas-api-revoke-sub-88b5cc45-jzt8s\" (UID: \"d02e9835-5ada-40e8-ba81-13f9f8fe6541\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b5cc45-jzt8s" Oct 30 13:11:58.540345 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.540324 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc9s2\" (UniqueName: \"kubernetes.io/projected/76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07-kube-api-access-dc9s2\") pod \"maas-api-revoke-sub-e5da9845-7bwfw\" (UID: \"76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da9845-7bwfw" Oct 30 13:11:58.540930 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.540900 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5da9845-7bwfw\" (UID: \"76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da9845-7bwfw" Oct 30 13:11:58.541068 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.541048 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d02e9835-5ada-40e8-ba81-13f9f8fe6541-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88b5cc45-jzt8s\" (UID: \"d02e9835-5ada-40e8-ba81-13f9f8fe6541\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b5cc45-jzt8s" Oct 30 13:11:58.542560 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.542537 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5da9845-7bwfw\" (UID: \"76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da9845-7bwfw" Oct 30 13:11:58.542687 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.542669 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d02e9835-5ada-40e8-ba81-13f9f8fe6541-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88b5cc45-jzt8s\" (UID: \"d02e9835-5ada-40e8-ba81-13f9f8fe6541\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b5cc45-jzt8s" Oct 30 13:11:58.564900 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.564868 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74m8s\" (UniqueName: \"kubernetes.io/projected/d02e9835-5ada-40e8-ba81-13f9f8fe6541-kube-api-access-74m8s\") pod \"maas-api-revoke-sub-88b5cc45-jzt8s\" (UID: \"d02e9835-5ada-40e8-ba81-13f9f8fe6541\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b5cc45-jzt8s" Oct 30 13:11:58.565222 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.565203 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc9s2\" (UniqueName: \"kubernetes.io/projected/76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07-kube-api-access-dc9s2\") pod \"maas-api-revoke-sub-e5da9845-7bwfw\" (UID: \"76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da9845-7bwfw" Oct 30 13:11:58.683505 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.683433 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b5cc45-jzt8s" Oct 30 13:11:58.688403 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.688383 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da9845-7bwfw" Oct 30 13:11:58.718027 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:58.717995 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:59.000727 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:59.000700 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88b5cc45-jzt8s"] Oct 30 13:11:59.001525 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:11:59.001496 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd02e9835_5ada_40e8_ba81_13f9f8fe6541.slice/crio-390b131992bb8bf606a339611e5b263f0e064e387987364262d98596bf87abef WatchSource:0}: Error finding container 390b131992bb8bf606a339611e5b263f0e064e387987364262d98596bf87abef: Status 404 returned error can't find the container with id 390b131992bb8bf606a339611e5b263f0e064e387987364262d98596bf87abef Oct 30 13:11:59.015955 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:59.015930 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5da9845-7bwfw"] Oct 30 13:11:59.016967 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:11:59.016944 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76a1b5bb_f7a5_4f5a_a663_c76c3cfe0c07.slice/crio-10528a6a423390919ac7d8f13ed3e988272631c3c8b0eb04aa47cdca6968dc8d WatchSource:0}: Error finding container 10528a6a423390919ac7d8f13ed3e988272631c3c8b0eb04aa47cdca6968dc8d: Status 404 returned error can't find the container with id 10528a6a423390919ac7d8f13ed3e988272631c3c8b0eb04aa47cdca6968dc8d Oct 30 13:11:59.202031 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:59.202001 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da9845-7bwfw" event={"ID":"76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07","Type":"ContainerStarted","Data":"10528a6a423390919ac7d8f13ed3e988272631c3c8b0eb04aa47cdca6968dc8d"} Oct 30 13:11:59.203813 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:11:59.203788 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b5cc45-jzt8s" event={"ID":"d02e9835-5ada-40e8-ba81-13f9f8fe6541","Type":"ContainerStarted","Data":"390b131992bb8bf606a339611e5b263f0e064e387987364262d98596bf87abef"} Oct 30 13:12:00.215279 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:00.215248 5922 generic.go:356] "Generic (PLEG): container finished" podID="76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07" containerID="dcfd34e36b17b2632487f20346286b00179b67724ea3aa31110167dbae92b2a6" exitCode=0 Oct 30 13:12:00.215680 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:00.215334 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da9845-7bwfw" event={"ID":"76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07","Type":"ContainerDied","Data":"dcfd34e36b17b2632487f20346286b00179b67724ea3aa31110167dbae92b2a6"} Oct 30 13:12:00.217066 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:00.217046 5922 generic.go:356] "Generic (PLEG): container finished" podID="d02e9835-5ada-40e8-ba81-13f9f8fe6541" containerID="3be15fdc0e9aa99da628b6b7c888f0d6cf92969398e5e9204a3066e9f04b0842" exitCode=0 Oct 30 13:12:00.217187 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:00.217127 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b5cc45-jzt8s" event={"ID":"d02e9835-5ada-40e8-ba81-13f9f8fe6541","Type":"ContainerDied","Data":"3be15fdc0e9aa99da628b6b7c888f0d6cf92969398e5e9204a3066e9f04b0842"} Oct 30 13:12:00.589690 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:00.589584 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:01.389415 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:01.389382 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da9845-7bwfw" Oct 30 13:12:01.415764 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:01.415741 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b5cc45-jzt8s" Oct 30 13:12:01.477539 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:01.477457 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07-maas-api-xks-ca\") pod \"76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07\" (UID: \"76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07\") " Oct 30 13:12:01.477539 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:01.477512 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d02e9835-5ada-40e8-ba81-13f9f8fe6541-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d02e9835-5ada-40e8-ba81-13f9f8fe6541-maas-api-xks-ca\") pod \"d02e9835-5ada-40e8-ba81-13f9f8fe6541\" (UID: \"d02e9835-5ada-40e8-ba81-13f9f8fe6541\") " Oct 30 13:12:01.477803 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:01.477597 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d02e9835-5ada-40e8-ba81-13f9f8fe6541-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d02e9835-5ada-40e8-ba81-13f9f8fe6541-maas-api-openshift-ca\") pod \"d02e9835-5ada-40e8-ba81-13f9f8fe6541\" (UID: \"d02e9835-5ada-40e8-ba81-13f9f8fe6541\") " Oct 30 13:12:01.477803 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:01.477751 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07-maas-api-openshift-ca\") pod \"76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07\" (UID: \"76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07\") " Oct 30 13:12:01.477905 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:01.477842 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d02e9835-5ada-40e8-ba81-13f9f8fe6541-kube-api-access-74m8s\" (UniqueName: \"kubernetes.io/projected/d02e9835-5ada-40e8-ba81-13f9f8fe6541-kube-api-access-74m8s\") pod \"d02e9835-5ada-40e8-ba81-13f9f8fe6541\" (UID: \"d02e9835-5ada-40e8-ba81-13f9f8fe6541\") " Oct 30 13:12:01.477905 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:01.477875 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07-kube-api-access-dc9s2\" (UniqueName: \"kubernetes.io/projected/76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07-kube-api-access-dc9s2\") pod \"76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07\" (UID: \"76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07\") " Oct 30 13:12:01.480588 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:01.479033 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d02e9835-5ada-40e8-ba81-13f9f8fe6541-maas-api-openshift-ca" pod "d02e9835-5ada-40e8-ba81-13f9f8fe6541" (UID: "d02e9835-5ada-40e8-ba81-13f9f8fe6541"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:12:01.480588 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:01.479124 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07-maas-api-openshift-ca" pod "76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07" (UID: "76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:12:01.481264 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:01.481240 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d02e9835-5ada-40e8-ba81-13f9f8fe6541-kube-api-access-74m8s" pod "d02e9835-5ada-40e8-ba81-13f9f8fe6541" (UID: "d02e9835-5ada-40e8-ba81-13f9f8fe6541"). InnerVolumeSpecName "kube-api-access-74m8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:01.481554 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:01.481527 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d02e9835-5ada-40e8-ba81-13f9f8fe6541-maas-api-xks-ca" pod "d02e9835-5ada-40e8-ba81-13f9f8fe6541" (UID: "d02e9835-5ada-40e8-ba81-13f9f8fe6541"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:12:01.483259 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:01.483234 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07-kube-api-access-dc9s2" pod "76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07" (UID: "76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07"). InnerVolumeSpecName "kube-api-access-dc9s2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:01.485441 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:01.485414 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07-maas-api-xks-ca" pod "76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07" (UID: "76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:12:01.579503 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:01.579479 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:01.579503 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:01.579502 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74m8s\" (UniqueName: \"kubernetes.io/projected/d02e9835-5ada-40e8-ba81-13f9f8fe6541-kube-api-access-74m8s\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:01.579695 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:01.579513 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dc9s2\" (UniqueName: \"kubernetes.io/projected/76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07-kube-api-access-dc9s2\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:01.579695 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:01.579522 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:01.579695 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:01.579530 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d02e9835-5ada-40e8-ba81-13f9f8fe6541-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:01.579695 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:01.579538 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d02e9835-5ada-40e8-ba81-13f9f8fe6541-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:02.234885 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:02.234858 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b5cc45-jzt8s" Oct 30 13:12:02.235045 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:02.234862 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b5cc45-jzt8s" event={"ID":"d02e9835-5ada-40e8-ba81-13f9f8fe6541","Type":"ContainerDied","Data":"390b131992bb8bf606a339611e5b263f0e064e387987364262d98596bf87abef"} Oct 30 13:12:02.235045 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:02.235033 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="390b131992bb8bf606a339611e5b263f0e064e387987364262d98596bf87abef" Oct 30 13:12:02.236401 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:02.236376 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da9845-7bwfw" Oct 30 13:12:02.236401 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:02.236386 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da9845-7bwfw" event={"ID":"76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07","Type":"ContainerDied","Data":"10528a6a423390919ac7d8f13ed3e988272631c3c8b0eb04aa47cdca6968dc8d"} Oct 30 13:12:02.236568 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:02.236424 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10528a6a423390919ac7d8f13ed3e988272631c3c8b0eb04aa47cdca6968dc8d" Oct 30 13:12:02.312297 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:02.312246 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-484607-5684dfc6c6-8z79j" podUID="82af0f22-cb2d-41c5-b4a8-adcecc306998" containerName="payload-processing" containerID="cri-o://aa05590eeb7de7aaa57966fe245390736ced2c4977a13ccf7666622926e189b4" gracePeriod=25 Oct 30 13:12:02.346143 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:02.346116 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:12:02.347488 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:02.347459 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-484607-6d646d7567-hd7j6" podUID="62a8727a-ad11-4c04-87e1-0b198af32183" containerName="payload-pre-processing" containerID="cri-o://5603efccb4fa9200a032236fd33f3c45f3cf5b097848295ac27a877bd8aa8874" gracePeriod=25 Oct 30 13:12:02.388756 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:02.388723 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:04.236209 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:04.236171 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:05.035915 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:05.035839 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0b5e49f-bn8ws"] Oct 30 13:12:05.047884 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:05.047856 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0b5e49f-bn8ws"] Oct 30 13:12:05.645198 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:05.645164 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cab4ff5-dfd0-4a13-b88c-f895992dd9c2" path="/var/lib/kubelet/pods/3cab4ff5-dfd0-4a13-b88c-f895992dd9c2/volumes" Oct 30 13:12:06.102446 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:06.102409 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:06.743193 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:06.743153 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" podUID="dad5bdad-cd6f-41f1-a124-540c3524ead7" containerName="main" containerID="cri-o://330dec08aa5847cd63b4903ab3adf5835b08240b106b2e489044666aaa1c27c8" gracePeriod=45 Oct 30 13:12:06.774236 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:06.774202 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" podUID="b487a069-f519-45a3-b1d8-f32f2431797b" containerName="main" containerID="cri-o://76966256847acca3935051c44da611a8c16596f36e378fb4dc8c1cc4a0791df2" gracePeriod=45 Oct 30 13:12:06.926259 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:06.926207 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" podUID="dad5bdad-cd6f-41f1-a124-540c3524ead7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": dial tcp 10.134.0.74:8000: connect: connection refused" Oct 30 13:12:07.037755 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.037719 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e7b6e1d-shd4w"] Oct 30 13:12:07.052066 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.052040 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e7b6e1d-shd4w"] Oct 30 13:12:07.053530 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.053505 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" Oct 30 13:12:07.059266 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.059234 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bdd6e80-w87p7"] Oct 30 13:12:07.060204 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.060183 5922 cpu_manager.go:414] "containerMap: removing container" podUID="dad5bdad-cd6f-41f1-a124-540c3524ead7" containerName="main" Oct 30 13:12:07.060204 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.060204 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dad5bdad-cd6f-41f1-a124-540c3524ead7" containerName="main" Oct 30 13:12:07.060394 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.060217 5922 cpu_manager.go:414] "containerMap: removing container" podUID="76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07" containerName="revoke-keys" Oct 30 13:12:07.060394 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.060223 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07" containerName="revoke-keys" Oct 30 13:12:07.060394 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.060245 5922 cpu_manager.go:414] "containerMap: removing container" podUID="d02e9835-5ada-40e8-ba81-13f9f8fe6541" containerName="revoke-keys" Oct 30 13:12:07.060394 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.060250 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d02e9835-5ada-40e8-ba81-13f9f8fe6541" containerName="revoke-keys" Oct 30 13:12:07.060602 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.060434 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="dad5bdad-cd6f-41f1-a124-540c3524ead7" containerName="main" Oct 30 13:12:07.060602 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.060454 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07" containerName="revoke-keys" Oct 30 13:12:07.060602 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.060467 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="d02e9835-5ada-40e8-ba81-13f9f8fe6541" containerName="revoke-keys" Oct 30 13:12:07.064144 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.064126 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bdd6e80-w87p7" Oct 30 13:12:07.078152 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.078130 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bdd6e80-w87p7"] Oct 30 13:12:07.080239 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.080213 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:12:07.080387 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.080242 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:12:07.080486 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.080226 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:12:07.081063 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.081044 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:12:07.106995 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.106971 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" Oct 30 13:12:07.151820 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.151791 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dad5bdad-cd6f-41f1-a124-540c3524ead7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad5bdad-cd6f-41f1-a124-540c3524ead7-tmp-dir\") pod \"dad5bdad-cd6f-41f1-a124-540c3524ead7\" (UID: \"dad5bdad-cd6f-41f1-a124-540c3524ead7\") " Oct 30 13:12:07.151997 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.151843 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dad5bdad-cd6f-41f1-a124-540c3524ead7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad5bdad-cd6f-41f1-a124-540c3524ead7-model-cache\") pod \"dad5bdad-cd6f-41f1-a124-540c3524ead7\" (UID: \"dad5bdad-cd6f-41f1-a124-540c3524ead7\") " Oct 30 13:12:07.151997 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.151897 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dad5bdad-cd6f-41f1-a124-540c3524ead7-tls-certs\" (UniqueName: \"kubernetes.io/secret/dad5bdad-cd6f-41f1-a124-540c3524ead7-tls-certs\") pod \"dad5bdad-cd6f-41f1-a124-540c3524ead7\" (UID: \"dad5bdad-cd6f-41f1-a124-540c3524ead7\") " Oct 30 13:12:07.151997 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.151942 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dad5bdad-cd6f-41f1-a124-540c3524ead7-kube-api-access-qvf97\" (UniqueName: \"kubernetes.io/projected/dad5bdad-cd6f-41f1-a124-540c3524ead7-kube-api-access-qvf97\") pod \"dad5bdad-cd6f-41f1-a124-540c3524ead7\" (UID: \"dad5bdad-cd6f-41f1-a124-540c3524ead7\") " Oct 30 13:12:07.152145 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.152083 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dad5bdad-cd6f-41f1-a124-540c3524ead7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad5bdad-cd6f-41f1-a124-540c3524ead7-dshm\") pod \"dad5bdad-cd6f-41f1-a124-540c3524ead7\" (UID: \"dad5bdad-cd6f-41f1-a124-540c3524ead7\") " Oct 30 13:12:07.152145 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.152107 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad5bdad-cd6f-41f1-a124-540c3524ead7-model-cache" pod "dad5bdad-cd6f-41f1-a124-540c3524ead7" (UID: "dad5bdad-cd6f-41f1-a124-540c3524ead7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:07.152145 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.152124 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dad5bdad-cd6f-41f1-a124-540c3524ead7-home\" (UniqueName: \"kubernetes.io/empty-dir/dad5bdad-cd6f-41f1-a124-540c3524ead7-home\") pod \"dad5bdad-cd6f-41f1-a124-540c3524ead7\" (UID: \"dad5bdad-cd6f-41f1-a124-540c3524ead7\") " Oct 30 13:12:07.152639 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.152449 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dba2c02d-ec04-41d1-a34a-1929af79ad19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bdd6e80-w87p7\" (UID: \"dba2c02d-ec04-41d1-a34a-1929af79ad19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bdd6e80-w87p7" Oct 30 13:12:07.152754 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.152665 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dba2c02d-ec04-41d1-a34a-1929af79ad19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bdd6e80-w87p7\" (UID: \"dba2c02d-ec04-41d1-a34a-1929af79ad19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bdd6e80-w87p7" Oct 30 13:12:07.153106 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.152933 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6r7c\" (UniqueName: \"kubernetes.io/projected/dba2c02d-ec04-41d1-a34a-1929af79ad19-kube-api-access-h6r7c\") pod \"maas-api-revoke-sub-2bdd6e80-w87p7\" (UID: \"dba2c02d-ec04-41d1-a34a-1929af79ad19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bdd6e80-w87p7" Oct 30 13:12:07.153106 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.153060 5922 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad5bdad-cd6f-41f1-a124-540c3524ead7-model-cache\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:07.153408 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.153375 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad5bdad-cd6f-41f1-a124-540c3524ead7-home" pod "dad5bdad-cd6f-41f1-a124-540c3524ead7" (UID: "dad5bdad-cd6f-41f1-a124-540c3524ead7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:07.153909 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.153886 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad5bdad-cd6f-41f1-a124-540c3524ead7-tmp-dir" pod "dad5bdad-cd6f-41f1-a124-540c3524ead7" (UID: "dad5bdad-cd6f-41f1-a124-540c3524ead7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:07.156056 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.156030 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dad5bdad-cd6f-41f1-a124-540c3524ead7-tls-certs" pod "dad5bdad-cd6f-41f1-a124-540c3524ead7" (UID: "dad5bdad-cd6f-41f1-a124-540c3524ead7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:12:07.157530 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.157505 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad5bdad-cd6f-41f1-a124-540c3524ead7-dshm" pod "dad5bdad-cd6f-41f1-a124-540c3524ead7" (UID: "dad5bdad-cd6f-41f1-a124-540c3524ead7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:07.157656 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.157537 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dad5bdad-cd6f-41f1-a124-540c3524ead7-kube-api-access-qvf97" pod "dad5bdad-cd6f-41f1-a124-540c3524ead7" (UID: "dad5bdad-cd6f-41f1-a124-540c3524ead7"). InnerVolumeSpecName "kube-api-access-qvf97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:07.254029 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.253998 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b487a069-f519-45a3-b1d8-f32f2431797b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b487a069-f519-45a3-b1d8-f32f2431797b-model-cache\") pod \"b487a069-f519-45a3-b1d8-f32f2431797b\" (UID: \"b487a069-f519-45a3-b1d8-f32f2431797b\") " Oct 30 13:12:07.254195 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.254093 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b487a069-f519-45a3-b1d8-f32f2431797b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b487a069-f519-45a3-b1d8-f32f2431797b-tmp-dir\") pod \"b487a069-f519-45a3-b1d8-f32f2431797b\" (UID: \"b487a069-f519-45a3-b1d8-f32f2431797b\") " Oct 30 13:12:07.254258 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.254206 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b487a069-f519-45a3-b1d8-f32f2431797b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b487a069-f519-45a3-b1d8-f32f2431797b-dshm\") pod \"b487a069-f519-45a3-b1d8-f32f2431797b\" (UID: \"b487a069-f519-45a3-b1d8-f32f2431797b\") " Oct 30 13:12:07.254258 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.254238 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b487a069-f519-45a3-b1d8-f32f2431797b-home\" (UniqueName: \"kubernetes.io/empty-dir/b487a069-f519-45a3-b1d8-f32f2431797b-home\") pod \"b487a069-f519-45a3-b1d8-f32f2431797b\" (UID: \"b487a069-f519-45a3-b1d8-f32f2431797b\") " Oct 30 13:12:07.254380 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.254288 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b487a069-f519-45a3-b1d8-f32f2431797b-model-cache" pod "b487a069-f519-45a3-b1d8-f32f2431797b" (UID: "b487a069-f519-45a3-b1d8-f32f2431797b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:07.254380 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.254307 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b487a069-f519-45a3-b1d8-f32f2431797b-kube-api-access-v7twz\" (UniqueName: \"kubernetes.io/projected/b487a069-f519-45a3-b1d8-f32f2431797b-kube-api-access-v7twz\") pod \"b487a069-f519-45a3-b1d8-f32f2431797b\" (UID: \"b487a069-f519-45a3-b1d8-f32f2431797b\") " Oct 30 13:12:07.254380 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.254313 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b487a069-f519-45a3-b1d8-f32f2431797b-tmp-dir" pod "b487a069-f519-45a3-b1d8-f32f2431797b" (UID: "b487a069-f519-45a3-b1d8-f32f2431797b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:07.254380 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.254340 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b487a069-f519-45a3-b1d8-f32f2431797b-tls-certs\" (UniqueName: \"kubernetes.io/secret/b487a069-f519-45a3-b1d8-f32f2431797b-tls-certs\") pod \"b487a069-f519-45a3-b1d8-f32f2431797b\" (UID: \"b487a069-f519-45a3-b1d8-f32f2431797b\") " Oct 30 13:12:07.254564 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.254507 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b487a069-f519-45a3-b1d8-f32f2431797b-home" pod "b487a069-f519-45a3-b1d8-f32f2431797b" (UID: "b487a069-f519-45a3-b1d8-f32f2431797b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:07.254734 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.254713 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6r7c\" (UniqueName: \"kubernetes.io/projected/dba2c02d-ec04-41d1-a34a-1929af79ad19-kube-api-access-h6r7c\") pod \"maas-api-revoke-sub-2bdd6e80-w87p7\" (UID: \"dba2c02d-ec04-41d1-a34a-1929af79ad19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bdd6e80-w87p7" Oct 30 13:12:07.254843 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.254830 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dba2c02d-ec04-41d1-a34a-1929af79ad19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bdd6e80-w87p7\" (UID: \"dba2c02d-ec04-41d1-a34a-1929af79ad19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bdd6e80-w87p7" Oct 30 13:12:07.254985 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.254970 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dba2c02d-ec04-41d1-a34a-1929af79ad19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bdd6e80-w87p7\" (UID: \"dba2c02d-ec04-41d1-a34a-1929af79ad19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bdd6e80-w87p7" Oct 30 13:12:07.255103 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.255088 5922 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b487a069-f519-45a3-b1d8-f32f2431797b-model-cache\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:07.255174 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.255110 5922 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b487a069-f519-45a3-b1d8-f32f2431797b-tmp-dir\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:07.255174 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.255124 5922 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad5bdad-cd6f-41f1-a124-540c3524ead7-tmp-dir\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:07.255174 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.255136 5922 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b487a069-f519-45a3-b1d8-f32f2431797b-home\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:07.255174 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.255148 5922 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad5bdad-cd6f-41f1-a124-540c3524ead7-tls-certs\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:07.255174 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.255164 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvf97\" (UniqueName: \"kubernetes.io/projected/dad5bdad-cd6f-41f1-a124-540c3524ead7-kube-api-access-qvf97\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:07.255339 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.255177 5922 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad5bdad-cd6f-41f1-a124-540c3524ead7-dshm\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:07.255339 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.255189 5922 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad5bdad-cd6f-41f1-a124-540c3524ead7-home\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:07.255727 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.255697 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dba2c02d-ec04-41d1-a34a-1929af79ad19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bdd6e80-w87p7\" (UID: \"dba2c02d-ec04-41d1-a34a-1929af79ad19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bdd6e80-w87p7" Oct 30 13:12:07.256339 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.256314 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b487a069-f519-45a3-b1d8-f32f2431797b-dshm" pod "b487a069-f519-45a3-b1d8-f32f2431797b" (UID: "b487a069-f519-45a3-b1d8-f32f2431797b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:07.256511 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.256485 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b487a069-f519-45a3-b1d8-f32f2431797b-tls-certs" pod "b487a069-f519-45a3-b1d8-f32f2431797b" (UID: "b487a069-f519-45a3-b1d8-f32f2431797b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:12:07.256578 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.256551 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b487a069-f519-45a3-b1d8-f32f2431797b-kube-api-access-v7twz" pod "b487a069-f519-45a3-b1d8-f32f2431797b" (UID: "b487a069-f519-45a3-b1d8-f32f2431797b"). InnerVolumeSpecName "kube-api-access-v7twz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:07.257196 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.257177 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dba2c02d-ec04-41d1-a34a-1929af79ad19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bdd6e80-w87p7\" (UID: \"dba2c02d-ec04-41d1-a34a-1929af79ad19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bdd6e80-w87p7" Oct 30 13:12:07.279164 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.279109 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6r7c\" (UniqueName: \"kubernetes.io/projected/dba2c02d-ec04-41d1-a34a-1929af79ad19-kube-api-access-h6r7c\") pod \"maas-api-revoke-sub-2bdd6e80-w87p7\" (UID: \"dba2c02d-ec04-41d1-a34a-1929af79ad19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bdd6e80-w87p7" Oct 30 13:12:07.302529 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.302501 5922 generic.go:356] "Generic (PLEG): container finished" podID="dad5bdad-cd6f-41f1-a124-540c3524ead7" containerID="330dec08aa5847cd63b4903ab3adf5835b08240b106b2e489044666aaa1c27c8" exitCode=0 Oct 30 13:12:07.302700 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.302547 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" Oct 30 13:12:07.302700 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.302584 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" event={"ID":"dad5bdad-cd6f-41f1-a124-540c3524ead7","Type":"ContainerDied","Data":"330dec08aa5847cd63b4903ab3adf5835b08240b106b2e489044666aaa1c27c8"} Oct 30 13:12:07.302700 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.302644 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7" event={"ID":"dad5bdad-cd6f-41f1-a124-540c3524ead7","Type":"ContainerDied","Data":"132b93fb8371cf88a6f56f12a8802392cc36659824f911aa504b8889d6062727"} Oct 30 13:12:07.302700 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.302668 5922 scope.go:122] "RemoveContainer" containerID="330dec08aa5847cd63b4903ab3adf5835b08240b106b2e489044666aaa1c27c8" Oct 30 13:12:07.304458 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.304432 5922 generic.go:356] "Generic (PLEG): container finished" podID="b487a069-f519-45a3-b1d8-f32f2431797b" containerID="76966256847acca3935051c44da611a8c16596f36e378fb4dc8c1cc4a0791df2" exitCode=0 Oct 30 13:12:07.304576 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.304464 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" event={"ID":"b487a069-f519-45a3-b1d8-f32f2431797b","Type":"ContainerDied","Data":"76966256847acca3935051c44da611a8c16596f36e378fb4dc8c1cc4a0791df2"} Oct 30 13:12:07.304576 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.304489 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" Oct 30 13:12:07.304576 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.304504 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5" event={"ID":"b487a069-f519-45a3-b1d8-f32f2431797b","Type":"ContainerDied","Data":"9fc2085328042296bd03cde91b5ad97199a81d01bd400dd6d6a72e934d9c0eb1"} Oct 30 13:12:07.315500 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.315482 5922 scope.go:122] "RemoveContainer" containerID="330dec08aa5847cd63b4903ab3adf5835b08240b106b2e489044666aaa1c27c8" Oct 30 13:12:07.315792 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:12:07.315769 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"330dec08aa5847cd63b4903ab3adf5835b08240b106b2e489044666aaa1c27c8\": container with ID starting with 330dec08aa5847cd63b4903ab3adf5835b08240b106b2e489044666aaa1c27c8 not found: ID does not exist" containerID="330dec08aa5847cd63b4903ab3adf5835b08240b106b2e489044666aaa1c27c8" Oct 30 13:12:07.315865 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.315799 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"330dec08aa5847cd63b4903ab3adf5835b08240b106b2e489044666aaa1c27c8"} err="failed to get container status \"330dec08aa5847cd63b4903ab3adf5835b08240b106b2e489044666aaa1c27c8\": rpc error: code = NotFound desc = could not find container \"330dec08aa5847cd63b4903ab3adf5835b08240b106b2e489044666aaa1c27c8\": container with ID starting with 330dec08aa5847cd63b4903ab3adf5835b08240b106b2e489044666aaa1c27c8 not found: ID does not exist" Oct 30 13:12:07.315865 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.315816 5922 scope.go:122] "RemoveContainer" containerID="76966256847acca3935051c44da611a8c16596f36e378fb4dc8c1cc4a0791df2" Oct 30 13:12:07.325356 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.325340 5922 scope.go:122] "RemoveContainer" containerID="76966256847acca3935051c44da611a8c16596f36e378fb4dc8c1cc4a0791df2" Oct 30 13:12:07.325622 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:12:07.325590 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76966256847acca3935051c44da611a8c16596f36e378fb4dc8c1cc4a0791df2\": container with ID starting with 76966256847acca3935051c44da611a8c16596f36e378fb4dc8c1cc4a0791df2 not found: ID does not exist" containerID="76966256847acca3935051c44da611a8c16596f36e378fb4dc8c1cc4a0791df2" Oct 30 13:12:07.325692 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.325632 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76966256847acca3935051c44da611a8c16596f36e378fb4dc8c1cc4a0791df2"} err="failed to get container status \"76966256847acca3935051c44da611a8c16596f36e378fb4dc8c1cc4a0791df2\": rpc error: code = NotFound desc = could not find container \"76966256847acca3935051c44da611a8c16596f36e378fb4dc8c1cc4a0791df2\": container with ID starting with 76966256847acca3935051c44da611a8c16596f36e378fb4dc8c1cc4a0791df2 not found: ID does not exist" Oct 30 13:12:07.356053 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.356031 5922 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b487a069-f519-45a3-b1d8-f32f2431797b-dshm\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:07.356053 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.356054 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7twz\" (UniqueName: \"kubernetes.io/projected/b487a069-f519-45a3-b1d8-f32f2431797b-kube-api-access-v7twz\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:07.356183 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.356064 5922 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b487a069-f519-45a3-b1d8-f32f2431797b-tls-certs\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:07.360407 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.360382 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5"] Oct 30 13:12:07.366089 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.366070 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-484607/simulator-w6-484607-kserve-85f449c7dd-b5gv5"] Oct 30 13:12:07.402318 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.402299 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bdd6e80-w87p7" Oct 30 13:12:07.402505 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.402488 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7"] Oct 30 13:12:07.411132 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.411109 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-484607/distinct-2-w6-484607-kserve-5c45cc7ff-sgxw7"] Oct 30 13:12:07.646103 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.646031 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3547727-fc7f-4d93-8d50-0a21fd349aa6" path="/var/lib/kubelet/pods/a3547727-fc7f-4d93-8d50-0a21fd349aa6/volumes" Oct 30 13:12:07.646450 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.646437 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b487a069-f519-45a3-b1d8-f32f2431797b" path="/var/lib/kubelet/pods/b487a069-f519-45a3-b1d8-f32f2431797b/volumes" Oct 30 13:12:07.646876 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.646861 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dad5bdad-cd6f-41f1-a124-540c3524ead7" path="/var/lib/kubelet/pods/dad5bdad-cd6f-41f1-a124-540c3524ead7/volumes" Oct 30 13:12:07.711355 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.711330 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bdd6e80-w87p7"] Oct 30 13:12:07.712067 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:12:07.712042 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddba2c02d_ec04_41d1_a34a_1929af79ad19.slice/crio-d221b3a6ca4867cb09452843ed102bb2604afc49366d99c96602bfca9055ceef WatchSource:0}: Error finding container d221b3a6ca4867cb09452843ed102bb2604afc49366d99c96602bfca9055ceef: Status 404 returned error can't find the container with id d221b3a6ca4867cb09452843ed102bb2604afc49366d99c96602bfca9055ceef Oct 30 13:12:07.871248 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:07.871217 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:08.316026 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:08.315956 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bdd6e80-w87p7" event={"ID":"dba2c02d-ec04-41d1-a34a-1929af79ad19","Type":"ContainerStarted","Data":"8db64bf786211700edec801e519a453da01f0e61bb1751ccf51bea7fcf411c36"} Oct 30 13:12:08.316026 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:08.316004 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bdd6e80-w87p7" event={"ID":"dba2c02d-ec04-41d1-a34a-1929af79ad19","Type":"ContainerStarted","Data":"d221b3a6ca4867cb09452843ed102bb2604afc49366d99c96602bfca9055ceef"} Oct 30 13:12:08.360495 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:08.360445 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bdd6e80-w87p7" podStartSLOduration=1.3604329339999999 podStartE2EDuration="1.360432934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:08.357707998 +0000 UTC m=+1184.827123761" watchObservedRunningTime="2026-10-30 13:12:08.360432934 +0000 UTC m=+1184.829848730" Oct 30 13:12:09.324900 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:09.324866 5922 generic.go:356] "Generic (PLEG): container finished" podID="dba2c02d-ec04-41d1-a34a-1929af79ad19" containerID="8db64bf786211700edec801e519a453da01f0e61bb1751ccf51bea7fcf411c36" exitCode=0 Oct 30 13:12:09.325277 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:09.324988 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bdd6e80-w87p7" event={"ID":"dba2c02d-ec04-41d1-a34a-1929af79ad19","Type":"ContainerDied","Data":"8db64bf786211700edec801e519a453da01f0e61bb1751ccf51bea7fcf411c36"} Oct 30 13:12:09.725836 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:09.725801 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:10.305790 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:10.305745 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-484607-5684dfc6c6-8z79j" podUID="82af0f22-cb2d-41c5-b4a8-adcecc306998" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:12:10.305969 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:10.305845 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-484607-6d646d7567-hd7j6" podUID="62a8727a-ad11-4c04-87e1-0b198af32183" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:12:10.499986 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:10.499961 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bdd6e80-w87p7" Oct 30 13:12:10.594924 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:10.594827 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dba2c02d-ec04-41d1-a34a-1929af79ad19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dba2c02d-ec04-41d1-a34a-1929af79ad19-maas-api-openshift-ca\") pod \"dba2c02d-ec04-41d1-a34a-1929af79ad19\" (UID: \"dba2c02d-ec04-41d1-a34a-1929af79ad19\") " Oct 30 13:12:10.595100 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:10.594983 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dba2c02d-ec04-41d1-a34a-1929af79ad19-kube-api-access-h6r7c\" (UniqueName: \"kubernetes.io/projected/dba2c02d-ec04-41d1-a34a-1929af79ad19-kube-api-access-h6r7c\") pod \"dba2c02d-ec04-41d1-a34a-1929af79ad19\" (UID: \"dba2c02d-ec04-41d1-a34a-1929af79ad19\") " Oct 30 13:12:10.595100 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:10.595035 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dba2c02d-ec04-41d1-a34a-1929af79ad19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dba2c02d-ec04-41d1-a34a-1929af79ad19-maas-api-xks-ca\") pod \"dba2c02d-ec04-41d1-a34a-1929af79ad19\" (UID: \"dba2c02d-ec04-41d1-a34a-1929af79ad19\") " Oct 30 13:12:10.595262 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:10.595232 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dba2c02d-ec04-41d1-a34a-1929af79ad19-maas-api-openshift-ca" pod "dba2c02d-ec04-41d1-a34a-1929af79ad19" (UID: "dba2c02d-ec04-41d1-a34a-1929af79ad19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:12:10.595846 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:10.595820 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dba2c02d-ec04-41d1-a34a-1929af79ad19-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:10.597240 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:10.597218 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dba2c02d-ec04-41d1-a34a-1929af79ad19-kube-api-access-h6r7c" pod "dba2c02d-ec04-41d1-a34a-1929af79ad19" (UID: "dba2c02d-ec04-41d1-a34a-1929af79ad19"). InnerVolumeSpecName "kube-api-access-h6r7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:10.597657 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:10.597638 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dba2c02d-ec04-41d1-a34a-1929af79ad19-maas-api-xks-ca" pod "dba2c02d-ec04-41d1-a34a-1929af79ad19" (UID: "dba2c02d-ec04-41d1-a34a-1929af79ad19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:12:10.697270 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:10.697233 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6r7c\" (UniqueName: \"kubernetes.io/projected/dba2c02d-ec04-41d1-a34a-1929af79ad19-kube-api-access-h6r7c\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:10.697270 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:10.697267 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dba2c02d-ec04-41d1-a34a-1929af79ad19-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:11.355913 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:11.355888 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bdd6e80-w87p7" Oct 30 13:12:11.356058 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:11.355921 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bdd6e80-w87p7" event={"ID":"dba2c02d-ec04-41d1-a34a-1929af79ad19","Type":"ContainerDied","Data":"d221b3a6ca4867cb09452843ed102bb2604afc49366d99c96602bfca9055ceef"} Oct 30 13:12:11.356058 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:11.355970 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d221b3a6ca4867cb09452843ed102bb2604afc49366d99c96602bfca9055ceef" Oct 30 13:12:11.412431 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:11.412396 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:13.002186 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:13.002138 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:13.559971 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:13.559929 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:13.784138 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:13.784097 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:13.828373 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:13.828286 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-484607-vs47h"] Oct 30 13:12:13.838895 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:13.838864 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-484607-vs47h"] Oct 30 13:12:13.949991 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:13.949960 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:14.327542 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:14.327502 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:15.645318 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:15.645286 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd0296bd-3eee-4fa4-896b-c6e1585a77c3" path="/var/lib/kubelet/pods/bd0296bd-3eee-4fa4-896b-c6e1585a77c3/volumes" Oct 30 13:12:15.787787 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:15.787749 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:17.674177 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:17.674145 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:17.987974 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:17.987943 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:12:18.296102 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:18.296021 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:18.521820 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:18.521784 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ac0c58-kn9tt"] Oct 30 13:12:18.522396 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:18.522380 5922 cpu_manager.go:414] "containerMap: removing container" podUID="dba2c02d-ec04-41d1-a34a-1929af79ad19" containerName="revoke-keys" Oct 30 13:12:18.522446 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:18.522398 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dba2c02d-ec04-41d1-a34a-1929af79ad19" containerName="revoke-keys" Oct 30 13:12:18.522446 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:18.522416 5922 cpu_manager.go:414] "containerMap: removing container" podUID="b487a069-f519-45a3-b1d8-f32f2431797b" containerName="main" Oct 30 13:12:18.522446 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:18.522421 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b487a069-f519-45a3-b1d8-f32f2431797b" containerName="main" Oct 30 13:12:18.522547 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:18.522512 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="dba2c02d-ec04-41d1-a34a-1929af79ad19" containerName="revoke-keys" Oct 30 13:12:18.522547 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:18.522523 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="b487a069-f519-45a3-b1d8-f32f2431797b" containerName="main" Oct 30 13:12:18.526496 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:18.526479 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ac0c58-kn9tt" Oct 30 13:12:18.545655 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:18.545630 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ac0c58-kn9tt"] Oct 30 13:12:18.552428 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:18.552363 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:12:18.552428 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:18.552406 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:12:18.553805 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:18.553787 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:12:18.553805 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:18.553795 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:12:18.595905 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:18.595874 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7550010c-6165-4d5f-90bf-52872faa8a8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-ac0c58-kn9tt\" (UID: \"7550010c-6165-4d5f-90bf-52872faa8a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ac0c58-kn9tt" Oct 30 13:12:18.596063 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:18.595919 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj26x\" (UniqueName: \"kubernetes.io/projected/7550010c-6165-4d5f-90bf-52872faa8a8c-kube-api-access-qj26x\") pod \"maas-api-revoke-keys-e2e-worker-w1-ac0c58-kn9tt\" (UID: \"7550010c-6165-4d5f-90bf-52872faa8a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ac0c58-kn9tt" Oct 30 13:12:18.596063 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:18.595994 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7550010c-6165-4d5f-90bf-52872faa8a8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-ac0c58-kn9tt\" (UID: \"7550010c-6165-4d5f-90bf-52872faa8a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ac0c58-kn9tt" Oct 30 13:12:18.707625 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:18.707565 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7550010c-6165-4d5f-90bf-52872faa8a8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-ac0c58-kn9tt\" (UID: \"7550010c-6165-4d5f-90bf-52872faa8a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ac0c58-kn9tt" Oct 30 13:12:18.708125 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:18.707655 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj26x\" (UniqueName: \"kubernetes.io/projected/7550010c-6165-4d5f-90bf-52872faa8a8c-kube-api-access-qj26x\") pod \"maas-api-revoke-keys-e2e-worker-w1-ac0c58-kn9tt\" (UID: \"7550010c-6165-4d5f-90bf-52872faa8a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ac0c58-kn9tt" Oct 30 13:12:18.708125 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:18.707717 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7550010c-6165-4d5f-90bf-52872faa8a8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-ac0c58-kn9tt\" (UID: \"7550010c-6165-4d5f-90bf-52872faa8a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ac0c58-kn9tt" Oct 30 13:12:18.708421 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:18.708399 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7550010c-6165-4d5f-90bf-52872faa8a8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-ac0c58-kn9tt\" (UID: \"7550010c-6165-4d5f-90bf-52872faa8a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ac0c58-kn9tt" Oct 30 13:12:18.711751 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:18.711724 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7550010c-6165-4d5f-90bf-52872faa8a8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-ac0c58-kn9tt\" (UID: \"7550010c-6165-4d5f-90bf-52872faa8a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ac0c58-kn9tt" Oct 30 13:12:18.733509 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:18.733481 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj26x\" (UniqueName: \"kubernetes.io/projected/7550010c-6165-4d5f-90bf-52872faa8a8c-kube-api-access-qj26x\") pod \"maas-api-revoke-keys-e2e-worker-w1-ac0c58-kn9tt\" (UID: \"7550010c-6165-4d5f-90bf-52872faa8a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ac0c58-kn9tt" Oct 30 13:12:18.840980 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:18.840872 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ac0c58-kn9tt" Oct 30 13:12:19.183421 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:19.183389 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ac0c58-kn9tt"] Oct 30 13:12:19.189173 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:12:19.189144 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7550010c_6165_4d5f_90bf_52872faa8a8c.slice/crio-3c4a1d815c359fc725612715773cf7d11d5458a6018ee12c116089a401dbbbeb WatchSource:0}: Error finding container 3c4a1d815c359fc725612715773cf7d11d5458a6018ee12c116089a401dbbbeb: Status 404 returned error can't find the container with id 3c4a1d815c359fc725612715773cf7d11d5458a6018ee12c116089a401dbbbeb Oct 30 13:12:19.442576 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:19.442480 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ac0c58-kn9tt" event={"ID":"7550010c-6165-4d5f-90bf-52872faa8a8c","Type":"ContainerStarted","Data":"3c4a1d815c359fc725612715773cf7d11d5458a6018ee12c116089a401dbbbeb"} Oct 30 13:12:20.305021 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:20.304983 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-484607-5684dfc6c6-8z79j" podUID="82af0f22-cb2d-41c5-b4a8-adcecc306998" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:12:20.305478 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:20.305271 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-484607-6d646d7567-hd7j6" podUID="62a8727a-ad11-4c04-87e1-0b198af32183" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:12:20.452299 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:20.452268 5922 generic.go:356] "Generic (PLEG): container finished" podID="7550010c-6165-4d5f-90bf-52872faa8a8c" containerID="d1d2f917c7a798cbdc510cae1348a6abff58464f94b6a230aa571610620596d2" exitCode=0 Oct 30 13:12:20.452447 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:20.452348 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ac0c58-kn9tt" event={"ID":"7550010c-6165-4d5f-90bf-52872faa8a8c","Type":"ContainerDied","Data":"d1d2f917c7a798cbdc510cae1348a6abff58464f94b6a230aa571610620596d2"} Oct 30 13:12:21.275835 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:21.275802 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:21.535265 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:21.535127 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:21.635800 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:21.635766 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ac0c58-kn9tt" Oct 30 13:12:21.655932 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:21.655903 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:21.691514 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:21.691481 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7550010c-6165-4d5f-90bf-52872faa8a8c-kube-api-access-qj26x\" (UniqueName: \"kubernetes.io/projected/7550010c-6165-4d5f-90bf-52872faa8a8c-kube-api-access-qj26x\") pod \"7550010c-6165-4d5f-90bf-52872faa8a8c\" (UID: \"7550010c-6165-4d5f-90bf-52872faa8a8c\") " Oct 30 13:12:21.691765 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:21.691743 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7550010c-6165-4d5f-90bf-52872faa8a8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7550010c-6165-4d5f-90bf-52872faa8a8c-maas-api-openshift-ca\") pod \"7550010c-6165-4d5f-90bf-52872faa8a8c\" (UID: \"7550010c-6165-4d5f-90bf-52872faa8a8c\") " Oct 30 13:12:21.691854 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:21.691839 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7550010c-6165-4d5f-90bf-52872faa8a8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7550010c-6165-4d5f-90bf-52872faa8a8c-maas-api-xks-ca\") pod \"7550010c-6165-4d5f-90bf-52872faa8a8c\" (UID: \"7550010c-6165-4d5f-90bf-52872faa8a8c\") " Oct 30 13:12:21.692165 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:21.692133 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7550010c-6165-4d5f-90bf-52872faa8a8c-maas-api-openshift-ca" pod "7550010c-6165-4d5f-90bf-52872faa8a8c" (UID: "7550010c-6165-4d5f-90bf-52872faa8a8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:12:21.692634 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:21.692585 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7550010c-6165-4d5f-90bf-52872faa8a8c-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:21.694498 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:21.694467 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7550010c-6165-4d5f-90bf-52872faa8a8c-maas-api-xks-ca" pod "7550010c-6165-4d5f-90bf-52872faa8a8c" (UID: "7550010c-6165-4d5f-90bf-52872faa8a8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:12:21.694711 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:21.694691 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7550010c-6165-4d5f-90bf-52872faa8a8c-kube-api-access-qj26x" pod "7550010c-6165-4d5f-90bf-52872faa8a8c" (UID: "7550010c-6165-4d5f-90bf-52872faa8a8c"). InnerVolumeSpecName "kube-api-access-qj26x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:21.793251 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:21.793194 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7550010c-6165-4d5f-90bf-52872faa8a8c-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:21.793251 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:21.793218 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj26x\" (UniqueName: \"kubernetes.io/projected/7550010c-6165-4d5f-90bf-52872faa8a8c-kube-api-access-qj26x\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:21.900576 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:21.900536 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47"] Oct 30 13:12:21.900828 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:21.900801 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" podUID="d62b6ccd-e8a1-4320-87d0-0b12377ad8ee" containerName="istio-proxy" containerID="cri-o://f169a2618499b3b38b132015ed8e2245e35c526f25ddea2745e124a56074a928" gracePeriod=30 Oct 30 13:12:21.927766 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:21.927731 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:21.940031 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:21.934534 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks"] Oct 30 13:12:21.940031 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:21.934769 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" podUID="d0132701-3b77-4cce-9989-a0d03ac479f3" containerName="istio-proxy" containerID="cri-o://2ead0939b95900ae542eba74cecb73cc7b39574b7015929f9fef7f6bc367af75" gracePeriod=30 Oct 30 13:12:21.940260 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:21.940141 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m"] Oct 30 13:12:21.940444 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:21.940364 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" podUID="7d034a1b-2482-4f51-b89c-6300d660b7ee" containerName="istio-proxy" containerID="cri-o://3bc9462e86526c0163c87edb7331ffec30154f643fe4ba31aebcfad8f7d9a8cf" gracePeriod=30 Oct 30 13:12:22.081592 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:22.079717 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c763adca-hn8ln"] Oct 30 13:12:22.088674 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:22.086962 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c763adca-hn8ln"] Oct 30 13:12:22.480262 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:22.480228 5922 generic.go:356] "Generic (PLEG): container finished" podID="62a8727a-ad11-4c04-87e1-0b198af32183" containerID="5603efccb4fa9200a032236fd33f3c45f3cf5b097848295ac27a877bd8aa8874" exitCode=0 Oct 30 13:12:22.480392 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:22.480272 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-484607-6d646d7567-hd7j6" event={"ID":"62a8727a-ad11-4c04-87e1-0b198af32183","Type":"ContainerDied","Data":"5603efccb4fa9200a032236fd33f3c45f3cf5b097848295ac27a877bd8aa8874"} Oct 30 13:12:22.483120 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:22.483095 5922 generic.go:356] "Generic (PLEG): container finished" podID="82af0f22-cb2d-41c5-b4a8-adcecc306998" containerID="aa05590eeb7de7aaa57966fe245390736ced2c4977a13ccf7666622926e189b4" exitCode=0 Oct 30 13:12:22.483556 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:22.483292 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-484607-5684dfc6c6-8z79j" event={"ID":"82af0f22-cb2d-41c5-b4a8-adcecc306998","Type":"ContainerDied","Data":"aa05590eeb7de7aaa57966fe245390736ced2c4977a13ccf7666622926e189b4"} Oct 30 13:12:22.485450 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:22.485429 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ac0c58-kn9tt" event={"ID":"7550010c-6165-4d5f-90bf-52872faa8a8c","Type":"ContainerDied","Data":"3c4a1d815c359fc725612715773cf7d11d5458a6018ee12c116089a401dbbbeb"} Oct 30 13:12:22.485740 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:22.485714 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c4a1d815c359fc725612715773cf7d11d5458a6018ee12c116089a401dbbbeb" Oct 30 13:12:22.485740 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:22.485588 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ac0c58-kn9tt" Oct 30 13:12:22.602545 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:22.602518 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-484607-5684dfc6c6-8z79j" Oct 30 13:12:22.690419 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:22.690391 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-484607-6d646d7567-hd7j6" Oct 30 13:12:22.718095 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:22.718070 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82af0f22-cb2d-41c5-b4a8-adcecc306998-kube-api-access-mhrdt\" (UniqueName: \"kubernetes.io/projected/82af0f22-cb2d-41c5-b4a8-adcecc306998-kube-api-access-mhrdt\") pod \"82af0f22-cb2d-41c5-b4a8-adcecc306998\" (UID: \"82af0f22-cb2d-41c5-b4a8-adcecc306998\") " Oct 30 13:12:22.718247 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:22.718137 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82af0f22-cb2d-41c5-b4a8-adcecc306998-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82af0f22-cb2d-41c5-b4a8-adcecc306998-plugins-config-volume\") pod \"82af0f22-cb2d-41c5-b4a8-adcecc306998\" (UID: \"82af0f22-cb2d-41c5-b4a8-adcecc306998\") " Oct 30 13:12:22.718518 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:22.718498 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82af0f22-cb2d-41c5-b4a8-adcecc306998-plugins-config-volume" pod "82af0f22-cb2d-41c5-b4a8-adcecc306998" (UID: "82af0f22-cb2d-41c5-b4a8-adcecc306998"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:12:22.718766 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:22.718744 5922 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82af0f22-cb2d-41c5-b4a8-adcecc306998-plugins-config-volume\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:22.720138 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:22.720112 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82af0f22-cb2d-41c5-b4a8-adcecc306998-kube-api-access-mhrdt" pod "82af0f22-cb2d-41c5-b4a8-adcecc306998" (UID: "82af0f22-cb2d-41c5-b4a8-adcecc306998"). InnerVolumeSpecName "kube-api-access-mhrdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:22.819329 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:22.819303 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62a8727a-ad11-4c04-87e1-0b198af32183-kube-api-access-t2j7d\" (UniqueName: \"kubernetes.io/projected/62a8727a-ad11-4c04-87e1-0b198af32183-kube-api-access-t2j7d\") pod \"62a8727a-ad11-4c04-87e1-0b198af32183\" (UID: \"62a8727a-ad11-4c04-87e1-0b198af32183\") " Oct 30 13:12:22.819477 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:22.819337 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62a8727a-ad11-4c04-87e1-0b198af32183-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62a8727a-ad11-4c04-87e1-0b198af32183-plugins-config-volume\") pod \"62a8727a-ad11-4c04-87e1-0b198af32183\" (UID: \"62a8727a-ad11-4c04-87e1-0b198af32183\") " Oct 30 13:12:22.819750 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:22.819728 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhrdt\" (UniqueName: \"kubernetes.io/projected/82af0f22-cb2d-41c5-b4a8-adcecc306998-kube-api-access-mhrdt\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:22.819750 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:22.819736 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62a8727a-ad11-4c04-87e1-0b198af32183-plugins-config-volume" pod "62a8727a-ad11-4c04-87e1-0b198af32183" (UID: "62a8727a-ad11-4c04-87e1-0b198af32183"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:12:22.821363 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:22.821333 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62a8727a-ad11-4c04-87e1-0b198af32183-kube-api-access-t2j7d" pod "62a8727a-ad11-4c04-87e1-0b198af32183" (UID: "62a8727a-ad11-4c04-87e1-0b198af32183"). InnerVolumeSpecName "kube-api-access-t2j7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:22.921304 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:22.921237 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2j7d\" (UniqueName: \"kubernetes.io/projected/62a8727a-ad11-4c04-87e1-0b198af32183-kube-api-access-t2j7d\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:22.921304 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:22.921265 5922 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62a8727a-ad11-4c04-87e1-0b198af32183-plugins-config-volume\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:23.160318 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:23.160274 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:23.397089 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:23.397039 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:23.495097 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:23.495067 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-484607-5684dfc6c6-8z79j" Oct 30 13:12:23.495279 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:23.495105 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-484607-5684dfc6c6-8z79j" event={"ID":"82af0f22-cb2d-41c5-b4a8-adcecc306998","Type":"ContainerDied","Data":"5f128ba84869d6da51d064f08e27df44dfc1f4c312a1643484d160af1d3974bc"} Oct 30 13:12:23.495279 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:23.495161 5922 scope.go:122] "RemoveContainer" containerID="aa05590eeb7de7aaa57966fe245390736ced2c4977a13ccf7666622926e189b4" Oct 30 13:12:23.496810 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:23.496788 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-484607-6d646d7567-hd7j6" Oct 30 13:12:23.496975 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:23.496873 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-484607-6d646d7567-hd7j6" event={"ID":"62a8727a-ad11-4c04-87e1-0b198af32183","Type":"ContainerDied","Data":"c27c994131b6c9d5206cc9ecdc4ec0e6c70c0d4b9f416064dc15f9880b065d62"} Oct 30 13:12:23.506591 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:23.506571 5922 scope.go:122] "RemoveContainer" containerID="5603efccb4fa9200a032236fd33f3c45f3cf5b097848295ac27a877bd8aa8874" Oct 30 13:12:23.552280 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:12:23.552251 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:12:23.559558 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:23.559533 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-484607-5684dfc6c6-8z79j"] Oct 30 13:12:23.570435 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:23.570412 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-484607-5684dfc6c6-8z79j"] Oct 30 13:12:23.598989 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:23.598967 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-484607-6d646d7567-hd7j6"] Oct 30 13:12:23.602940 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:23.602919 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-484607-6d646d7567-hd7j6"] Oct 30 13:12:23.645246 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:23.645224 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62a8727a-ad11-4c04-87e1-0b198af32183" path="/var/lib/kubelet/pods/62a8727a-ad11-4c04-87e1-0b198af32183/volumes" Oct 30 13:12:23.645601 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:23.645589 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82af0f22-cb2d-41c5-b4a8-adcecc306998" path="/var/lib/kubelet/pods/82af0f22-cb2d-41c5-b4a8-adcecc306998/volumes" Oct 30 13:12:23.646038 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:23.646023 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="967aa55f-1503-41bd-8089-a909532dd160" path="/var/lib/kubelet/pods/967aa55f-1503-41bd-8089-a909532dd160/volumes" Oct 30 13:12:24.186371 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:24.186340 5922 scope.go:122] "RemoveContainer" containerID="a9e66db09439796f2e60293ac8baf2f0b8fe97ccc3eff288c7cfa2fd6dad9caf" Oct 30 13:12:25.072496 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:25.072460 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:25.937972 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:25.937933 5922 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:12:25.938140 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:25.937989 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" podUID="d62b6ccd-e8a1-4320-87d0-0b12377ad8ee" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:12:26.723899 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:26.723866 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:27.035293 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:12:27.035262 5922 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d034a1b_2482_4f51_b89c_6300d660b7ee.slice/crio-3bc9462e86526c0163c87edb7331ffec30154f643fe4ba31aebcfad8f7d9a8cf.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d034a1b_2482_4f51_b89c_6300d660b7ee.slice/crio-conmon-3bc9462e86526c0163c87edb7331ffec30154f643fe4ba31aebcfad8f7d9a8cf.scope\": RecentStats: unable to find data in memory cache]" Oct 30 13:12:27.082264 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.082205 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:27.226351 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.226325 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" Oct 30 13:12:27.235958 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.235900 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-54994c-7mkl6"] Oct 30 13:12:27.236480 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.236466 5922 cpu_manager.go:414] "containerMap: removing container" podUID="7550010c-6165-4d5f-90bf-52872faa8a8c" containerName="revoke-keys" Oct 30 13:12:27.236534 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.236483 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7550010c-6165-4d5f-90bf-52872faa8a8c" containerName="revoke-keys" Oct 30 13:12:27.236534 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.236506 5922 cpu_manager.go:414] "containerMap: removing container" podUID="62a8727a-ad11-4c04-87e1-0b198af32183" containerName="payload-pre-processing" Oct 30 13:12:27.236534 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.236514 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62a8727a-ad11-4c04-87e1-0b198af32183" containerName="payload-pre-processing" Oct 30 13:12:27.236534 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.236528 5922 cpu_manager.go:414] "containerMap: removing container" podUID="82af0f22-cb2d-41c5-b4a8-adcecc306998" containerName="payload-processing" Oct 30 13:12:27.236698 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.236536 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82af0f22-cb2d-41c5-b4a8-adcecc306998" containerName="payload-processing" Oct 30 13:12:27.236698 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.236547 5922 cpu_manager.go:414] "containerMap: removing container" podUID="d0132701-3b77-4cce-9989-a0d03ac479f3" containerName="istio-proxy" Oct 30 13:12:27.236698 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.236554 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0132701-3b77-4cce-9989-a0d03ac479f3" containerName="istio-proxy" Oct 30 13:12:27.236851 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.236714 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="7550010c-6165-4d5f-90bf-52872faa8a8c" containerName="revoke-keys" Oct 30 13:12:27.236851 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.236725 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="62a8727a-ad11-4c04-87e1-0b198af32183" containerName="payload-pre-processing" Oct 30 13:12:27.236851 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.236732 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="82af0f22-cb2d-41c5-b4a8-adcecc306998" containerName="payload-processing" Oct 30 13:12:27.236851 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.236739 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0132701-3b77-4cce-9989-a0d03ac479f3" containerName="istio-proxy" Oct 30 13:12:27.240207 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.240189 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-54994c-7mkl6" Oct 30 13:12:27.256761 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.256728 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:12:27.257414 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.257083 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:12:27.257414 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.257119 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:12:27.257414 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.257181 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:12:27.261678 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.261010 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-54994c-7mkl6"] Oct 30 13:12:27.274153 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.274130 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" Oct 30 13:12:27.373283 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.373250 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0132701-3b77-4cce-9989-a0d03ac479f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0132701-3b77-4cce-9989-a0d03ac479f3-credential-socket\") pod \"d0132701-3b77-4cce-9989-a0d03ac479f3\" (UID: \"d0132701-3b77-4cce-9989-a0d03ac479f3\") " Oct 30 13:12:27.373473 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.373311 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d034a1b-2482-4f51-b89c-6300d660b7ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d034a1b-2482-4f51-b89c-6300d660b7ee-workload-certs\") pod \"7d034a1b-2482-4f51-b89c-6300d660b7ee\" (UID: \"7d034a1b-2482-4f51-b89c-6300d660b7ee\") " Oct 30 13:12:27.373473 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.373372 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d034a1b-2482-4f51-b89c-6300d660b7ee-kube-api-access-skrnn\" (UniqueName: \"kubernetes.io/projected/7d034a1b-2482-4f51-b89c-6300d660b7ee-kube-api-access-skrnn\") pod \"7d034a1b-2482-4f51-b89c-6300d660b7ee\" (UID: \"7d034a1b-2482-4f51-b89c-6300d660b7ee\") " Oct 30 13:12:27.373473 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.373417 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0132701-3b77-4cce-9989-a0d03ac479f3-kube-api-access-g5fs7\" (UniqueName: \"kubernetes.io/projected/d0132701-3b77-4cce-9989-a0d03ac479f3-kube-api-access-g5fs7\") pod \"d0132701-3b77-4cce-9989-a0d03ac479f3\" (UID: \"d0132701-3b77-4cce-9989-a0d03ac479f3\") " Oct 30 13:12:27.373664 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.373487 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0132701-3b77-4cce-9989-a0d03ac479f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0132701-3b77-4cce-9989-a0d03ac479f3-istio-envoy\") pod \"d0132701-3b77-4cce-9989-a0d03ac479f3\" (UID: \"d0132701-3b77-4cce-9989-a0d03ac479f3\") " Oct 30 13:12:27.373664 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.373534 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0132701-3b77-4cce-9989-a0d03ac479f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0132701-3b77-4cce-9989-a0d03ac479f3-istio-data\") pod \"d0132701-3b77-4cce-9989-a0d03ac479f3\" (UID: \"d0132701-3b77-4cce-9989-a0d03ac479f3\") " Oct 30 13:12:27.373664 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.373542 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0132701-3b77-4cce-9989-a0d03ac479f3-credential-socket" pod "d0132701-3b77-4cce-9989-a0d03ac479f3" (UID: "d0132701-3b77-4cce-9989-a0d03ac479f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:27.373664 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.373575 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d034a1b-2482-4f51-b89c-6300d660b7ee-istio-token\" (UniqueName: \"kubernetes.io/projected/7d034a1b-2482-4f51-b89c-6300d660b7ee-istio-token\") pod \"7d034a1b-2482-4f51-b89c-6300d660b7ee\" (UID: \"7d034a1b-2482-4f51-b89c-6300d660b7ee\") " Oct 30 13:12:27.373890 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.373671 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0132701-3b77-4cce-9989-a0d03ac479f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0132701-3b77-4cce-9989-a0d03ac479f3-workload-certs\") pod \"d0132701-3b77-4cce-9989-a0d03ac479f3\" (UID: \"d0132701-3b77-4cce-9989-a0d03ac479f3\") " Oct 30 13:12:27.373890 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.373703 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d034a1b-2482-4f51-b89c-6300d660b7ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d034a1b-2482-4f51-b89c-6300d660b7ee-workload-socket\") pod \"7d034a1b-2482-4f51-b89c-6300d660b7ee\" (UID: \"7d034a1b-2482-4f51-b89c-6300d660b7ee\") " Oct 30 13:12:27.373890 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.373732 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0132701-3b77-4cce-9989-a0d03ac479f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0132701-3b77-4cce-9989-a0d03ac479f3-istiod-ca-cert\") pod \"d0132701-3b77-4cce-9989-a0d03ac479f3\" (UID: \"d0132701-3b77-4cce-9989-a0d03ac479f3\") " Oct 30 13:12:27.373890 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.373776 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d034a1b-2482-4f51-b89c-6300d660b7ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d034a1b-2482-4f51-b89c-6300d660b7ee-istio-podinfo\") pod \"7d034a1b-2482-4f51-b89c-6300d660b7ee\" (UID: \"7d034a1b-2482-4f51-b89c-6300d660b7ee\") " Oct 30 13:12:27.373890 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.373833 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d034a1b-2482-4f51-b89c-6300d660b7ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d034a1b-2482-4f51-b89c-6300d660b7ee-istio-envoy\") pod \"7d034a1b-2482-4f51-b89c-6300d660b7ee\" (UID: \"7d034a1b-2482-4f51-b89c-6300d660b7ee\") " Oct 30 13:12:27.373890 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.373869 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0132701-3b77-4cce-9989-a0d03ac479f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0132701-3b77-4cce-9989-a0d03ac479f3-workload-socket\") pod \"d0132701-3b77-4cce-9989-a0d03ac479f3\" (UID: \"d0132701-3b77-4cce-9989-a0d03ac479f3\") " Oct 30 13:12:27.374186 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.373910 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0132701-3b77-4cce-9989-a0d03ac479f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0132701-3b77-4cce-9989-a0d03ac479f3-istio-podinfo\") pod \"d0132701-3b77-4cce-9989-a0d03ac479f3\" (UID: \"d0132701-3b77-4cce-9989-a0d03ac479f3\") " Oct 30 13:12:27.374186 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.373935 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d034a1b-2482-4f51-b89c-6300d660b7ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d034a1b-2482-4f51-b89c-6300d660b7ee-istio-data\") pod \"7d034a1b-2482-4f51-b89c-6300d660b7ee\" (UID: \"7d034a1b-2482-4f51-b89c-6300d660b7ee\") " Oct 30 13:12:27.374186 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.373970 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0132701-3b77-4cce-9989-a0d03ac479f3-istio-token\" (UniqueName: \"kubernetes.io/projected/d0132701-3b77-4cce-9989-a0d03ac479f3-istio-token\") pod \"d0132701-3b77-4cce-9989-a0d03ac479f3\" (UID: \"d0132701-3b77-4cce-9989-a0d03ac479f3\") " Oct 30 13:12:27.374186 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.374020 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d034a1b-2482-4f51-b89c-6300d660b7ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d034a1b-2482-4f51-b89c-6300d660b7ee-istiod-ca-cert\") pod \"7d034a1b-2482-4f51-b89c-6300d660b7ee\" (UID: \"7d034a1b-2482-4f51-b89c-6300d660b7ee\") " Oct 30 13:12:27.374186 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.374094 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d034a1b-2482-4f51-b89c-6300d660b7ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d034a1b-2482-4f51-b89c-6300d660b7ee-credential-socket\") pod \"7d034a1b-2482-4f51-b89c-6300d660b7ee\" (UID: \"7d034a1b-2482-4f51-b89c-6300d660b7ee\") " Oct 30 13:12:27.374430 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.374330 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d034a1b-2482-4f51-b89c-6300d660b7ee-workload-certs" pod "7d034a1b-2482-4f51-b89c-6300d660b7ee" (UID: "7d034a1b-2482-4f51-b89c-6300d660b7ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:27.374430 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.374417 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/856abf62-76be-4153-ac58-97236322f107-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-54994c-7mkl6\" (UID: \"856abf62-76be-4153-ac58-97236322f107\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-54994c-7mkl6" Oct 30 13:12:27.374542 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.374430 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0132701-3b77-4cce-9989-a0d03ac479f3-workload-certs" pod "d0132701-3b77-4cce-9989-a0d03ac479f3" (UID: "d0132701-3b77-4cce-9989-a0d03ac479f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:27.374542 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.374484 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d034a1b-2482-4f51-b89c-6300d660b7ee-workload-socket" pod "7d034a1b-2482-4f51-b89c-6300d660b7ee" (UID: "7d034a1b-2482-4f51-b89c-6300d660b7ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:27.374542 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.374527 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d034a1b-2482-4f51-b89c-6300d660b7ee-credential-socket" pod "7d034a1b-2482-4f51-b89c-6300d660b7ee" (UID: "7d034a1b-2482-4f51-b89c-6300d660b7ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:27.374800 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.374774 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0132701-3b77-4cce-9989-a0d03ac479f3-workload-socket" pod "d0132701-3b77-4cce-9989-a0d03ac479f3" (UID: "d0132701-3b77-4cce-9989-a0d03ac479f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:27.374889 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.374861 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/856abf62-76be-4153-ac58-97236322f107-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-54994c-7mkl6\" (UID: \"856abf62-76be-4153-ac58-97236322f107\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-54994c-7mkl6" Oct 30 13:12:27.374956 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.374931 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv8qh\" (UniqueName: \"kubernetes.io/projected/856abf62-76be-4153-ac58-97236322f107-kube-api-access-vv8qh\") pod \"maas-api-revoke-keys-e2e-worker-w4-54994c-7mkl6\" (UID: \"856abf62-76be-4153-ac58-97236322f107\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-54994c-7mkl6" Oct 30 13:12:27.375014 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.374955 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d034a1b-2482-4f51-b89c-6300d660b7ee-istio-data" pod "7d034a1b-2482-4f51-b89c-6300d660b7ee" (UID: "7d034a1b-2482-4f51-b89c-6300d660b7ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:27.375148 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.375130 5922 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0132701-3b77-4cce-9989-a0d03ac479f3-workload-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.375213 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.375152 5922 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d034a1b-2482-4f51-b89c-6300d660b7ee-istio-data\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.375213 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.375167 5922 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d034a1b-2482-4f51-b89c-6300d660b7ee-credential-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.375213 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.375182 5922 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0132701-3b77-4cce-9989-a0d03ac479f3-credential-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.375213 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.375196 5922 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d034a1b-2482-4f51-b89c-6300d660b7ee-workload-certs\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.375213 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.375210 5922 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0132701-3b77-4cce-9989-a0d03ac479f3-workload-certs\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.376291 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.375222 5922 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d034a1b-2482-4f51-b89c-6300d660b7ee-workload-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.376291 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.375721 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d034a1b-2482-4f51-b89c-6300d660b7ee-istiod-ca-cert" pod "7d034a1b-2482-4f51-b89c-6300d660b7ee" (UID: "7d034a1b-2482-4f51-b89c-6300d660b7ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:12:27.376291 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.376019 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0132701-3b77-4cce-9989-a0d03ac479f3-istiod-ca-cert" pod "d0132701-3b77-4cce-9989-a0d03ac479f3" (UID: "d0132701-3b77-4cce-9989-a0d03ac479f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:12:27.376587 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.376306 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0132701-3b77-4cce-9989-a0d03ac479f3-istio-data" pod "d0132701-3b77-4cce-9989-a0d03ac479f3" (UID: "d0132701-3b77-4cce-9989-a0d03ac479f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:27.378199 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.378162 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0132701-3b77-4cce-9989-a0d03ac479f3-istio-token" pod "d0132701-3b77-4cce-9989-a0d03ac479f3" (UID: "d0132701-3b77-4cce-9989-a0d03ac479f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:27.379501 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.379474 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0132701-3b77-4cce-9989-a0d03ac479f3-istio-envoy" pod "d0132701-3b77-4cce-9989-a0d03ac479f3" (UID: "d0132701-3b77-4cce-9989-a0d03ac479f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:27.380034 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.379986 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0132701-3b77-4cce-9989-a0d03ac479f3-istio-podinfo" pod "d0132701-3b77-4cce-9989-a0d03ac479f3" (UID: "d0132701-3b77-4cce-9989-a0d03ac479f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:12:27.380034 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.379989 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d034a1b-2482-4f51-b89c-6300d660b7ee-istio-token" pod "7d034a1b-2482-4f51-b89c-6300d660b7ee" (UID: "7d034a1b-2482-4f51-b89c-6300d660b7ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:27.380034 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.380012 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d034a1b-2482-4f51-b89c-6300d660b7ee-kube-api-access-skrnn" pod "7d034a1b-2482-4f51-b89c-6300d660b7ee" (UID: "7d034a1b-2482-4f51-b89c-6300d660b7ee"). InnerVolumeSpecName "kube-api-access-skrnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:27.380245 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.380109 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0132701-3b77-4cce-9989-a0d03ac479f3-kube-api-access-g5fs7" pod "d0132701-3b77-4cce-9989-a0d03ac479f3" (UID: "d0132701-3b77-4cce-9989-a0d03ac479f3"). InnerVolumeSpecName "kube-api-access-g5fs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:27.380874 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.380848 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d034a1b-2482-4f51-b89c-6300d660b7ee-istio-podinfo" pod "7d034a1b-2482-4f51-b89c-6300d660b7ee" (UID: "7d034a1b-2482-4f51-b89c-6300d660b7ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:12:27.381204 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.381183 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d034a1b-2482-4f51-b89c-6300d660b7ee-istio-envoy" pod "7d034a1b-2482-4f51-b89c-6300d660b7ee" (UID: "7d034a1b-2482-4f51-b89c-6300d660b7ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:27.476821 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.476781 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/856abf62-76be-4153-ac58-97236322f107-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-54994c-7mkl6\" (UID: \"856abf62-76be-4153-ac58-97236322f107\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-54994c-7mkl6" Oct 30 13:12:27.477016 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.476982 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/856abf62-76be-4153-ac58-97236322f107-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-54994c-7mkl6\" (UID: \"856abf62-76be-4153-ac58-97236322f107\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-54994c-7mkl6" Oct 30 13:12:27.477092 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.477017 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv8qh\" (UniqueName: \"kubernetes.io/projected/856abf62-76be-4153-ac58-97236322f107-kube-api-access-vv8qh\") pod \"maas-api-revoke-keys-e2e-worker-w4-54994c-7mkl6\" (UID: \"856abf62-76be-4153-ac58-97236322f107\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-54994c-7mkl6" Oct 30 13:12:27.477188 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.477171 5922 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0132701-3b77-4cce-9989-a0d03ac479f3-istio-envoy\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.477274 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.477193 5922 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0132701-3b77-4cce-9989-a0d03ac479f3-istio-data\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.477274 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.477206 5922 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d034a1b-2482-4f51-b89c-6300d660b7ee-istio-token\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.477274 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.477222 5922 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0132701-3b77-4cce-9989-a0d03ac479f3-istiod-ca-cert\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.477274 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.477234 5922 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d034a1b-2482-4f51-b89c-6300d660b7ee-istio-podinfo\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.477274 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.477246 5922 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d034a1b-2482-4f51-b89c-6300d660b7ee-istio-envoy\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.477274 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.477260 5922 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0132701-3b77-4cce-9989-a0d03ac479f3-istio-podinfo\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.477274 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.477272 5922 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0132701-3b77-4cce-9989-a0d03ac479f3-istio-token\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.477721 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.477284 5922 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d034a1b-2482-4f51-b89c-6300d660b7ee-istiod-ca-cert\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.477721 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.477297 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skrnn\" (UniqueName: \"kubernetes.io/projected/7d034a1b-2482-4f51-b89c-6300d660b7ee-kube-api-access-skrnn\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.477721 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.477310 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5fs7\" (UniqueName: \"kubernetes.io/projected/d0132701-3b77-4cce-9989-a0d03ac479f3-kube-api-access-g5fs7\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.477721 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.477637 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/856abf62-76be-4153-ac58-97236322f107-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-54994c-7mkl6\" (UID: \"856abf62-76be-4153-ac58-97236322f107\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-54994c-7mkl6" Oct 30 13:12:27.480863 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.480835 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/856abf62-76be-4153-ac58-97236322f107-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-54994c-7mkl6\" (UID: \"856abf62-76be-4153-ac58-97236322f107\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-54994c-7mkl6" Oct 30 13:12:27.502845 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.502816 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv8qh\" (UniqueName: \"kubernetes.io/projected/856abf62-76be-4153-ac58-97236322f107-kube-api-access-vv8qh\") pod \"maas-api-revoke-keys-e2e-worker-w4-54994c-7mkl6\" (UID: \"856abf62-76be-4153-ac58-97236322f107\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-54994c-7mkl6" Oct 30 13:12:27.533094 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.533064 5922 generic.go:356] "Generic (PLEG): container finished" podID="d62b6ccd-e8a1-4320-87d0-0b12377ad8ee" containerID="f169a2618499b3b38b132015ed8e2245e35c526f25ddea2745e124a56074a928" exitCode=0 Oct 30 13:12:27.533243 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.533193 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" event={"ID":"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee","Type":"ContainerDied","Data":"f169a2618499b3b38b132015ed8e2245e35c526f25ddea2745e124a56074a928"} Oct 30 13:12:27.533243 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.533241 5922 scope.go:122] "RemoveContainer" containerID="5af4f5e7c7ee7cea6851d1ac6622767bda512d8d889d9f19d691bf996d53c70f" Oct 30 13:12:27.536068 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.536042 5922 generic.go:356] "Generic (PLEG): container finished" podID="7d034a1b-2482-4f51-b89c-6300d660b7ee" containerID="3bc9462e86526c0163c87edb7331ffec30154f643fe4ba31aebcfad8f7d9a8cf" exitCode=0 Oct 30 13:12:27.536179 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.536136 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" event={"ID":"7d034a1b-2482-4f51-b89c-6300d660b7ee","Type":"ContainerDied","Data":"3bc9462e86526c0163c87edb7331ffec30154f643fe4ba31aebcfad8f7d9a8cf"} Oct 30 13:12:27.536245 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.536179 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" event={"ID":"7d034a1b-2482-4f51-b89c-6300d660b7ee","Type":"ContainerDied","Data":"15ba2a9876f623fe541b1a2e7f02c11138ccc805e70bc631838e1831147cb608"} Oct 30 13:12:27.536308 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.536254 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m" Oct 30 13:12:27.544950 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.544922 5922 generic.go:356] "Generic (PLEG): container finished" podID="d0132701-3b77-4cce-9989-a0d03ac479f3" containerID="2ead0939b95900ae542eba74cecb73cc7b39574b7015929f9fef7f6bc367af75" exitCode=0 Oct 30 13:12:27.545059 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.544988 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" Oct 30 13:12:27.545189 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.545160 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" event={"ID":"d0132701-3b77-4cce-9989-a0d03ac479f3","Type":"ContainerDied","Data":"2ead0939b95900ae542eba74cecb73cc7b39574b7015929f9fef7f6bc367af75"} Oct 30 13:12:27.545311 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.545214 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks" event={"ID":"d0132701-3b77-4cce-9989-a0d03ac479f3","Type":"ContainerDied","Data":"f7c1e374ece95af5c4b408676e8ef3367cc3fbffedc52e1933d5f5255fc58a75"} Oct 30 13:12:27.551521 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.551499 5922 scope.go:122] "RemoveContainer" containerID="3bc9462e86526c0163c87edb7331ffec30154f643fe4ba31aebcfad8f7d9a8cf" Oct 30 13:12:27.562431 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.562409 5922 scope.go:122] "RemoveContainer" containerID="3bc9462e86526c0163c87edb7331ffec30154f643fe4ba31aebcfad8f7d9a8cf" Oct 30 13:12:27.562839 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:12:27.562805 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bc9462e86526c0163c87edb7331ffec30154f643fe4ba31aebcfad8f7d9a8cf\": container with ID starting with 3bc9462e86526c0163c87edb7331ffec30154f643fe4ba31aebcfad8f7d9a8cf not found: ID does not exist" containerID="3bc9462e86526c0163c87edb7331ffec30154f643fe4ba31aebcfad8f7d9a8cf" Oct 30 13:12:27.562935 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.562847 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bc9462e86526c0163c87edb7331ffec30154f643fe4ba31aebcfad8f7d9a8cf"} err="failed to get container status \"3bc9462e86526c0163c87edb7331ffec30154f643fe4ba31aebcfad8f7d9a8cf\": rpc error: code = NotFound desc = could not find container \"3bc9462e86526c0163c87edb7331ffec30154f643fe4ba31aebcfad8f7d9a8cf\": container with ID starting with 3bc9462e86526c0163c87edb7331ffec30154f643fe4ba31aebcfad8f7d9a8cf not found: ID does not exist" Oct 30 13:12:27.562935 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.562870 5922 scope.go:122] "RemoveContainer" containerID="2ead0939b95900ae542eba74cecb73cc7b39574b7015929f9fef7f6bc367af75" Oct 30 13:12:27.571868 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.571836 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-54994c-7mkl6" Oct 30 13:12:27.577109 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.577091 5922 scope.go:122] "RemoveContainer" containerID="2ead0939b95900ae542eba74cecb73cc7b39574b7015929f9fef7f6bc367af75" Oct 30 13:12:27.577418 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:12:27.577389 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ead0939b95900ae542eba74cecb73cc7b39574b7015929f9fef7f6bc367af75\": container with ID starting with 2ead0939b95900ae542eba74cecb73cc7b39574b7015929f9fef7f6bc367af75 not found: ID does not exist" containerID="2ead0939b95900ae542eba74cecb73cc7b39574b7015929f9fef7f6bc367af75" Oct 30 13:12:27.577488 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.577428 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ead0939b95900ae542eba74cecb73cc7b39574b7015929f9fef7f6bc367af75"} err="failed to get container status \"2ead0939b95900ae542eba74cecb73cc7b39574b7015929f9fef7f6bc367af75\": rpc error: code = NotFound desc = could not find container \"2ead0939b95900ae542eba74cecb73cc7b39574b7015929f9fef7f6bc367af75\": container with ID starting with 2ead0939b95900ae542eba74cecb73cc7b39574b7015929f9fef7f6bc367af75 not found: ID does not exist" Oct 30 13:12:27.599409 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.599387 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" Oct 30 13:12:27.605249 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.605223 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m"] Oct 30 13:12:27.610987 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.610964 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-kcv9m"] Oct 30 13:12:27.655534 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.655502 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d034a1b-2482-4f51-b89c-6300d660b7ee" path="/var/lib/kubelet/pods/7d034a1b-2482-4f51-b89c-6300d660b7ee/volumes" Oct 30 13:12:27.662309 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.662286 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks"] Oct 30 13:12:27.662426 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.662327 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-xw5ks"] Oct 30 13:12:27.679546 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.679505 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-istiod-ca-cert\") pod \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\" (UID: \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\") " Oct 30 13:12:27.679696 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.679555 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-credential-socket\") pod \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\" (UID: \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\") " Oct 30 13:12:27.679696 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.679635 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-istio-envoy\") pod \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\" (UID: \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\") " Oct 30 13:12:27.679696 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.679678 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-istio-data\") pod \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\" (UID: \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\") " Oct 30 13:12:27.679858 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.679748 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-istio-token\" (UniqueName: \"kubernetes.io/projected/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-istio-token\") pod \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\" (UID: \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\") " Oct 30 13:12:27.679858 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.679839 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-workload-socket\") pod \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\" (UID: \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\") " Oct 30 13:12:27.679957 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.679933 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-kube-api-access-78zlh\" (UniqueName: \"kubernetes.io/projected/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-kube-api-access-78zlh\") pod \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\" (UID: \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\") " Oct 30 13:12:27.680017 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.679994 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-istio-podinfo\") pod \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\" (UID: \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\") " Oct 30 13:12:27.680123 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.680104 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-workload-certs\") pod \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\" (UID: \"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee\") " Oct 30 13:12:27.681061 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.680865 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-workload-certs" pod "d62b6ccd-e8a1-4320-87d0-0b12377ad8ee" (UID: "d62b6ccd-e8a1-4320-87d0-0b12377ad8ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:27.681061 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.680916 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-credential-socket" pod "d62b6ccd-e8a1-4320-87d0-0b12377ad8ee" (UID: "d62b6ccd-e8a1-4320-87d0-0b12377ad8ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:27.681247 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.681126 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-istio-data" pod "d62b6ccd-e8a1-4320-87d0-0b12377ad8ee" (UID: "d62b6ccd-e8a1-4320-87d0-0b12377ad8ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:27.681396 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.681359 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-workload-socket" pod "d62b6ccd-e8a1-4320-87d0-0b12377ad8ee" (UID: "d62b6ccd-e8a1-4320-87d0-0b12377ad8ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:27.681706 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.681687 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-istiod-ca-cert" pod "d62b6ccd-e8a1-4320-87d0-0b12377ad8ee" (UID: "d62b6ccd-e8a1-4320-87d0-0b12377ad8ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:12:27.685029 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.684934 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-istio-podinfo" pod "d62b6ccd-e8a1-4320-87d0-0b12377ad8ee" (UID: "d62b6ccd-e8a1-4320-87d0-0b12377ad8ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:12:27.685029 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.684953 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-istio-token" pod "d62b6ccd-e8a1-4320-87d0-0b12377ad8ee" (UID: "d62b6ccd-e8a1-4320-87d0-0b12377ad8ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:27.685029 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.685003 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-kube-api-access-78zlh" pod "d62b6ccd-e8a1-4320-87d0-0b12377ad8ee" (UID: "d62b6ccd-e8a1-4320-87d0-0b12377ad8ee"). InnerVolumeSpecName "kube-api-access-78zlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:27.686214 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.686194 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-istio-envoy" pod "d62b6ccd-e8a1-4320-87d0-0b12377ad8ee" (UID: "d62b6ccd-e8a1-4320-87d0-0b12377ad8ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:27.781487 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.781414 5922 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-istio-token\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.781487 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.781444 5922 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-workload-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.781487 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.781453 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78zlh\" (UniqueName: \"kubernetes.io/projected/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-kube-api-access-78zlh\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.781487 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.781462 5922 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-istio-podinfo\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.781487 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.781470 5922 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-workload-certs\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.781487 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.781479 5922 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-istiod-ca-cert\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.781487 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.781486 5922 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-credential-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.782060 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.781494 5922 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-istio-envoy\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.782060 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.781502 5922 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee-istio-data\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.892357 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:27.892321 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-54994c-7mkl6"] Oct 30 13:12:28.045866 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:28.045800 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-530952d9-qjk4g"] Oct 30 13:12:28.057399 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:28.057371 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-530952d9-qjk4g"] Oct 30 13:12:28.557269 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:28.557229 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-54994c-7mkl6" event={"ID":"856abf62-76be-4153-ac58-97236322f107","Type":"ContainerStarted","Data":"66bd5a442ae52e7c5cdfa22e310fde3ed76b3bf90a2abc7eb57c59c8f0c363f5"} Oct 30 13:12:28.557426 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:28.557292 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-54994c-7mkl6" event={"ID":"856abf62-76be-4153-ac58-97236322f107","Type":"ContainerStarted","Data":"89839936ac82a37314eb9387fd014c4efb72222b84b1dadf80f71b79061833b3"} Oct 30 13:12:28.559678 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:28.559649 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" event={"ID":"d62b6ccd-e8a1-4320-87d0-0b12377ad8ee","Type":"ContainerDied","Data":"57a86d45e02c20e3126d7de12b1bce5bf2d0c67c254ba69c2d540ff75e68af83"} Oct 30 13:12:28.559678 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:28.559674 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47" Oct 30 13:12:28.559867 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:28.559701 5922 scope.go:122] "RemoveContainer" containerID="f169a2618499b3b38b132015ed8e2245e35c526f25ddea2745e124a56074a928" Oct 30 13:12:28.601719 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:28.601673 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-54994c-7mkl6" podStartSLOduration=1.60165914 podStartE2EDuration="1.60165914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:28.598802854 +0000 UTC m=+1205.068218618" watchObservedRunningTime="2026-10-30 13:12:28.60165914 +0000 UTC m=+1205.071074923" Oct 30 13:12:28.640628 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:28.640585 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47"] Oct 30 13:12:28.645760 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:28.645736 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-pdf47"] Oct 30 13:12:28.963551 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:28.963517 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:28.993827 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:28.993798 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-ac0c58-7df9968d48-46vcj"] Oct 30 13:12:29.153410 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:29.153375 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:29.275158 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:29.275071 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:29.570523 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:29.570432 5922 generic.go:356] "Generic (PLEG): container finished" podID="856abf62-76be-4153-ac58-97236322f107" containerID="66bd5a442ae52e7c5cdfa22e310fde3ed76b3bf90a2abc7eb57c59c8f0c363f5" exitCode=0 Oct 30 13:12:29.570523 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:29.570472 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-54994c-7mkl6" event={"ID":"856abf62-76be-4153-ac58-97236322f107","Type":"ContainerDied","Data":"66bd5a442ae52e7c5cdfa22e310fde3ed76b3bf90a2abc7eb57c59c8f0c363f5"} Oct 30 13:12:29.645353 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:29.645325 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0132701-3b77-4cce-9989-a0d03ac479f3" path="/var/lib/kubelet/pods/d0132701-3b77-4cce-9989-a0d03ac479f3/volumes" Oct 30 13:12:29.645840 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:29.645813 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d62b6ccd-e8a1-4320-87d0-0b12377ad8ee" path="/var/lib/kubelet/pods/d62b6ccd-e8a1-4320-87d0-0b12377ad8ee/volumes" Oct 30 13:12:29.646229 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:29.646215 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddd94e09-f766-4de6-b4f5-0a2fc927c3db" path="/var/lib/kubelet/pods/ddd94e09-f766-4de6-b4f5-0a2fc927c3db/volumes" Oct 30 13:12:30.721852 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:30.721822 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-54994c-7mkl6" Oct 30 13:12:30.821910 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:30.821875 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/856abf62-76be-4153-ac58-97236322f107-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/856abf62-76be-4153-ac58-97236322f107-maas-api-xks-ca\") pod \"856abf62-76be-4153-ac58-97236322f107\" (UID: \"856abf62-76be-4153-ac58-97236322f107\") " Oct 30 13:12:30.822068 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:30.821918 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/856abf62-76be-4153-ac58-97236322f107-kube-api-access-vv8qh\" (UniqueName: \"kubernetes.io/projected/856abf62-76be-4153-ac58-97236322f107-kube-api-access-vv8qh\") pod \"856abf62-76be-4153-ac58-97236322f107\" (UID: \"856abf62-76be-4153-ac58-97236322f107\") " Oct 30 13:12:30.822068 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:30.821960 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/856abf62-76be-4153-ac58-97236322f107-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/856abf62-76be-4153-ac58-97236322f107-maas-api-openshift-ca\") pod \"856abf62-76be-4153-ac58-97236322f107\" (UID: \"856abf62-76be-4153-ac58-97236322f107\") " Oct 30 13:12:30.822337 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:30.822314 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/856abf62-76be-4153-ac58-97236322f107-maas-api-openshift-ca" pod "856abf62-76be-4153-ac58-97236322f107" (UID: "856abf62-76be-4153-ac58-97236322f107"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:12:30.822649 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:30.822597 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/856abf62-76be-4153-ac58-97236322f107-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:30.824219 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:30.824197 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/856abf62-76be-4153-ac58-97236322f107-maas-api-xks-ca" pod "856abf62-76be-4153-ac58-97236322f107" (UID: "856abf62-76be-4153-ac58-97236322f107"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:12:30.824329 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:30.824200 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/856abf62-76be-4153-ac58-97236322f107-kube-api-access-vv8qh" pod "856abf62-76be-4153-ac58-97236322f107" (UID: "856abf62-76be-4153-ac58-97236322f107"). InnerVolumeSpecName "kube-api-access-vv8qh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:30.924080 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:30.924047 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/856abf62-76be-4153-ac58-97236322f107-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:30.924213 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:30.924083 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv8qh\" (UniqueName: \"kubernetes.io/projected/856abf62-76be-4153-ac58-97236322f107-kube-api-access-vv8qh\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:30.934593 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:30.934552 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:31.058126 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:31.058083 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:31.192715 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:31.191514 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:31.492635 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:31.492529 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:31.588480 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:31.588449 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-54994c-7mkl6" Oct 30 13:12:31.588480 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:31.588463 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-54994c-7mkl6" event={"ID":"856abf62-76be-4153-ac58-97236322f107","Type":"ContainerDied","Data":"89839936ac82a37314eb9387fd014c4efb72222b84b1dadf80f71b79061833b3"} Oct 30 13:12:31.588707 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:31.588504 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89839936ac82a37314eb9387fd014c4efb72222b84b1dadf80f71b79061833b3" Oct 30 13:12:31.672251 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:31.672213 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:32.038681 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.038644 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86"] Oct 30 13:12:32.050671 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.050643 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr"] Oct 30 13:12:32.090021 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.089991 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:32.227231 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.227191 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:32.668146 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.668106 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b817f58-b2kvn"] Oct 30 13:12:32.668805 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.668788 5922 cpu_manager.go:414] "containerMap: removing container" podUID="7d034a1b-2482-4f51-b89c-6300d660b7ee" containerName="istio-proxy" Oct 30 13:12:32.668857 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.668807 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d034a1b-2482-4f51-b89c-6300d660b7ee" containerName="istio-proxy" Oct 30 13:12:32.668857 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.668821 5922 cpu_manager.go:414] "containerMap: removing container" podUID="856abf62-76be-4153-ac58-97236322f107" containerName="revoke-keys" Oct 30 13:12:32.668857 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.668826 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="856abf62-76be-4153-ac58-97236322f107" containerName="revoke-keys" Oct 30 13:12:32.668857 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.668836 5922 cpu_manager.go:414] "containerMap: removing container" podUID="d62b6ccd-e8a1-4320-87d0-0b12377ad8ee" containerName="istio-proxy" Oct 30 13:12:32.668857 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.668842 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d62b6ccd-e8a1-4320-87d0-0b12377ad8ee" containerName="istio-proxy" Oct 30 13:12:32.669023 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.668933 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d034a1b-2482-4f51-b89c-6300d660b7ee" containerName="istio-proxy" Oct 30 13:12:32.669023 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.668943 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="856abf62-76be-4153-ac58-97236322f107" containerName="revoke-keys" Oct 30 13:12:32.669023 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.668950 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="d62b6ccd-e8a1-4320-87d0-0b12377ad8ee" containerName="istio-proxy" Oct 30 13:12:32.669023 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.668956 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="d62b6ccd-e8a1-4320-87d0-0b12377ad8ee" containerName="istio-proxy" Oct 30 13:12:32.672738 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.672721 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b817f58-b2kvn" Oct 30 13:12:32.706810 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.706763 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:12:32.706810 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.706797 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:12:32.706966 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.706836 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:12:32.706966 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.706919 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:12:32.717270 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.717248 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b817f58-b2kvn"] Oct 30 13:12:32.722314 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.722291 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9876320b-9tg8r"] Oct 30 13:12:32.723131 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.723110 5922 cpu_manager.go:414] "containerMap: removing container" podUID="d62b6ccd-e8a1-4320-87d0-0b12377ad8ee" containerName="istio-proxy" Oct 30 13:12:32.723208 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.723133 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d62b6ccd-e8a1-4320-87d0-0b12377ad8ee" containerName="istio-proxy" Oct 30 13:12:32.727269 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.727249 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9876320b-9tg8r" Oct 30 13:12:32.758945 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.758922 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9876320b-9tg8r"] Oct 30 13:12:32.847191 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.847163 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2833e9b-a45f-4557-b5c9-075474e38f3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b817f58-b2kvn\" (UID: \"a2833e9b-a45f-4557-b5c9-075474e38f3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b817f58-b2kvn" Oct 30 13:12:32.847340 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.847286 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2833e9b-a45f-4557-b5c9-075474e38f3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b817f58-b2kvn\" (UID: \"a2833e9b-a45f-4557-b5c9-075474e38f3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b817f58-b2kvn" Oct 30 13:12:32.847340 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.847336 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b0d2d00-e66c-410a-8383-04a6c34620fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9876320b-9tg8r\" (UID: \"8b0d2d00-e66c-410a-8383-04a6c34620fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9876320b-9tg8r" Oct 30 13:12:32.847430 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.847370 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2h45\" (UniqueName: \"kubernetes.io/projected/a2833e9b-a45f-4557-b5c9-075474e38f3f-kube-api-access-t2h45\") pod \"maas-api-revoke-sub-6b817f58-b2kvn\" (UID: \"a2833e9b-a45f-4557-b5c9-075474e38f3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b817f58-b2kvn" Oct 30 13:12:32.847469 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.847436 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b0d2d00-e66c-410a-8383-04a6c34620fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9876320b-9tg8r\" (UID: \"8b0d2d00-e66c-410a-8383-04a6c34620fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9876320b-9tg8r" Oct 30 13:12:32.847503 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.847472 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjlps\" (UniqueName: \"kubernetes.io/projected/8b0d2d00-e66c-410a-8383-04a6c34620fd-kube-api-access-tjlps\") pod \"maas-api-revoke-sub-9876320b-9tg8r\" (UID: \"8b0d2d00-e66c-410a-8383-04a6c34620fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9876320b-9tg8r" Oct 30 13:12:32.948876 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.948852 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2833e9b-a45f-4557-b5c9-075474e38f3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b817f58-b2kvn\" (UID: \"a2833e9b-a45f-4557-b5c9-075474e38f3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b817f58-b2kvn" Oct 30 13:12:32.949026 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.948932 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2833e9b-a45f-4557-b5c9-075474e38f3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b817f58-b2kvn\" (UID: \"a2833e9b-a45f-4557-b5c9-075474e38f3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b817f58-b2kvn" Oct 30 13:12:32.949026 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.948990 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b0d2d00-e66c-410a-8383-04a6c34620fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9876320b-9tg8r\" (UID: \"8b0d2d00-e66c-410a-8383-04a6c34620fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9876320b-9tg8r" Oct 30 13:12:32.949145 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.949036 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2h45\" (UniqueName: \"kubernetes.io/projected/a2833e9b-a45f-4557-b5c9-075474e38f3f-kube-api-access-t2h45\") pod \"maas-api-revoke-sub-6b817f58-b2kvn\" (UID: \"a2833e9b-a45f-4557-b5c9-075474e38f3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b817f58-b2kvn" Oct 30 13:12:32.949145 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.949095 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b0d2d00-e66c-410a-8383-04a6c34620fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9876320b-9tg8r\" (UID: \"8b0d2d00-e66c-410a-8383-04a6c34620fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9876320b-9tg8r" Oct 30 13:12:32.949145 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.949136 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjlps\" (UniqueName: \"kubernetes.io/projected/8b0d2d00-e66c-410a-8383-04a6c34620fd-kube-api-access-tjlps\") pod \"maas-api-revoke-sub-9876320b-9tg8r\" (UID: \"8b0d2d00-e66c-410a-8383-04a6c34620fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9876320b-9tg8r" Oct 30 13:12:32.949722 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.949700 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2833e9b-a45f-4557-b5c9-075474e38f3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b817f58-b2kvn\" (UID: \"a2833e9b-a45f-4557-b5c9-075474e38f3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b817f58-b2kvn" Oct 30 13:12:32.949925 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.949899 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b0d2d00-e66c-410a-8383-04a6c34620fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9876320b-9tg8r\" (UID: \"8b0d2d00-e66c-410a-8383-04a6c34620fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9876320b-9tg8r" Oct 30 13:12:32.951343 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.951323 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b0d2d00-e66c-410a-8383-04a6c34620fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9876320b-9tg8r\" (UID: \"8b0d2d00-e66c-410a-8383-04a6c34620fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9876320b-9tg8r" Oct 30 13:12:32.951343 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.951332 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2833e9b-a45f-4557-b5c9-075474e38f3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b817f58-b2kvn\" (UID: \"a2833e9b-a45f-4557-b5c9-075474e38f3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b817f58-b2kvn" Oct 30 13:12:32.977151 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.977131 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjlps\" (UniqueName: \"kubernetes.io/projected/8b0d2d00-e66c-410a-8383-04a6c34620fd-kube-api-access-tjlps\") pod \"maas-api-revoke-sub-9876320b-9tg8r\" (UID: \"8b0d2d00-e66c-410a-8383-04a6c34620fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9876320b-9tg8r" Oct 30 13:12:32.979053 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.979034 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2h45\" (UniqueName: \"kubernetes.io/projected/a2833e9b-a45f-4557-b5c9-075474e38f3f-kube-api-access-t2h45\") pod \"maas-api-revoke-sub-6b817f58-b2kvn\" (UID: \"a2833e9b-a45f-4557-b5c9-075474e38f3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b817f58-b2kvn" Oct 30 13:12:32.986898 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:32.986874 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b817f58-b2kvn" Oct 30 13:12:33.044517 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:33.044488 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9876320b-9tg8r" Oct 30 13:12:33.048980 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:33.048925 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:33.292499 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:33.292469 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b817f58-b2kvn"] Oct 30 13:12:33.292864 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:12:33.292823 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2833e9b_a45f_4557_b5c9_075474e38f3f.slice/crio-51a19aa5d1fb6381ee7a225e1434d189771b83942b289e801e7aec61eaf20da7 WatchSource:0}: Error finding container 51a19aa5d1fb6381ee7a225e1434d189771b83942b289e801e7aec61eaf20da7: Status 404 returned error can't find the container with id 51a19aa5d1fb6381ee7a225e1434d189771b83942b289e801e7aec61eaf20da7 Oct 30 13:12:33.393579 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:33.393554 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9876320b-9tg8r"] Oct 30 13:12:33.394578 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:12:33.394551 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b0d2d00_e66c_410a_8383_04a6c34620fd.slice/crio-cd7e76b8b568ae0d4db9d8196920b53ad01bde3455697d7975e493d89dce843d WatchSource:0}: Error finding container cd7e76b8b568ae0d4db9d8196920b53ad01bde3455697d7975e493d89dce843d: Status 404 returned error can't find the container with id cd7e76b8b568ae0d4db9d8196920b53ad01bde3455697d7975e493d89dce843d Oct 30 13:12:33.502032 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:33.501989 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:33.604799 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:33.604769 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9876320b-9tg8r" event={"ID":"8b0d2d00-e66c-410a-8383-04a6c34620fd","Type":"ContainerStarted","Data":"cd7e76b8b568ae0d4db9d8196920b53ad01bde3455697d7975e493d89dce843d"} Oct 30 13:12:33.606505 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:33.606482 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b817f58-b2kvn" event={"ID":"a2833e9b-a45f-4557-b5c9-075474e38f3f","Type":"ContainerStarted","Data":"80b0b7ad6165b05d13b96a3dd18d10c8572298212df926ba9854f3e0221cfaef"} Oct 30 13:12:33.606641 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:33.606522 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b817f58-b2kvn" event={"ID":"a2833e9b-a45f-4557-b5c9-075474e38f3f","Type":"ContainerStarted","Data":"51a19aa5d1fb6381ee7a225e1434d189771b83942b289e801e7aec61eaf20da7"} Oct 30 13:12:33.647873 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:33.647829 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b817f58-b2kvn" podStartSLOduration=1.647813744 podStartE2EDuration="1.647813744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:33.647195642 +0000 UTC m=+1210.116611409" watchObservedRunningTime="2026-10-30 13:12:33.647813744 +0000 UTC m=+1210.117229559" Oct 30 13:12:33.994996 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:33.994957 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-ac0c58-7df9968d48-46vcj" podUID="b4fd49fb-f890-446a-9d22-435e43054ea5" containerName="payload-processing" containerID="cri-o://d74bfad8de0eeede6269e4d45d87d57b58039f528e0fd5837bb3ef20c8d3ec0e" gracePeriod=25 Oct 30 13:12:34.271156 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:34.271128 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-ac0c58-7df9968d48-46vcj" Oct 30 13:12:34.371380 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:34.371353 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4fd49fb-f890-446a-9d22-435e43054ea5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4fd49fb-f890-446a-9d22-435e43054ea5-plugins-config-volume\") pod \"b4fd49fb-f890-446a-9d22-435e43054ea5\" (UID: \"b4fd49fb-f890-446a-9d22-435e43054ea5\") " Oct 30 13:12:34.371494 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:34.371423 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4fd49fb-f890-446a-9d22-435e43054ea5-kube-api-access-84fqm\" (UniqueName: \"kubernetes.io/projected/b4fd49fb-f890-446a-9d22-435e43054ea5-kube-api-access-84fqm\") pod \"b4fd49fb-f890-446a-9d22-435e43054ea5\" (UID: \"b4fd49fb-f890-446a-9d22-435e43054ea5\") " Oct 30 13:12:34.371706 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:34.371680 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4fd49fb-f890-446a-9d22-435e43054ea5-plugins-config-volume" pod "b4fd49fb-f890-446a-9d22-435e43054ea5" (UID: "b4fd49fb-f890-446a-9d22-435e43054ea5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:12:34.371945 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:34.371925 5922 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4fd49fb-f890-446a-9d22-435e43054ea5-plugins-config-volume\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:34.373268 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:34.373247 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4fd49fb-f890-446a-9d22-435e43054ea5-kube-api-access-84fqm" pod "b4fd49fb-f890-446a-9d22-435e43054ea5" (UID: "b4fd49fb-f890-446a-9d22-435e43054ea5"). InnerVolumeSpecName "kube-api-access-84fqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:34.473191 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:34.473166 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84fqm\" (UniqueName: \"kubernetes.io/projected/b4fd49fb-f890-446a-9d22-435e43054ea5-kube-api-access-84fqm\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:34.616046 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:34.615970 5922 generic.go:356] "Generic (PLEG): container finished" podID="8b0d2d00-e66c-410a-8383-04a6c34620fd" containerID="22aa21c487fb95e41891bcad5aaa0f908a44037b9ff1168dfbfc524df1aa32e7" exitCode=0 Oct 30 13:12:34.616174 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:34.616058 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9876320b-9tg8r" event={"ID":"8b0d2d00-e66c-410a-8383-04a6c34620fd","Type":"ContainerDied","Data":"22aa21c487fb95e41891bcad5aaa0f908a44037b9ff1168dfbfc524df1aa32e7"} Oct 30 13:12:34.617737 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:34.617710 5922 generic.go:356] "Generic (PLEG): container finished" podID="a2833e9b-a45f-4557-b5c9-075474e38f3f" containerID="80b0b7ad6165b05d13b96a3dd18d10c8572298212df926ba9854f3e0221cfaef" exitCode=0 Oct 30 13:12:34.617849 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:34.617786 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b817f58-b2kvn" event={"ID":"a2833e9b-a45f-4557-b5c9-075474e38f3f","Type":"ContainerDied","Data":"80b0b7ad6165b05d13b96a3dd18d10c8572298212df926ba9854f3e0221cfaef"} Oct 30 13:12:34.619198 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:34.619177 5922 generic.go:356] "Generic (PLEG): container finished" podID="b4fd49fb-f890-446a-9d22-435e43054ea5" containerID="d74bfad8de0eeede6269e4d45d87d57b58039f528e0fd5837bb3ef20c8d3ec0e" exitCode=0 Oct 30 13:12:34.619310 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:34.619218 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-ac0c58-7df9968d48-46vcj" Oct 30 13:12:34.619310 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:34.619245 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-ac0c58-7df9968d48-46vcj" event={"ID":"b4fd49fb-f890-446a-9d22-435e43054ea5","Type":"ContainerDied","Data":"d74bfad8de0eeede6269e4d45d87d57b58039f528e0fd5837bb3ef20c8d3ec0e"} Oct 30 13:12:34.619310 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:34.619287 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-ac0c58-7df9968d48-46vcj" event={"ID":"b4fd49fb-f890-446a-9d22-435e43054ea5","Type":"ContainerDied","Data":"8b5272a22fba991f048881011de4712d580ffd0d639c542e3b4420f6a22e1a0d"} Oct 30 13:12:34.619480 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:34.619316 5922 scope.go:122] "RemoveContainer" containerID="d74bfad8de0eeede6269e4d45d87d57b58039f528e0fd5837bb3ef20c8d3ec0e" Oct 30 13:12:34.630582 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:34.630564 5922 scope.go:122] "RemoveContainer" containerID="d74bfad8de0eeede6269e4d45d87d57b58039f528e0fd5837bb3ef20c8d3ec0e" Oct 30 13:12:34.630908 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:12:34.630882 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d74bfad8de0eeede6269e4d45d87d57b58039f528e0fd5837bb3ef20c8d3ec0e\": container with ID starting with d74bfad8de0eeede6269e4d45d87d57b58039f528e0fd5837bb3ef20c8d3ec0e not found: ID does not exist" containerID="d74bfad8de0eeede6269e4d45d87d57b58039f528e0fd5837bb3ef20c8d3ec0e" Oct 30 13:12:34.631001 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:34.630919 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d74bfad8de0eeede6269e4d45d87d57b58039f528e0fd5837bb3ef20c8d3ec0e"} err="failed to get container status \"d74bfad8de0eeede6269e4d45d87d57b58039f528e0fd5837bb3ef20c8d3ec0e\": rpc error: code = NotFound desc = could not find container \"d74bfad8de0eeede6269e4d45d87d57b58039f528e0fd5837bb3ef20c8d3ec0e\": container with ID starting with d74bfad8de0eeede6269e4d45d87d57b58039f528e0fd5837bb3ef20c8d3ec0e not found: ID does not exist" Oct 30 13:12:34.698701 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:34.698672 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-ac0c58-7df9968d48-46vcj"] Oct 30 13:12:34.703221 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:34.703199 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-ac0c58-7df9968d48-46vcj"] Oct 30 13:12:34.898264 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:34.898192 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:35.653738 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:35.653689 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4fd49fb-f890-446a-9d22-435e43054ea5" path="/var/lib/kubelet/pods/b4fd49fb-f890-446a-9d22-435e43054ea5/volumes" Oct 30 13:12:35.793109 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:35.793081 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9876320b-9tg8r" Oct 30 13:12:35.824519 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:35.824494 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b817f58-b2kvn" Oct 30 13:12:35.889735 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:35.889666 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b0d2d00-e66c-410a-8383-04a6c34620fd-kube-api-access-tjlps\" (UniqueName: \"kubernetes.io/projected/8b0d2d00-e66c-410a-8383-04a6c34620fd-kube-api-access-tjlps\") pod \"8b0d2d00-e66c-410a-8383-04a6c34620fd\" (UID: \"8b0d2d00-e66c-410a-8383-04a6c34620fd\") " Oct 30 13:12:35.889878 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:35.889745 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b0d2d00-e66c-410a-8383-04a6c34620fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b0d2d00-e66c-410a-8383-04a6c34620fd-maas-api-xks-ca\") pod \"8b0d2d00-e66c-410a-8383-04a6c34620fd\" (UID: \"8b0d2d00-e66c-410a-8383-04a6c34620fd\") " Oct 30 13:12:35.889878 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:35.889812 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b0d2d00-e66c-410a-8383-04a6c34620fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b0d2d00-e66c-410a-8383-04a6c34620fd-maas-api-openshift-ca\") pod \"8b0d2d00-e66c-410a-8383-04a6c34620fd\" (UID: \"8b0d2d00-e66c-410a-8383-04a6c34620fd\") " Oct 30 13:12:35.890684 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:35.890542 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b0d2d00-e66c-410a-8383-04a6c34620fd-maas-api-openshift-ca" pod "8b0d2d00-e66c-410a-8383-04a6c34620fd" (UID: "8b0d2d00-e66c-410a-8383-04a6c34620fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:12:35.891994 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:35.891967 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b0d2d00-e66c-410a-8383-04a6c34620fd-kube-api-access-tjlps" pod "8b0d2d00-e66c-410a-8383-04a6c34620fd" (UID: "8b0d2d00-e66c-410a-8383-04a6c34620fd"). InnerVolumeSpecName "kube-api-access-tjlps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:35.892096 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:35.891990 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b0d2d00-e66c-410a-8383-04a6c34620fd-maas-api-xks-ca" pod "8b0d2d00-e66c-410a-8383-04a6c34620fd" (UID: "8b0d2d00-e66c-410a-8383-04a6c34620fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:12:35.991152 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:35.991124 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2833e9b-a45f-4557-b5c9-075474e38f3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2833e9b-a45f-4557-b5c9-075474e38f3f-maas-api-xks-ca\") pod \"a2833e9b-a45f-4557-b5c9-075474e38f3f\" (UID: \"a2833e9b-a45f-4557-b5c9-075474e38f3f\") " Oct 30 13:12:35.991255 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:35.991243 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2833e9b-a45f-4557-b5c9-075474e38f3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2833e9b-a45f-4557-b5c9-075474e38f3f-maas-api-openshift-ca\") pod \"a2833e9b-a45f-4557-b5c9-075474e38f3f\" (UID: \"a2833e9b-a45f-4557-b5c9-075474e38f3f\") " Oct 30 13:12:35.991387 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:35.991371 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2833e9b-a45f-4557-b5c9-075474e38f3f-kube-api-access-t2h45\" (UniqueName: \"kubernetes.io/projected/a2833e9b-a45f-4557-b5c9-075474e38f3f-kube-api-access-t2h45\") pod \"a2833e9b-a45f-4557-b5c9-075474e38f3f\" (UID: \"a2833e9b-a45f-4557-b5c9-075474e38f3f\") " Oct 30 13:12:35.991563 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:35.991542 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2833e9b-a45f-4557-b5c9-075474e38f3f-maas-api-openshift-ca" pod "a2833e9b-a45f-4557-b5c9-075474e38f3f" (UID: "a2833e9b-a45f-4557-b5c9-075474e38f3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:12:35.991815 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:35.991797 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b0d2d00-e66c-410a-8383-04a6c34620fd-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:35.991886 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:35.991816 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b0d2d00-e66c-410a-8383-04a6c34620fd-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:35.991886 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:35.991828 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2833e9b-a45f-4557-b5c9-075474e38f3f-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:35.991886 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:35.991836 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjlps\" (UniqueName: \"kubernetes.io/projected/8b0d2d00-e66c-410a-8383-04a6c34620fd-kube-api-access-tjlps\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:35.993232 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:35.993208 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2833e9b-a45f-4557-b5c9-075474e38f3f-maas-api-xks-ca" pod "a2833e9b-a45f-4557-b5c9-075474e38f3f" (UID: "a2833e9b-a45f-4557-b5c9-075474e38f3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:12:35.993299 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:35.993275 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2833e9b-a45f-4557-b5c9-075474e38f3f-kube-api-access-t2h45" pod "a2833e9b-a45f-4557-b5c9-075474e38f3f" (UID: "a2833e9b-a45f-4557-b5c9-075474e38f3f"). InnerVolumeSpecName "kube-api-access-t2h45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:36.093428 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:36.093399 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2h45\" (UniqueName: \"kubernetes.io/projected/a2833e9b-a45f-4557-b5c9-075474e38f3f-kube-api-access-t2h45\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:36.093428 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:36.093426 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2833e9b-a45f-4557-b5c9-075474e38f3f-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:36.638254 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:36.638226 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9876320b-9tg8r" Oct 30 13:12:36.638427 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:36.638279 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9876320b-9tg8r" event={"ID":"8b0d2d00-e66c-410a-8383-04a6c34620fd","Type":"ContainerDied","Data":"cd7e76b8b568ae0d4db9d8196920b53ad01bde3455697d7975e493d89dce843d"} Oct 30 13:12:36.638427 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:36.638329 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd7e76b8b568ae0d4db9d8196920b53ad01bde3455697d7975e493d89dce843d" Oct 30 13:12:36.639876 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:36.639850 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b817f58-b2kvn" Oct 30 13:12:36.639996 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:36.639895 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b817f58-b2kvn" event={"ID":"a2833e9b-a45f-4557-b5c9-075474e38f3f","Type":"ContainerDied","Data":"51a19aa5d1fb6381ee7a225e1434d189771b83942b289e801e7aec61eaf20da7"} Oct 30 13:12:36.639996 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:36.639945 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51a19aa5d1fb6381ee7a225e1434d189771b83942b289e801e7aec61eaf20da7" Oct 30 13:12:36.688985 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:36.688951 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:37.699183 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:37.699150 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-54994c-f69fc5d86-68fv5"] Oct 30 13:12:37.738466 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:37.738439 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-54994c-75d97544d7-lvpzs"] Oct 30 13:12:37.850084 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:37.850054 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:37.968268 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:37.968175 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:38.037343 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:38.037307 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a702fe8a-ggd29"] Oct 30 13:12:38.041466 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:38.041435 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a702fe8a-ggd29"] Oct 30 13:12:38.277309 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:38.277221 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac6ffc9f-jr8sq"] Oct 30 13:12:38.277866 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:38.277844 5922 cpu_manager.go:414] "containerMap: removing container" podUID="a2833e9b-a45f-4557-b5c9-075474e38f3f" containerName="revoke-keys" Oct 30 13:12:38.277866 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:38.277866 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2833e9b-a45f-4557-b5c9-075474e38f3f" containerName="revoke-keys" Oct 30 13:12:38.278047 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:38.277906 5922 cpu_manager.go:414] "containerMap: removing container" podUID="b4fd49fb-f890-446a-9d22-435e43054ea5" containerName="payload-processing" Oct 30 13:12:38.278047 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:38.277914 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4fd49fb-f890-446a-9d22-435e43054ea5" containerName="payload-processing" Oct 30 13:12:38.278047 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:38.277941 5922 cpu_manager.go:414] "containerMap: removing container" podUID="8b0d2d00-e66c-410a-8383-04a6c34620fd" containerName="revoke-keys" Oct 30 13:12:38.278047 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:38.277948 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b0d2d00-e66c-410a-8383-04a6c34620fd" containerName="revoke-keys" Oct 30 13:12:38.278235 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:38.278083 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b0d2d00-e66c-410a-8383-04a6c34620fd" containerName="revoke-keys" Oct 30 13:12:38.278235 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:38.278105 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2833e9b-a45f-4557-b5c9-075474e38f3f" containerName="revoke-keys" Oct 30 13:12:38.278235 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:38.278119 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4fd49fb-f890-446a-9d22-435e43054ea5" containerName="payload-processing" Oct 30 13:12:38.281981 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:38.281959 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac6ffc9f-jr8sq" Oct 30 13:12:38.298229 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:38.298205 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:12:38.298329 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:38.298301 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:12:38.300924 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:38.300887 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac6ffc9f-jr8sq"] Oct 30 13:12:38.307141 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:38.307116 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:12:38.307236 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:38.307119 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:12:38.314077 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:38.314051 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3391a008-kbn48"] Oct 30 13:12:38.319209 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:38.319191 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3391a008-kbn48" Oct 30 13:12:38.339650 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:38.337579 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3391a008-kbn48"] Oct 30 13:12:38.420942 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:38.420879 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9prqr\" (UniqueName: \"kubernetes.io/projected/6eac003e-b8a0-4b88-a296-59e5a9c85a05-kube-api-access-9prqr\") pod \"maas-api-revoke-sub-3391a008-kbn48\" (UID: \"6eac003e-b8a0-4b88-a296-59e5a9c85a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3391a008-kbn48" Oct 30 13:12:38.421116 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:38.420953 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfg4h\" (UniqueName: \"kubernetes.io/projected/63616383-42a4-4a01-96c0-a7aafaf76c28-kube-api-access-jfg4h\") pod \"maas-api-revoke-sub-ac6ffc9f-jr8sq\" (UID: \"63616383-42a4-4a01-96c0-a7aafaf76c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac6ffc9f-jr8sq" Oct 30 13:12:38.421116 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:38.421076 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6eac003e-b8a0-4b88-a296-59e5a9c85a05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3391a008-kbn48\" (UID: \"6eac003e-b8a0-4b88-a296-59e5a9c85a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3391a008-kbn48" Oct 30 13:12:38.421226 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:38.421124 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63616383-42a4-4a01-96c0-a7aafaf76c28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac6ffc9f-jr8sq\" (UID: \"63616383-42a4-4a01-96c0-a7aafaf76c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac6ffc9f-jr8sq" Oct 30 13:12:38.421226 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:38.421160 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6eac003e-b8a0-4b88-a296-59e5a9c85a05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3391a008-kbn48\" (UID: \"6eac003e-b8a0-4b88-a296-59e5a9c85a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3391a008-kbn48" Oct 30 13:12:38.421480 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:38.421457 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63616383-42a4-4a01-96c0-a7aafaf76c28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac6ffc9f-jr8sq\" (UID: \"63616383-42a4-4a01-96c0-a7aafaf76c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac6ffc9f-jr8sq" Oct 30 13:12:38.523149 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:38.523114 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63616383-42a4-4a01-96c0-a7aafaf76c28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac6ffc9f-jr8sq\" (UID: \"63616383-42a4-4a01-96c0-a7aafaf76c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac6ffc9f-jr8sq" Oct 30 13:12:38.523318 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:38.523190 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9prqr\" (UniqueName: \"kubernetes.io/projected/6eac003e-b8a0-4b88-a296-59e5a9c85a05-kube-api-access-9prqr\") pod \"maas-api-revoke-sub-3391a008-kbn48\" (UID: \"6eac003e-b8a0-4b88-a296-59e5a9c85a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3391a008-kbn48" Oct 30 13:12:38.523318 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:38.523214 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfg4h\" (UniqueName: \"kubernetes.io/projected/63616383-42a4-4a01-96c0-a7aafaf76c28-kube-api-access-jfg4h\") pod \"maas-api-revoke-sub-ac6ffc9f-jr8sq\" (UID: \"63616383-42a4-4a01-96c0-a7aafaf76c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac6ffc9f-jr8sq" Oct 30 13:12:38.523318 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:38.523292 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6eac003e-b8a0-4b88-a296-59e5a9c85a05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3391a008-kbn48\" (UID: \"6eac003e-b8a0-4b88-a296-59e5a9c85a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3391a008-kbn48" Oct 30 13:12:38.523318 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:38.523315 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63616383-42a4-4a01-96c0-a7aafaf76c28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac6ffc9f-jr8sq\" (UID: \"63616383-42a4-4a01-96c0-a7aafaf76c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac6ffc9f-jr8sq" Oct 30 13:12:38.523559 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:38.523343 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6eac003e-b8a0-4b88-a296-59e5a9c85a05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3391a008-kbn48\" (UID: \"6eac003e-b8a0-4b88-a296-59e5a9c85a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3391a008-kbn48" Oct 30 13:12:38.524004 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:38.523977 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63616383-42a4-4a01-96c0-a7aafaf76c28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac6ffc9f-jr8sq\" (UID: \"63616383-42a4-4a01-96c0-a7aafaf76c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac6ffc9f-jr8sq" Oct 30 13:12:38.524114 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:38.524034 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6eac003e-b8a0-4b88-a296-59e5a9c85a05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3391a008-kbn48\" (UID: \"6eac003e-b8a0-4b88-a296-59e5a9c85a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3391a008-kbn48" Oct 30 13:12:38.525761 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:38.525744 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63616383-42a4-4a01-96c0-a7aafaf76c28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac6ffc9f-jr8sq\" (UID: \"63616383-42a4-4a01-96c0-a7aafaf76c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac6ffc9f-jr8sq" Oct 30 13:12:38.526022 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:38.526001 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6eac003e-b8a0-4b88-a296-59e5a9c85a05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3391a008-kbn48\" (UID: \"6eac003e-b8a0-4b88-a296-59e5a9c85a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3391a008-kbn48" Oct 30 13:12:38.547462 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:38.547409 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfg4h\" (UniqueName: \"kubernetes.io/projected/63616383-42a4-4a01-96c0-a7aafaf76c28-kube-api-access-jfg4h\") pod \"maas-api-revoke-sub-ac6ffc9f-jr8sq\" (UID: \"63616383-42a4-4a01-96c0-a7aafaf76c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac6ffc9f-jr8sq" Oct 30 13:12:38.547581 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:38.547567 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9prqr\" (UniqueName: \"kubernetes.io/projected/6eac003e-b8a0-4b88-a296-59e5a9c85a05-kube-api-access-9prqr\") pod \"maas-api-revoke-sub-3391a008-kbn48\" (UID: \"6eac003e-b8a0-4b88-a296-59e5a9c85a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3391a008-kbn48" Oct 30 13:12:38.596776 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:38.596749 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac6ffc9f-jr8sq" Oct 30 13:12:38.640870 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:38.640840 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3391a008-kbn48" Oct 30 13:12:38.902453 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:38.902421 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac6ffc9f-jr8sq"] Oct 30 13:12:38.902871 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:12:38.902846 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63616383_42a4_4a01_96c0_a7aafaf76c28.slice/crio-b4724c0bb27e82d97b197593ad38536d329e8381b977236bc6eb08ba7cfb0559 WatchSource:0}: Error finding container b4724c0bb27e82d97b197593ad38536d329e8381b977236bc6eb08ba7cfb0559: Status 404 returned error can't find the container with id b4724c0bb27e82d97b197593ad38536d329e8381b977236bc6eb08ba7cfb0559 Oct 30 13:12:38.967737 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:38.967706 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3391a008-kbn48"] Oct 30 13:12:39.064013 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:12:39.063981 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eac003e_b8a0_4b88_a296_59e5a9c85a05.slice/crio-63dcebe7fbc251cf45d0b136d66eca49875cc4caa6013e8eaf3aab61b7fb8604 WatchSource:0}: Error finding container 63dcebe7fbc251cf45d0b136d66eca49875cc4caa6013e8eaf3aab61b7fb8604: Status 404 returned error can't find the container with id 63dcebe7fbc251cf45d0b136d66eca49875cc4caa6013e8eaf3aab61b7fb8604 Oct 30 13:12:39.166153 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:39.166095 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:39.644169 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:39.644137 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f068e94-22e4-4154-b2ca-fe19368f0c86" path="/var/lib/kubelet/pods/0f068e94-22e4-4154-b2ca-fe19368f0c86/volumes" Oct 30 13:12:39.645047 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:39.645019 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:39.678906 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:39.678879 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3391a008-kbn48" event={"ID":"6eac003e-b8a0-4b88-a296-59e5a9c85a05","Type":"ContainerStarted","Data":"45a21f64fbecf72930cb4860987efc909d9f8bb5417773bb06baed95c7f54ea1"} Oct 30 13:12:39.679025 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:39.678935 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3391a008-kbn48" event={"ID":"6eac003e-b8a0-4b88-a296-59e5a9c85a05","Type":"ContainerStarted","Data":"63dcebe7fbc251cf45d0b136d66eca49875cc4caa6013e8eaf3aab61b7fb8604"} Oct 30 13:12:39.680976 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:39.680952 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac6ffc9f-jr8sq" event={"ID":"63616383-42a4-4a01-96c0-a7aafaf76c28","Type":"ContainerStarted","Data":"a3159e6cdfb74c3491c86fa9e7552ed0afc53029e08cef787d03227e91598f2f"} Oct 30 13:12:39.681069 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:39.680994 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac6ffc9f-jr8sq" event={"ID":"63616383-42a4-4a01-96c0-a7aafaf76c28","Type":"ContainerStarted","Data":"b4724c0bb27e82d97b197593ad38536d329e8381b977236bc6eb08ba7cfb0559"} Oct 30 13:12:39.726533 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:39.726451 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3391a008-kbn48" podStartSLOduration=1.726439331 podStartE2EDuration="1.726439331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:39.723075438 +0000 UTC m=+1216.192491183" watchObservedRunningTime="2026-10-30 13:12:39.726439331 +0000 UTC m=+1216.195855076" Oct 30 13:12:39.768151 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:39.768097 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac6ffc9f-jr8sq" podStartSLOduration=1.76807948 podStartE2EDuration="1.76807948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:39.766005021 +0000 UTC m=+1216.235420779" watchObservedRunningTime="2026-10-30 13:12:39.76807948 +0000 UTC m=+1216.237495228" Oct 30 13:12:40.116062 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:40.116028 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:40.689983 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:40.689950 5922 generic.go:356] "Generic (PLEG): container finished" podID="63616383-42a4-4a01-96c0-a7aafaf76c28" containerID="a3159e6cdfb74c3491c86fa9e7552ed0afc53029e08cef787d03227e91598f2f" exitCode=0 Oct 30 13:12:40.690149 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:40.690033 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac6ffc9f-jr8sq" event={"ID":"63616383-42a4-4a01-96c0-a7aafaf76c28","Type":"ContainerDied","Data":"a3159e6cdfb74c3491c86fa9e7552ed0afc53029e08cef787d03227e91598f2f"} Oct 30 13:12:40.691532 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:40.691506 5922 generic.go:356] "Generic (PLEG): container finished" podID="6eac003e-b8a0-4b88-a296-59e5a9c85a05" containerID="45a21f64fbecf72930cb4860987efc909d9f8bb5417773bb06baed95c7f54ea1" exitCode=0 Oct 30 13:12:40.691676 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:40.691542 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3391a008-kbn48" event={"ID":"6eac003e-b8a0-4b88-a296-59e5a9c85a05","Type":"ContainerDied","Data":"45a21f64fbecf72930cb4860987efc909d9f8bb5417773bb06baed95c7f54ea1"} Oct 30 13:12:41.310547 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:41.310515 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:41.853958 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:41.853932 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3391a008-kbn48" Oct 30 13:12:41.907243 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:41.907216 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac6ffc9f-jr8sq" Oct 30 13:12:41.930293 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:41.930261 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:12:41.965582 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:41.965516 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63616383-42a4-4a01-96c0-a7aafaf76c28-kube-api-access-jfg4h\" (UniqueName: \"kubernetes.io/projected/63616383-42a4-4a01-96c0-a7aafaf76c28-kube-api-access-jfg4h\") pod \"63616383-42a4-4a01-96c0-a7aafaf76c28\" (UID: \"63616383-42a4-4a01-96c0-a7aafaf76c28\") " Oct 30 13:12:41.965744 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:41.965633 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6eac003e-b8a0-4b88-a296-59e5a9c85a05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6eac003e-b8a0-4b88-a296-59e5a9c85a05-maas-api-openshift-ca\") pod \"6eac003e-b8a0-4b88-a296-59e5a9c85a05\" (UID: \"6eac003e-b8a0-4b88-a296-59e5a9c85a05\") " Oct 30 13:12:41.965744 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:41.965680 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6eac003e-b8a0-4b88-a296-59e5a9c85a05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6eac003e-b8a0-4b88-a296-59e5a9c85a05-maas-api-xks-ca\") pod \"6eac003e-b8a0-4b88-a296-59e5a9c85a05\" (UID: \"6eac003e-b8a0-4b88-a296-59e5a9c85a05\") " Oct 30 13:12:41.965744 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:41.965738 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63616383-42a4-4a01-96c0-a7aafaf76c28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63616383-42a4-4a01-96c0-a7aafaf76c28-maas-api-openshift-ca\") pod \"63616383-42a4-4a01-96c0-a7aafaf76c28\" (UID: \"63616383-42a4-4a01-96c0-a7aafaf76c28\") " Oct 30 13:12:41.965933 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:41.965769 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63616383-42a4-4a01-96c0-a7aafaf76c28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63616383-42a4-4a01-96c0-a7aafaf76c28-maas-api-xks-ca\") pod \"63616383-42a4-4a01-96c0-a7aafaf76c28\" (UID: \"63616383-42a4-4a01-96c0-a7aafaf76c28\") " Oct 30 13:12:41.965933 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:41.965859 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eac003e-b8a0-4b88-a296-59e5a9c85a05-kube-api-access-9prqr\" (UniqueName: \"kubernetes.io/projected/6eac003e-b8a0-4b88-a296-59e5a9c85a05-kube-api-access-9prqr\") pod \"6eac003e-b8a0-4b88-a296-59e5a9c85a05\" (UID: \"6eac003e-b8a0-4b88-a296-59e5a9c85a05\") " Oct 30 13:12:41.966126 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:41.966100 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eac003e-b8a0-4b88-a296-59e5a9c85a05-maas-api-openshift-ca" pod "6eac003e-b8a0-4b88-a296-59e5a9c85a05" (UID: "6eac003e-b8a0-4b88-a296-59e5a9c85a05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:12:41.966377 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:41.966301 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63616383-42a4-4a01-96c0-a7aafaf76c28-maas-api-openshift-ca" pod "63616383-42a4-4a01-96c0-a7aafaf76c28" (UID: "63616383-42a4-4a01-96c0-a7aafaf76c28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:12:41.966499 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:41.966465 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6eac003e-b8a0-4b88-a296-59e5a9c85a05-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:41.966499 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:41.966488 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63616383-42a4-4a01-96c0-a7aafaf76c28-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:41.968159 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:41.968065 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63616383-42a4-4a01-96c0-a7aafaf76c28-kube-api-access-jfg4h" pod "63616383-42a4-4a01-96c0-a7aafaf76c28" (UID: "63616383-42a4-4a01-96c0-a7aafaf76c28"). InnerVolumeSpecName "kube-api-access-jfg4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:41.968735 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:41.968567 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6eac003e-b8a0-4b88-a296-59e5a9c85a05-maas-api-xks-ca" pod "6eac003e-b8a0-4b88-a296-59e5a9c85a05" (UID: "6eac003e-b8a0-4b88-a296-59e5a9c85a05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:12:41.968735 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:41.968710 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eac003e-b8a0-4b88-a296-59e5a9c85a05-kube-api-access-9prqr" pod "6eac003e-b8a0-4b88-a296-59e5a9c85a05" (UID: "6eac003e-b8a0-4b88-a296-59e5a9c85a05"). InnerVolumeSpecName "kube-api-access-9prqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:41.970289 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:41.970265 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63616383-42a4-4a01-96c0-a7aafaf76c28-maas-api-xks-ca" pod "63616383-42a4-4a01-96c0-a7aafaf76c28" (UID: "63616383-42a4-4a01-96c0-a7aafaf76c28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:12:42.067458 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:42.067418 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63616383-42a4-4a01-96c0-a7aafaf76c28-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:42.067458 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:42.067455 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9prqr\" (UniqueName: \"kubernetes.io/projected/6eac003e-b8a0-4b88-a296-59e5a9c85a05-kube-api-access-9prqr\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:42.067458 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:42.067465 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfg4h\" (UniqueName: \"kubernetes.io/projected/63616383-42a4-4a01-96c0-a7aafaf76c28-kube-api-access-jfg4h\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:42.067700 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:42.067476 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6eac003e-b8a0-4b88-a296-59e5a9c85a05-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:42.699753 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:42.699716 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-54994c-f69fc5d86-68fv5" podUID="727a0820-c1f2-469e-827d-3f623e9df963" containerName="payload-processing" containerID="cri-o://c316cebabc1a6f07a6e8b66c0b4e6cd81ef845b3954ca7962283e92872d1781a" gracePeriod=25 Oct 30 13:12:42.712529 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:42.712505 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac6ffc9f-jr8sq" Oct 30 13:12:42.712529 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:42.712522 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac6ffc9f-jr8sq" event={"ID":"63616383-42a4-4a01-96c0-a7aafaf76c28","Type":"ContainerDied","Data":"b4724c0bb27e82d97b197593ad38536d329e8381b977236bc6eb08ba7cfb0559"} Oct 30 13:12:42.712787 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:42.712572 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4724c0bb27e82d97b197593ad38536d329e8381b977236bc6eb08ba7cfb0559" Oct 30 13:12:42.714300 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:42.714283 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3391a008-kbn48" Oct 30 13:12:42.714411 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:42.714335 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3391a008-kbn48" event={"ID":"6eac003e-b8a0-4b88-a296-59e5a9c85a05","Type":"ContainerDied","Data":"63dcebe7fbc251cf45d0b136d66eca49875cc4caa6013e8eaf3aab61b7fb8604"} Oct 30 13:12:42.714411 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:42.714382 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63dcebe7fbc251cf45d0b136d66eca49875cc4caa6013e8eaf3aab61b7fb8604" Oct 30 13:12:42.738976 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:42.738950 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-54994c-75d97544d7-lvpzs" podUID="ac077401-1be7-4461-a1e5-bc4f186c9a94" containerName="payload-pre-processing" containerID="cri-o://fc5b24620981b6fad12282a9e22d5812e805032b934d1e30da4d192be0d4b6be" gracePeriod=25 Oct 30 13:12:43.366158 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:43.366122 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:44.338393 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:44.338316 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-54994c-f69fc5d86-68fv5" podUID="727a0820-c1f2-469e-827d-3f623e9df963" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:12:44.338921 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:44.338688 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-54994c-75d97544d7-lvpzs" podUID="ac077401-1be7-4461-a1e5-bc4f186c9a94" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:12:44.909822 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:44.909766 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:45.044995 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:45.044960 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04d0392c-j658p"] Oct 30 13:12:45.057921 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:45.057894 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04d0392c-j658p"] Oct 30 13:12:45.095433 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:45.095396 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg"] Oct 30 13:12:45.229972 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:45.229936 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:45.643998 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:45.643923 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3091a06-b26b-4793-8d07-4f1d96bc1aa4" path="/var/lib/kubelet/pods/d3091a06-b26b-4793-8d07-4f1d96bc1aa4/volumes" Oct 30 13:12:45.761435 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:45.761404 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:45.950338 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:45.950282 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:45.981545 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:45.981519 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ac0c58-kn9tt"] Oct 30 13:12:45.993171 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:45.993140 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ac0c58-kn9tt"] Oct 30 13:12:46.088079 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:46.088046 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:46.555501 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:46.555466 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:47.039176 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.039140 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-527643e2-9knlt"] Oct 30 13:12:47.050981 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.050957 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-527643e2-9knlt"] Oct 30 13:12:47.059909 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.059880 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" podUID="7751779d-b082-409a-a20f-ff0ff6263808" containerName="main" containerID="cri-o://f5908974c0c7d89bba23811e14edabc6cf167e63cc0957868f331697322b9c40" gracePeriod=45 Oct 30 13:12:47.070070 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.070041 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" podUID="35619ab9-cbeb-45da-8506-3af63a273177" containerName="main" containerID="cri-o://c9da677773175ee3fc9879e486f99b1fa783cce48bb86126759c48c48c28305c" gracePeriod=45 Oct 30 13:12:47.135426 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.135395 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" podUID="7751779d-b082-409a-a20f-ff0ff6263808" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": dial tcp 10.134.0.57:8000: connect: connection refused" Oct 30 13:12:47.355505 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.355481 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" Oct 30 13:12:47.385250 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.385229 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" Oct 30 13:12:47.432483 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.432450 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35619ab9-cbeb-45da-8506-3af63a273177-home\" (UniqueName: \"kubernetes.io/empty-dir/35619ab9-cbeb-45da-8506-3af63a273177-home\") pod \"35619ab9-cbeb-45da-8506-3af63a273177\" (UID: \"35619ab9-cbeb-45da-8506-3af63a273177\") " Oct 30 13:12:47.432656 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.432526 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7751779d-b082-409a-a20f-ff0ff6263808-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7751779d-b082-409a-a20f-ff0ff6263808-model-cache\") pod \"7751779d-b082-409a-a20f-ff0ff6263808\" (UID: \"7751779d-b082-409a-a20f-ff0ff6263808\") " Oct 30 13:12:47.432656 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.432566 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7751779d-b082-409a-a20f-ff0ff6263808-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7751779d-b082-409a-a20f-ff0ff6263808-tmp-dir\") pod \"7751779d-b082-409a-a20f-ff0ff6263808\" (UID: \"7751779d-b082-409a-a20f-ff0ff6263808\") " Oct 30 13:12:47.432784 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.432652 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35619ab9-cbeb-45da-8506-3af63a273177-kube-api-access-mlwzf\" (UniqueName: \"kubernetes.io/projected/35619ab9-cbeb-45da-8506-3af63a273177-kube-api-access-mlwzf\") pod \"35619ab9-cbeb-45da-8506-3af63a273177\" (UID: \"35619ab9-cbeb-45da-8506-3af63a273177\") " Oct 30 13:12:47.432784 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.432701 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35619ab9-cbeb-45da-8506-3af63a273177-tls-certs\" (UniqueName: \"kubernetes.io/secret/35619ab9-cbeb-45da-8506-3af63a273177-tls-certs\") pod \"35619ab9-cbeb-45da-8506-3af63a273177\" (UID: \"35619ab9-cbeb-45da-8506-3af63a273177\") " Oct 30 13:12:47.432784 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.432732 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35619ab9-cbeb-45da-8506-3af63a273177-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35619ab9-cbeb-45da-8506-3af63a273177-model-cache\") pod \"35619ab9-cbeb-45da-8506-3af63a273177\" (UID: \"35619ab9-cbeb-45da-8506-3af63a273177\") " Oct 30 13:12:47.432784 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.432757 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35619ab9-cbeb-45da-8506-3af63a273177-home" pod "35619ab9-cbeb-45da-8506-3af63a273177" (UID: "35619ab9-cbeb-45da-8506-3af63a273177"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:47.432784 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.432777 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35619ab9-cbeb-45da-8506-3af63a273177-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35619ab9-cbeb-45da-8506-3af63a273177-tmp-dir\") pod \"35619ab9-cbeb-45da-8506-3af63a273177\" (UID: \"35619ab9-cbeb-45da-8506-3af63a273177\") " Oct 30 13:12:47.433074 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.432804 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7751779d-b082-409a-a20f-ff0ff6263808-model-cache" pod "7751779d-b082-409a-a20f-ff0ff6263808" (UID: "7751779d-b082-409a-a20f-ff0ff6263808"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:47.433074 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.432824 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7751779d-b082-409a-a20f-ff0ff6263808-home\" (UniqueName: \"kubernetes.io/empty-dir/7751779d-b082-409a-a20f-ff0ff6263808-home\") pod \"7751779d-b082-409a-a20f-ff0ff6263808\" (UID: \"7751779d-b082-409a-a20f-ff0ff6263808\") " Oct 30 13:12:47.433074 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.432853 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7751779d-b082-409a-a20f-ff0ff6263808-tmp-dir" pod "7751779d-b082-409a-a20f-ff0ff6263808" (UID: "7751779d-b082-409a-a20f-ff0ff6263808"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:47.433074 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.432911 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7751779d-b082-409a-a20f-ff0ff6263808-kube-api-access-s2bn2\" (UniqueName: \"kubernetes.io/projected/7751779d-b082-409a-a20f-ff0ff6263808-kube-api-access-s2bn2\") pod \"7751779d-b082-409a-a20f-ff0ff6263808\" (UID: \"7751779d-b082-409a-a20f-ff0ff6263808\") " Oct 30 13:12:47.433074 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.432981 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7751779d-b082-409a-a20f-ff0ff6263808-tls-certs\" (UniqueName: \"kubernetes.io/secret/7751779d-b082-409a-a20f-ff0ff6263808-tls-certs\") pod \"7751779d-b082-409a-a20f-ff0ff6263808\" (UID: \"7751779d-b082-409a-a20f-ff0ff6263808\") " Oct 30 13:12:47.433074 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.433028 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35619ab9-cbeb-45da-8506-3af63a273177-dshm\" (UniqueName: \"kubernetes.io/empty-dir/35619ab9-cbeb-45da-8506-3af63a273177-dshm\") pod \"35619ab9-cbeb-45da-8506-3af63a273177\" (UID: \"35619ab9-cbeb-45da-8506-3af63a273177\") " Oct 30 13:12:47.433074 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.433035 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7751779d-b082-409a-a20f-ff0ff6263808-home" pod "7751779d-b082-409a-a20f-ff0ff6263808" (UID: "7751779d-b082-409a-a20f-ff0ff6263808"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:47.433074 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.433068 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35619ab9-cbeb-45da-8506-3af63a273177-model-cache" pod "35619ab9-cbeb-45da-8506-3af63a273177" (UID: "35619ab9-cbeb-45da-8506-3af63a273177"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:47.433486 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.433095 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7751779d-b082-409a-a20f-ff0ff6263808-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7751779d-b082-409a-a20f-ff0ff6263808-dshm\") pod \"7751779d-b082-409a-a20f-ff0ff6263808\" (UID: \"7751779d-b082-409a-a20f-ff0ff6263808\") " Oct 30 13:12:47.433486 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.433240 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35619ab9-cbeb-45da-8506-3af63a273177-tmp-dir" pod "35619ab9-cbeb-45da-8506-3af63a273177" (UID: "35619ab9-cbeb-45da-8506-3af63a273177"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:47.433792 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.433766 5922 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35619ab9-cbeb-45da-8506-3af63a273177-tmp-dir\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:47.433904 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.433796 5922 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7751779d-b082-409a-a20f-ff0ff6263808-home\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:47.433904 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.433812 5922 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35619ab9-cbeb-45da-8506-3af63a273177-home\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:47.433904 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.433823 5922 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7751779d-b082-409a-a20f-ff0ff6263808-model-cache\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:47.433904 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.433838 5922 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7751779d-b082-409a-a20f-ff0ff6263808-tmp-dir\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:47.433904 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.433851 5922 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35619ab9-cbeb-45da-8506-3af63a273177-model-cache\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:47.435435 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.435410 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35619ab9-cbeb-45da-8506-3af63a273177-tls-certs" pod "35619ab9-cbeb-45da-8506-3af63a273177" (UID: "35619ab9-cbeb-45da-8506-3af63a273177"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:12:47.436307 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.436269 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7751779d-b082-409a-a20f-ff0ff6263808-kube-api-access-s2bn2" pod "7751779d-b082-409a-a20f-ff0ff6263808" (UID: "7751779d-b082-409a-a20f-ff0ff6263808"). InnerVolumeSpecName "kube-api-access-s2bn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:47.436413 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.436317 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35619ab9-cbeb-45da-8506-3af63a273177-dshm" pod "35619ab9-cbeb-45da-8506-3af63a273177" (UID: "35619ab9-cbeb-45da-8506-3af63a273177"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:47.436821 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.436789 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7751779d-b082-409a-a20f-ff0ff6263808-tls-certs" pod "7751779d-b082-409a-a20f-ff0ff6263808" (UID: "7751779d-b082-409a-a20f-ff0ff6263808"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:12:47.437736 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.437709 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35619ab9-cbeb-45da-8506-3af63a273177-kube-api-access-mlwzf" pod "35619ab9-cbeb-45da-8506-3af63a273177" (UID: "35619ab9-cbeb-45da-8506-3af63a273177"). InnerVolumeSpecName "kube-api-access-mlwzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:47.438180 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.437900 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7751779d-b082-409a-a20f-ff0ff6263808-dshm" pod "7751779d-b082-409a-a20f-ff0ff6263808" (UID: "7751779d-b082-409a-a20f-ff0ff6263808"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:47.535263 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.535229 5922 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7751779d-b082-409a-a20f-ff0ff6263808-tls-certs\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:47.535263 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.535256 5922 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35619ab9-cbeb-45da-8506-3af63a273177-dshm\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:47.535263 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.535265 5922 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7751779d-b082-409a-a20f-ff0ff6263808-dshm\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:47.535492 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.535273 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlwzf\" (UniqueName: \"kubernetes.io/projected/35619ab9-cbeb-45da-8506-3af63a273177-kube-api-access-mlwzf\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:47.535492 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.535283 5922 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35619ab9-cbeb-45da-8506-3af63a273177-tls-certs\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:47.535492 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.535292 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2bn2\" (UniqueName: \"kubernetes.io/projected/7751779d-b082-409a-a20f-ff0ff6263808-kube-api-access-s2bn2\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:47.654432 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.654343 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7550010c-6165-4d5f-90bf-52872faa8a8c" path="/var/lib/kubelet/pods/7550010c-6165-4d5f-90bf-52872faa8a8c/volumes" Oct 30 13:12:47.654964 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.654944 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f871ce6c-6eae-42e7-840f-1c897f8c53ce" path="/var/lib/kubelet/pods/f871ce6c-6eae-42e7-840f-1c897f8c53ce/volumes" Oct 30 13:12:47.754778 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.754747 5922 generic.go:356] "Generic (PLEG): container finished" podID="35619ab9-cbeb-45da-8506-3af63a273177" containerID="c9da677773175ee3fc9879e486f99b1fa783cce48bb86126759c48c48c28305c" exitCode=0 Oct 30 13:12:47.754944 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.754806 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" Oct 30 13:12:47.754944 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.754823 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" event={"ID":"35619ab9-cbeb-45da-8506-3af63a273177","Type":"ContainerDied","Data":"c9da677773175ee3fc9879e486f99b1fa783cce48bb86126759c48c48c28305c"} Oct 30 13:12:47.754944 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.754867 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr" event={"ID":"35619ab9-cbeb-45da-8506-3af63a273177","Type":"ContainerDied","Data":"ba4385f4c37ed704a7a569f4799d7dc2443bdc4ebeb05a922b6e32beb5f67289"} Oct 30 13:12:47.754944 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.754888 5922 scope.go:122] "RemoveContainer" containerID="c9da677773175ee3fc9879e486f99b1fa783cce48bb86126759c48c48c28305c" Oct 30 13:12:47.756830 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.756805 5922 generic.go:356] "Generic (PLEG): container finished" podID="7751779d-b082-409a-a20f-ff0ff6263808" containerID="f5908974c0c7d89bba23811e14edabc6cf167e63cc0957868f331697322b9c40" exitCode=0 Oct 30 13:12:47.756969 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.756912 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" Oct 30 13:12:47.756969 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.756938 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" event={"ID":"7751779d-b082-409a-a20f-ff0ff6263808","Type":"ContainerDied","Data":"f5908974c0c7d89bba23811e14edabc6cf167e63cc0957868f331697322b9c40"} Oct 30 13:12:47.757081 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.756970 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86" event={"ID":"7751779d-b082-409a-a20f-ff0ff6263808","Type":"ContainerDied","Data":"916756c58cca3da6d680d85426798484ebcbeed1257e6ba1b6fc2eb1ed9e2701"} Oct 30 13:12:47.766214 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.766187 5922 scope.go:122] "RemoveContainer" containerID="c9da677773175ee3fc9879e486f99b1fa783cce48bb86126759c48c48c28305c" Oct 30 13:12:47.766749 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:12:47.766726 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9da677773175ee3fc9879e486f99b1fa783cce48bb86126759c48c48c28305c\": container with ID starting with c9da677773175ee3fc9879e486f99b1fa783cce48bb86126759c48c48c28305c not found: ID does not exist" containerID="c9da677773175ee3fc9879e486f99b1fa783cce48bb86126759c48c48c28305c" Oct 30 13:12:47.766849 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.766754 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9da677773175ee3fc9879e486f99b1fa783cce48bb86126759c48c48c28305c"} err="failed to get container status \"c9da677773175ee3fc9879e486f99b1fa783cce48bb86126759c48c48c28305c\": rpc error: code = NotFound desc = could not find container \"c9da677773175ee3fc9879e486f99b1fa783cce48bb86126759c48c48c28305c\": container with ID starting with c9da677773175ee3fc9879e486f99b1fa783cce48bb86126759c48c48c28305c not found: ID does not exist" Oct 30 13:12:47.766849 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.766772 5922 scope.go:122] "RemoveContainer" containerID="f5908974c0c7d89bba23811e14edabc6cf167e63cc0957868f331697322b9c40" Oct 30 13:12:47.776320 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.776297 5922 scope.go:122] "RemoveContainer" containerID="f5908974c0c7d89bba23811e14edabc6cf167e63cc0957868f331697322b9c40" Oct 30 13:12:47.776586 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:12:47.776566 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5908974c0c7d89bba23811e14edabc6cf167e63cc0957868f331697322b9c40\": container with ID starting with f5908974c0c7d89bba23811e14edabc6cf167e63cc0957868f331697322b9c40 not found: ID does not exist" containerID="f5908974c0c7d89bba23811e14edabc6cf167e63cc0957868f331697322b9c40" Oct 30 13:12:47.776709 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.776590 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5908974c0c7d89bba23811e14edabc6cf167e63cc0957868f331697322b9c40"} err="failed to get container status \"f5908974c0c7d89bba23811e14edabc6cf167e63cc0957868f331697322b9c40\": rpc error: code = NotFound desc = could not find container \"f5908974c0c7d89bba23811e14edabc6cf167e63cc0957868f331697322b9c40\": container with ID starting with f5908974c0c7d89bba23811e14edabc6cf167e63cc0957868f331697322b9c40 not found: ID does not exist" Oct 30 13:12:47.811964 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.811937 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86"] Oct 30 13:12:47.819530 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.819505 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-54994c/premium-w4-54994c-kserve-5c69d8c5bd-hbs86"] Oct 30 13:12:47.855584 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.855558 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr"] Oct 30 13:12:47.860030 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.860000 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:47.861347 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:47.861329 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-54994c/simulator-w4-54994c-kserve-f546dd889-h27lr"] Oct 30 13:12:48.040872 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:48.040841 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:49.644849 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:49.644819 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35619ab9-cbeb-45da-8506-3af63a273177" path="/var/lib/kubelet/pods/35619ab9-cbeb-45da-8506-3af63a273177/volumes" Oct 30 13:12:49.645253 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:49.645237 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7751779d-b082-409a-a20f-ff0ff6263808" path="/var/lib/kubelet/pods/7751779d-b082-409a-a20f-ff0ff6263808/volumes" Oct 30 13:12:49.928853 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:49.928771 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:50.766959 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:50.766929 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f"] Oct 30 13:12:50.767427 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:50.767413 5922 cpu_manager.go:414] "containerMap: removing container" podUID="7751779d-b082-409a-a20f-ff0ff6263808" containerName="main" Oct 30 13:12:50.767483 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:50.767428 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7751779d-b082-409a-a20f-ff0ff6263808" containerName="main" Oct 30 13:12:50.767483 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:50.767453 5922 cpu_manager.go:414] "containerMap: removing container" podUID="35619ab9-cbeb-45da-8506-3af63a273177" containerName="main" Oct 30 13:12:50.767483 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:50.767459 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35619ab9-cbeb-45da-8506-3af63a273177" containerName="main" Oct 30 13:12:50.767483 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:50.767479 5922 cpu_manager.go:414] "containerMap: removing container" podUID="6eac003e-b8a0-4b88-a296-59e5a9c85a05" containerName="revoke-keys" Oct 30 13:12:50.767483 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:50.767485 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eac003e-b8a0-4b88-a296-59e5a9c85a05" containerName="revoke-keys" Oct 30 13:12:50.767656 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:50.767496 5922 cpu_manager.go:414] "containerMap: removing container" podUID="63616383-42a4-4a01-96c0-a7aafaf76c28" containerName="revoke-keys" Oct 30 13:12:50.767656 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:50.767501 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63616383-42a4-4a01-96c0-a7aafaf76c28" containerName="revoke-keys" Oct 30 13:12:50.767656 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:50.767574 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="63616383-42a4-4a01-96c0-a7aafaf76c28" containerName="revoke-keys" Oct 30 13:12:50.767656 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:50.767587 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="7751779d-b082-409a-a20f-ff0ff6263808" containerName="main" Oct 30 13:12:50.767656 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:50.767596 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="35619ab9-cbeb-45da-8506-3af63a273177" containerName="main" Oct 30 13:12:50.767656 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:50.767620 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eac003e-b8a0-4b88-a296-59e5a9c85a05" containerName="revoke-keys" Oct 30 13:12:50.772750 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:50.772730 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f" Oct 30 13:12:50.810198 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:50.810163 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f"] Oct 30 13:12:50.814756 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:50.814732 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-bb71e2-59f79f4f99-xh9lj"] Oct 30 13:12:50.871040 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:50.871010 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19779cce-d1a1-429b-9ece-85f9932374c5-istio-envoy\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f\" (UID: \"19779cce-d1a1-429b-9ece-85f9932374c5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f" Oct 30 13:12:50.871223 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:50.871148 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19779cce-d1a1-429b-9ece-85f9932374c5-istiod-ca-cert\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f\" (UID: \"19779cce-d1a1-429b-9ece-85f9932374c5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f" Oct 30 13:12:50.871340 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:50.871218 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19779cce-d1a1-429b-9ece-85f9932374c5-istio-data\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f\" (UID: \"19779cce-d1a1-429b-9ece-85f9932374c5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f" Oct 30 13:12:50.871340 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:50.871261 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19779cce-d1a1-429b-9ece-85f9932374c5-istio-token\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f\" (UID: \"19779cce-d1a1-429b-9ece-85f9932374c5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f" Oct 30 13:12:50.871340 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:50.871293 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2tcp\" (UniqueName: \"kubernetes.io/projected/19779cce-d1a1-429b-9ece-85f9932374c5-kube-api-access-z2tcp\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f\" (UID: \"19779cce-d1a1-429b-9ece-85f9932374c5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f" Oct 30 13:12:50.871478 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:50.871363 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19779cce-d1a1-429b-9ece-85f9932374c5-workload-certs\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f\" (UID: \"19779cce-d1a1-429b-9ece-85f9932374c5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f" Oct 30 13:12:50.871478 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:50.871407 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19779cce-d1a1-429b-9ece-85f9932374c5-workload-socket\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f\" (UID: \"19779cce-d1a1-429b-9ece-85f9932374c5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f" Oct 30 13:12:50.871478 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:50.871432 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19779cce-d1a1-429b-9ece-85f9932374c5-istio-podinfo\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f\" (UID: \"19779cce-d1a1-429b-9ece-85f9932374c5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f" Oct 30 13:12:50.871578 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:50.871485 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19779cce-d1a1-429b-9ece-85f9932374c5-credential-socket\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f\" (UID: \"19779cce-d1a1-429b-9ece-85f9932374c5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f" Oct 30 13:12:50.973101 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:50.973054 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19779cce-d1a1-429b-9ece-85f9932374c5-workload-certs\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f\" (UID: \"19779cce-d1a1-429b-9ece-85f9932374c5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f" Oct 30 13:12:50.973297 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:50.973122 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19779cce-d1a1-429b-9ece-85f9932374c5-workload-socket\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f\" (UID: \"19779cce-d1a1-429b-9ece-85f9932374c5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f" Oct 30 13:12:50.973297 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:50.973148 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19779cce-d1a1-429b-9ece-85f9932374c5-istio-podinfo\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f\" (UID: \"19779cce-d1a1-429b-9ece-85f9932374c5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f" Oct 30 13:12:50.973297 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:50.973192 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19779cce-d1a1-429b-9ece-85f9932374c5-credential-socket\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f\" (UID: \"19779cce-d1a1-429b-9ece-85f9932374c5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f" Oct 30 13:12:50.973297 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:50.973275 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19779cce-d1a1-429b-9ece-85f9932374c5-istio-envoy\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f\" (UID: \"19779cce-d1a1-429b-9ece-85f9932374c5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f" Oct 30 13:12:50.973623 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:50.973371 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19779cce-d1a1-429b-9ece-85f9932374c5-istiod-ca-cert\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f\" (UID: \"19779cce-d1a1-429b-9ece-85f9932374c5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f" Oct 30 13:12:50.973623 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:50.973407 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19779cce-d1a1-429b-9ece-85f9932374c5-istio-data\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f\" (UID: \"19779cce-d1a1-429b-9ece-85f9932374c5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f" Oct 30 13:12:50.973623 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:50.973438 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19779cce-d1a1-429b-9ece-85f9932374c5-istio-token\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f\" (UID: \"19779cce-d1a1-429b-9ece-85f9932374c5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f" Oct 30 13:12:50.973623 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:50.973464 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2tcp\" (UniqueName: \"kubernetes.io/projected/19779cce-d1a1-429b-9ece-85f9932374c5-kube-api-access-z2tcp\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f\" (UID: \"19779cce-d1a1-429b-9ece-85f9932374c5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f" Oct 30 13:12:50.973623 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:50.973518 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19779cce-d1a1-429b-9ece-85f9932374c5-workload-certs\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f\" (UID: \"19779cce-d1a1-429b-9ece-85f9932374c5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f" Oct 30 13:12:50.973946 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:50.973922 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19779cce-d1a1-429b-9ece-85f9932374c5-workload-socket\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f\" (UID: \"19779cce-d1a1-429b-9ece-85f9932374c5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f" Oct 30 13:12:50.974081 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:50.974059 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19779cce-d1a1-429b-9ece-85f9932374c5-credential-socket\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f\" (UID: \"19779cce-d1a1-429b-9ece-85f9932374c5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f" Oct 30 13:12:50.974538 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:50.974512 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19779cce-d1a1-429b-9ece-85f9932374c5-istio-data\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f\" (UID: \"19779cce-d1a1-429b-9ece-85f9932374c5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f" Oct 30 13:12:50.974943 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:50.974921 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19779cce-d1a1-429b-9ece-85f9932374c5-istiod-ca-cert\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f\" (UID: \"19779cce-d1a1-429b-9ece-85f9932374c5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f" Oct 30 13:12:50.976337 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:50.976310 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19779cce-d1a1-429b-9ece-85f9932374c5-istio-envoy\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f\" (UID: \"19779cce-d1a1-429b-9ece-85f9932374c5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f" Oct 30 13:12:50.976427 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:50.976314 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19779cce-d1a1-429b-9ece-85f9932374c5-istio-podinfo\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f\" (UID: \"19779cce-d1a1-429b-9ece-85f9932374c5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f" Oct 30 13:12:51.000282 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:51.000257 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19779cce-d1a1-429b-9ece-85f9932374c5-istio-token\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f\" (UID: \"19779cce-d1a1-429b-9ece-85f9932374c5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f" Oct 30 13:12:51.004558 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:51.004539 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2tcp\" (UniqueName: \"kubernetes.io/projected/19779cce-d1a1-429b-9ece-85f9932374c5-kube-api-access-z2tcp\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f\" (UID: \"19779cce-d1a1-429b-9ece-85f9932374c5\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f" Oct 30 13:12:51.063855 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:51.063787 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:51.087563 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:51.087541 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f" Oct 30 13:12:51.262647 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:51.262577 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:51.422199 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:51.422167 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f"] Oct 30 13:12:51.423547 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:12:51.423518 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19779cce_d1a1_429b_9ece_85f9932374c5.slice/crio-0dfaf7ba784492ba1474c7f389ca11f95f9adbeb6bd5cdc6325631deaa1ad464 WatchSource:0}: Error finding container 0dfaf7ba784492ba1474c7f389ca11f95f9adbeb6bd5cdc6325631deaa1ad464: Status 404 returned error can't find the container with id 0dfaf7ba784492ba1474c7f389ca11f95f9adbeb6bd5cdc6325631deaa1ad464 Oct 30 13:12:51.425902 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:51.425869 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:12:51.425960 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:51.425939 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:12:51.426002 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:51.425972 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:12:51.592099 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:51.592067 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:51.792443 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:51.792400 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f" event={"ID":"19779cce-d1a1-429b-9ece-85f9932374c5","Type":"ContainerStarted","Data":"3732aeaed83dcf345f3bacac79c90ea328eff4f4fc5652c8dd1bfe3f8257b377"} Oct 30 13:12:51.792870 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:51.792498 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f" event={"ID":"19779cce-d1a1-429b-9ece-85f9932374c5","Type":"ContainerStarted","Data":"0dfaf7ba784492ba1474c7f389ca11f95f9adbeb6bd5cdc6325631deaa1ad464"} Oct 30 13:12:51.843272 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:51.843218 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f" podStartSLOduration=1.84320456 podStartE2EDuration="1.84320456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:51.838511757 +0000 UTC m=+1228.307927515" watchObservedRunningTime="2026-10-30 13:12:51.84320456 +0000 UTC m=+1228.312620305" Oct 30 13:12:52.067681 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:52.056801 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6787f73-9vvt2"] Oct 30 13:12:52.067681 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:52.062278 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6787f73-9vvt2" Oct 30 13:12:52.087035 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:52.084748 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:12:52.087035 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:52.084928 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:12:52.087035 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:52.085034 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:12:52.087035 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:52.085634 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:12:52.095980 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:52.095948 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b126f0c-889e-4485-a4af-4ee23b88ab57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6787f73-9vvt2\" (UID: \"8b126f0c-889e-4485-a4af-4ee23b88ab57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6787f73-9vvt2" Oct 30 13:12:52.096235 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:52.096215 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b126f0c-889e-4485-a4af-4ee23b88ab57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6787f73-9vvt2\" (UID: \"8b126f0c-889e-4485-a4af-4ee23b88ab57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6787f73-9vvt2" Oct 30 13:12:52.096344 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:52.096330 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlbs2\" (UniqueName: \"kubernetes.io/projected/8b126f0c-889e-4485-a4af-4ee23b88ab57-kube-api-access-vlbs2\") pod \"maas-api-revoke-sub-a6787f73-9vvt2\" (UID: \"8b126f0c-889e-4485-a4af-4ee23b88ab57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6787f73-9vvt2" Oct 30 13:12:52.098731 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:52.098708 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f" Oct 30 13:12:52.108154 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:52.107718 5922 patch_prober.go:29] interesting pod/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 13:12:52.108364 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:52.108340 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f" podUID="19779cce-d1a1-429b-9ece-85f9932374c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 13:12:52.109442 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:52.109414 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6787f73-9vvt2"] Oct 30 13:12:52.197333 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:52.197285 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b126f0c-889e-4485-a4af-4ee23b88ab57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6787f73-9vvt2\" (UID: \"8b126f0c-889e-4485-a4af-4ee23b88ab57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6787f73-9vvt2" Oct 30 13:12:52.197520 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:52.197338 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlbs2\" (UniqueName: \"kubernetes.io/projected/8b126f0c-889e-4485-a4af-4ee23b88ab57-kube-api-access-vlbs2\") pod \"maas-api-revoke-sub-a6787f73-9vvt2\" (UID: \"8b126f0c-889e-4485-a4af-4ee23b88ab57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6787f73-9vvt2" Oct 30 13:12:52.197571 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:52.197529 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b126f0c-889e-4485-a4af-4ee23b88ab57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6787f73-9vvt2\" (UID: \"8b126f0c-889e-4485-a4af-4ee23b88ab57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6787f73-9vvt2" Oct 30 13:12:52.198906 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:52.198876 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b126f0c-889e-4485-a4af-4ee23b88ab57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6787f73-9vvt2\" (UID: \"8b126f0c-889e-4485-a4af-4ee23b88ab57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6787f73-9vvt2" Oct 30 13:12:52.201003 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:52.200957 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b126f0c-889e-4485-a4af-4ee23b88ab57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6787f73-9vvt2\" (UID: \"8b126f0c-889e-4485-a4af-4ee23b88ab57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6787f73-9vvt2" Oct 30 13:12:52.222204 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:52.222178 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlbs2\" (UniqueName: \"kubernetes.io/projected/8b126f0c-889e-4485-a4af-4ee23b88ab57-kube-api-access-vlbs2\") pod \"maas-api-revoke-sub-a6787f73-9vvt2\" (UID: \"8b126f0c-889e-4485-a4af-4ee23b88ab57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6787f73-9vvt2" Oct 30 13:12:52.288018 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:52.287957 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:52.396099 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:52.396009 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6787f73-9vvt2" Oct 30 13:12:52.587493 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:52.587438 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:52.599649 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:52.599595 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-54994c-7mkl6"] Oct 30 13:12:52.604533 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:52.604505 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-54994c-7mkl6"] Oct 30 13:12:52.703692 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:52.703661 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6787f73-9vvt2"] Oct 30 13:12:52.703977 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:12:52.703955 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b126f0c_889e_4485_a4af_4ee23b88ab57.slice/crio-6308fc2b3dc2e0ee2a8bc4db15904843572689f53eeb08fe273a7dbf4b278a6f WatchSource:0}: Error finding container 6308fc2b3dc2e0ee2a8bc4db15904843572689f53eeb08fe273a7dbf4b278a6f: Status 404 returned error can't find the container with id 6308fc2b3dc2e0ee2a8bc4db15904843572689f53eeb08fe273a7dbf4b278a6f Oct 30 13:12:52.743571 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:52.743545 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:52.815301 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:52.815247 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6787f73-9vvt2" event={"ID":"8b126f0c-889e-4485-a4af-4ee23b88ab57","Type":"ContainerStarted","Data":"6308fc2b3dc2e0ee2a8bc4db15904843572689f53eeb08fe273a7dbf4b278a6f"} Oct 30 13:12:53.087306 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:53.087263 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:53.088248 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:53.088224 5922 patch_prober.go:29] interesting pod/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 13:12:53.088346 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:53.088274 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f" podUID="19779cce-d1a1-429b-9ece-85f9932374c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 13:12:53.554029 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:12:53.553994 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:12:53.644755 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:53.644725 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="856abf62-76be-4153-ac58-97236322f107" path="/var/lib/kubelet/pods/856abf62-76be-4153-ac58-97236322f107/volumes" Oct 30 13:12:53.825917 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:53.825831 5922 generic.go:356] "Generic (PLEG): container finished" podID="8b126f0c-889e-4485-a4af-4ee23b88ab57" containerID="3ddc81c56d742dc0b5ff31f97dcff7f3cddbd1da88d900d17dff91f50db01b32" exitCode=0 Oct 30 13:12:53.826293 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:53.825924 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6787f73-9vvt2" event={"ID":"8b126f0c-889e-4485-a4af-4ee23b88ab57","Type":"ContainerDied","Data":"3ddc81c56d742dc0b5ff31f97dcff7f3cddbd1da88d900d17dff91f50db01b32"} Oct 30 13:12:54.088120 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:54.088032 5922 patch_prober.go:29] interesting pod/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 13:12:54.088120 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:54.088090 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f" podUID="19779cce-d1a1-429b-9ece-85f9932374c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 13:12:54.337301 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:54.337266 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-54994c-75d97544d7-lvpzs" podUID="ac077401-1be7-4461-a1e5-bc4f186c9a94" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:12:54.337491 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:54.337446 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-54994c-f69fc5d86-68fv5" podUID="727a0820-c1f2-469e-827d-3f623e9df963" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:12:54.597229 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:54.597194 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:54.733877 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:54.733837 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:54.860101 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:54.860013 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:55.041833 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:55.041371 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f"] Oct 30 13:12:55.042157 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:55.042124 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f" podUID="19779cce-d1a1-429b-9ece-85f9932374c5" containerName="istio-proxy" containerID="cri-o://3732aeaed83dcf345f3bacac79c90ea328eff4f4fc5652c8dd1bfe3f8257b377" gracePeriod=30 Oct 30 13:12:55.048402 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:55.048376 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6787f73-9vvt2" Oct 30 13:12:55.058435 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:55.058384 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd"] Oct 30 13:12:55.059676 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:55.058657 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" podUID="ef454a96-712b-4ece-b4d8-b116964867a5" containerName="istio-proxy" containerID="cri-o://eb6bae3ed9ff47988290782ac4816999a680baf832d88b2a74e4563e5c9ac15c" gracePeriod=30 Oct 30 13:12:55.099035 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:55.098962 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec450da7-jgsxh"] Oct 30 13:12:55.100830 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:55.100802 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:55.106126 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:55.105699 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f38a233c-nglhq"] Oct 30 13:12:55.110865 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:55.110768 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec450da7-jgsxh"] Oct 30 13:12:55.124721 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:55.124691 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f38a233c-nglhq"] Oct 30 13:12:55.136033 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:55.135918 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b126f0c-889e-4485-a4af-4ee23b88ab57-kube-api-access-vlbs2\" (UniqueName: \"kubernetes.io/projected/8b126f0c-889e-4485-a4af-4ee23b88ab57-kube-api-access-vlbs2\") pod \"8b126f0c-889e-4485-a4af-4ee23b88ab57\" (UID: \"8b126f0c-889e-4485-a4af-4ee23b88ab57\") " Oct 30 13:12:55.136344 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:55.136274 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b126f0c-889e-4485-a4af-4ee23b88ab57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b126f0c-889e-4485-a4af-4ee23b88ab57-maas-api-xks-ca\") pod \"8b126f0c-889e-4485-a4af-4ee23b88ab57\" (UID: \"8b126f0c-889e-4485-a4af-4ee23b88ab57\") " Oct 30 13:12:55.136632 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:55.136584 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b126f0c-889e-4485-a4af-4ee23b88ab57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b126f0c-889e-4485-a4af-4ee23b88ab57-maas-api-openshift-ca\") pod \"8b126f0c-889e-4485-a4af-4ee23b88ab57\" (UID: \"8b126f0c-889e-4485-a4af-4ee23b88ab57\") " Oct 30 13:12:55.137992 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:55.137897 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b126f0c-889e-4485-a4af-4ee23b88ab57-maas-api-openshift-ca" pod "8b126f0c-889e-4485-a4af-4ee23b88ab57" (UID: "8b126f0c-889e-4485-a4af-4ee23b88ab57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:12:55.141906 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:55.141854 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b126f0c-889e-4485-a4af-4ee23b88ab57-maas-api-xks-ca" pod "8b126f0c-889e-4485-a4af-4ee23b88ab57" (UID: "8b126f0c-889e-4485-a4af-4ee23b88ab57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:12:55.142033 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:55.141907 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b126f0c-889e-4485-a4af-4ee23b88ab57-kube-api-access-vlbs2" pod "8b126f0c-889e-4485-a4af-4ee23b88ab57" (UID: "8b126f0c-889e-4485-a4af-4ee23b88ab57"). InnerVolumeSpecName "kube-api-access-vlbs2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:55.297649 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:55.247563 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b126f0c-889e-4485-a4af-4ee23b88ab57-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:55.302454 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:55.302423 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlbs2\" (UniqueName: \"kubernetes.io/projected/8b126f0c-889e-4485-a4af-4ee23b88ab57-kube-api-access-vlbs2\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:55.302696 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:55.302678 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b126f0c-889e-4485-a4af-4ee23b88ab57-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:12:55.650596 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:55.650551 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bec1a3c3-11b9-4a4d-ab7a-721ced5436c1" path="/var/lib/kubelet/pods/bec1a3c3-11b9-4a4d-ab7a-721ced5436c1/volumes" Oct 30 13:12:55.651035 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:55.651019 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3cd207c-108f-4f32-9bed-fb30557b8ffa" path="/var/lib/kubelet/pods/e3cd207c-108f-4f32-9bed-fb30557b8ffa/volumes" Oct 30 13:12:55.815679 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:55.815637 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bb71e2-59f79f4f99-xh9lj" podUID="72a0ed5e-1d31-4695-9dc5-92a304c0dc69" containerName="payload-pre-processing" containerID="cri-o://c39f7418cd4472879b43de64fe85b261a2cdd81d409b658f68edd9650c50375f" gracePeriod=25 Oct 30 13:12:55.847481 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:55.847450 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6787f73-9vvt2" event={"ID":"8b126f0c-889e-4485-a4af-4ee23b88ab57","Type":"ContainerDied","Data":"6308fc2b3dc2e0ee2a8bc4db15904843572689f53eeb08fe273a7dbf4b278a6f"} Oct 30 13:12:55.847633 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:55.847497 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6308fc2b3dc2e0ee2a8bc4db15904843572689f53eeb08fe273a7dbf4b278a6f" Oct 30 13:12:55.847633 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:55.847502 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6787f73-9vvt2" Oct 30 13:12:56.648650 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:56.645050 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:57.737582 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:57.737538 5922 patch_prober.go:29] interesting pod/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:12:57.738102 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:57.737598 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" podUID="ef454a96-712b-4ece-b4d8-b116964867a5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:12:58.063934 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:58.063851 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af2d3dcf-jn6ph"] Oct 30 13:12:58.075881 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:58.075853 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af2d3dcf-jn6ph"] Oct 30 13:12:58.816857 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:58.816822 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:59.644226 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:12:59.644196 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2a01a75-f5d3-4ef1-8215-1f72d13743df" path="/var/lib/kubelet/pods/f2a01a75-f5d3-4ef1-8215-1f72d13743df/volumes" Oct 30 13:13:00.119332 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.119296 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" podUID="a85de36b-f8c8-4ade-a468-c114080cb312" containerName="main" containerID="cri-o://b814923153b3fa16975c2e0f656cf351391cfd1f88e77b0e2c6fe56e40626e67" gracePeriod=45 Oct 30 13:13:00.306567 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.306533 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bb71e2-59f79f4f99-xh9lj" podUID="72a0ed5e-1d31-4695-9dc5-92a304c0dc69" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:13:00.407177 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.407120 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" Oct 30 13:13:00.427101 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.427080 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" Oct 30 13:13:00.497167 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.497141 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef454a96-712b-4ece-b4d8-b116964867a5-kube-api-access-2wm6z\" (UniqueName: \"kubernetes.io/projected/ef454a96-712b-4ece-b4d8-b116964867a5-kube-api-access-2wm6z\") pod \"ef454a96-712b-4ece-b4d8-b116964867a5\" (UID: \"ef454a96-712b-4ece-b4d8-b116964867a5\") " Oct 30 13:13:00.497326 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.497196 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef454a96-712b-4ece-b4d8-b116964867a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef454a96-712b-4ece-b4d8-b116964867a5-istio-data\") pod \"ef454a96-712b-4ece-b4d8-b116964867a5\" (UID: \"ef454a96-712b-4ece-b4d8-b116964867a5\") " Oct 30 13:13:00.497326 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.497216 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef454a96-712b-4ece-b4d8-b116964867a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef454a96-712b-4ece-b4d8-b116964867a5-workload-certs\") pod \"ef454a96-712b-4ece-b4d8-b116964867a5\" (UID: \"ef454a96-712b-4ece-b4d8-b116964867a5\") " Oct 30 13:13:00.497326 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.497240 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef454a96-712b-4ece-b4d8-b116964867a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef454a96-712b-4ece-b4d8-b116964867a5-istio-podinfo\") pod \"ef454a96-712b-4ece-b4d8-b116964867a5\" (UID: \"ef454a96-712b-4ece-b4d8-b116964867a5\") " Oct 30 13:13:00.497326 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.497267 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a85de36b-f8c8-4ade-a468-c114080cb312-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a85de36b-f8c8-4ade-a468-c114080cb312-tmp-dir\") pod \"a85de36b-f8c8-4ade-a468-c114080cb312\" (UID: \"a85de36b-f8c8-4ade-a468-c114080cb312\") " Oct 30 13:13:00.497326 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.497289 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a85de36b-f8c8-4ade-a468-c114080cb312-home\" (UniqueName: \"kubernetes.io/empty-dir/a85de36b-f8c8-4ade-a468-c114080cb312-home\") pod \"a85de36b-f8c8-4ade-a468-c114080cb312\" (UID: \"a85de36b-f8c8-4ade-a468-c114080cb312\") " Oct 30 13:13:00.497605 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.497332 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a85de36b-f8c8-4ade-a468-c114080cb312-kube-api-access-27rdg\" (UniqueName: \"kubernetes.io/projected/a85de36b-f8c8-4ade-a468-c114080cb312-kube-api-access-27rdg\") pod \"a85de36b-f8c8-4ade-a468-c114080cb312\" (UID: \"a85de36b-f8c8-4ade-a468-c114080cb312\") " Oct 30 13:13:00.497605 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.497374 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a85de36b-f8c8-4ade-a468-c114080cb312-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a85de36b-f8c8-4ade-a468-c114080cb312-dshm\") pod \"a85de36b-f8c8-4ade-a468-c114080cb312\" (UID: \"a85de36b-f8c8-4ade-a468-c114080cb312\") " Oct 30 13:13:00.497605 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.497402 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef454a96-712b-4ece-b4d8-b116964867a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef454a96-712b-4ece-b4d8-b116964867a5-istiod-ca-cert\") pod \"ef454a96-712b-4ece-b4d8-b116964867a5\" (UID: \"ef454a96-712b-4ece-b4d8-b116964867a5\") " Oct 30 13:13:00.497605 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.497431 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef454a96-712b-4ece-b4d8-b116964867a5-istio-token\" (UniqueName: \"kubernetes.io/projected/ef454a96-712b-4ece-b4d8-b116964867a5-istio-token\") pod \"ef454a96-712b-4ece-b4d8-b116964867a5\" (UID: \"ef454a96-712b-4ece-b4d8-b116964867a5\") " Oct 30 13:13:00.497605 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.497496 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef454a96-712b-4ece-b4d8-b116964867a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef454a96-712b-4ece-b4d8-b116964867a5-workload-socket\") pod \"ef454a96-712b-4ece-b4d8-b116964867a5\" (UID: \"ef454a96-712b-4ece-b4d8-b116964867a5\") " Oct 30 13:13:00.497605 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.497552 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a85de36b-f8c8-4ade-a468-c114080cb312-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a85de36b-f8c8-4ade-a468-c114080cb312-model-cache\") pod \"a85de36b-f8c8-4ade-a468-c114080cb312\" (UID: \"a85de36b-f8c8-4ade-a468-c114080cb312\") " Oct 30 13:13:00.497605 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.497566 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef454a96-712b-4ece-b4d8-b116964867a5-workload-certs" pod "ef454a96-712b-4ece-b4d8-b116964867a5" (UID: "ef454a96-712b-4ece-b4d8-b116964867a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:00.497605 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.497582 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a85de36b-f8c8-4ade-a468-c114080cb312-tls-certs\" (UniqueName: \"kubernetes.io/secret/a85de36b-f8c8-4ade-a468-c114080cb312-tls-certs\") pod \"a85de36b-f8c8-4ade-a468-c114080cb312\" (UID: \"a85de36b-f8c8-4ade-a468-c114080cb312\") " Oct 30 13:13:00.497605 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.497580 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef454a96-712b-4ece-b4d8-b116964867a5-istio-data" pod "ef454a96-712b-4ece-b4d8-b116964867a5" (UID: "ef454a96-712b-4ece-b4d8-b116964867a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:00.498087 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.497669 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef454a96-712b-4ece-b4d8-b116964867a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef454a96-712b-4ece-b4d8-b116964867a5-credential-socket\") pod \"ef454a96-712b-4ece-b4d8-b116964867a5\" (UID: \"ef454a96-712b-4ece-b4d8-b116964867a5\") " Oct 30 13:13:00.498087 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.497797 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef454a96-712b-4ece-b4d8-b116964867a5-workload-socket" pod "ef454a96-712b-4ece-b4d8-b116964867a5" (UID: "ef454a96-712b-4ece-b4d8-b116964867a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:00.498087 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.497864 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef454a96-712b-4ece-b4d8-b116964867a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef454a96-712b-4ece-b4d8-b116964867a5-istio-envoy\") pod \"ef454a96-712b-4ece-b4d8-b116964867a5\" (UID: \"ef454a96-712b-4ece-b4d8-b116964867a5\") " Oct 30 13:13:00.498260 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.498229 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a85de36b-f8c8-4ade-a468-c114080cb312-model-cache" pod "a85de36b-f8c8-4ade-a468-c114080cb312" (UID: "a85de36b-f8c8-4ade-a468-c114080cb312"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:00.498381 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.498361 5922 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a85de36b-f8c8-4ade-a468-c114080cb312-model-cache\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.498448 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.498386 5922 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef454a96-712b-4ece-b4d8-b116964867a5-istio-data\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.498448 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.498400 5922 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef454a96-712b-4ece-b4d8-b116964867a5-workload-certs\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.498448 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.498413 5922 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef454a96-712b-4ece-b4d8-b116964867a5-workload-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.499739 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.498752 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a85de36b-f8c8-4ade-a468-c114080cb312-home" pod "a85de36b-f8c8-4ade-a468-c114080cb312" (UID: "a85de36b-f8c8-4ade-a468-c114080cb312"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:00.499739 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.499140 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a85de36b-f8c8-4ade-a468-c114080cb312-tmp-dir" pod "a85de36b-f8c8-4ade-a468-c114080cb312" (UID: "a85de36b-f8c8-4ade-a468-c114080cb312"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:00.499739 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.499325 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef454a96-712b-4ece-b4d8-b116964867a5-credential-socket" pod "ef454a96-712b-4ece-b4d8-b116964867a5" (UID: "ef454a96-712b-4ece-b4d8-b116964867a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:00.499739 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.499678 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef454a96-712b-4ece-b4d8-b116964867a5-istiod-ca-cert" pod "ef454a96-712b-4ece-b4d8-b116964867a5" (UID: "ef454a96-712b-4ece-b4d8-b116964867a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:13:00.500040 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.499876 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef454a96-712b-4ece-b4d8-b116964867a5-kube-api-access-2wm6z" pod "ef454a96-712b-4ece-b4d8-b116964867a5" (UID: "ef454a96-712b-4ece-b4d8-b116964867a5"). InnerVolumeSpecName "kube-api-access-2wm6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:00.500560 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.500535 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a85de36b-f8c8-4ade-a468-c114080cb312-kube-api-access-27rdg" pod "a85de36b-f8c8-4ade-a468-c114080cb312" (UID: "a85de36b-f8c8-4ade-a468-c114080cb312"). InnerVolumeSpecName "kube-api-access-27rdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:00.500681 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.500601 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef454a96-712b-4ece-b4d8-b116964867a5-istio-envoy" pod "ef454a96-712b-4ece-b4d8-b116964867a5" (UID: "ef454a96-712b-4ece-b4d8-b116964867a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:00.502127 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.502105 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef454a96-712b-4ece-b4d8-b116964867a5-istio-podinfo" pod "ef454a96-712b-4ece-b4d8-b116964867a5" (UID: "ef454a96-712b-4ece-b4d8-b116964867a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:13:00.502454 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.502433 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef454a96-712b-4ece-b4d8-b116964867a5-istio-token" pod "ef454a96-712b-4ece-b4d8-b116964867a5" (UID: "ef454a96-712b-4ece-b4d8-b116964867a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:00.502454 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.502435 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a85de36b-f8c8-4ade-a468-c114080cb312-tls-certs" pod "a85de36b-f8c8-4ade-a468-c114080cb312" (UID: "a85de36b-f8c8-4ade-a468-c114080cb312"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:13:00.503597 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.503574 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a85de36b-f8c8-4ade-a468-c114080cb312-dshm" pod "a85de36b-f8c8-4ade-a468-c114080cb312" (UID: "a85de36b-f8c8-4ade-a468-c114080cb312"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:00.600096 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.600066 5922 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef454a96-712b-4ece-b4d8-b116964867a5-istio-podinfo\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.600096 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.600093 5922 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a85de36b-f8c8-4ade-a468-c114080cb312-tmp-dir\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.600096 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.600102 5922 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a85de36b-f8c8-4ade-a468-c114080cb312-home\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.600314 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.600110 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27rdg\" (UniqueName: \"kubernetes.io/projected/a85de36b-f8c8-4ade-a468-c114080cb312-kube-api-access-27rdg\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.600314 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.600119 5922 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a85de36b-f8c8-4ade-a468-c114080cb312-dshm\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.600314 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.600127 5922 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef454a96-712b-4ece-b4d8-b116964867a5-istiod-ca-cert\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.600314 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.600135 5922 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef454a96-712b-4ece-b4d8-b116964867a5-istio-token\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.600314 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.600143 5922 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a85de36b-f8c8-4ade-a468-c114080cb312-tls-certs\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.600314 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.600162 5922 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef454a96-712b-4ece-b4d8-b116964867a5-credential-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.600314 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.600172 5922 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef454a96-712b-4ece-b4d8-b116964867a5-istio-envoy\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.600314 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.600180 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wm6z\" (UniqueName: \"kubernetes.io/projected/ef454a96-712b-4ece-b4d8-b116964867a5-kube-api-access-2wm6z\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.669922 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.669818 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:00.923993 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.923909 5922 generic.go:356] "Generic (PLEG): container finished" podID="a85de36b-f8c8-4ade-a468-c114080cb312" containerID="b814923153b3fa16975c2e0f656cf351391cfd1f88e77b0e2c6fe56e40626e67" exitCode=0 Oct 30 13:13:00.923993 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.923952 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" Oct 30 13:13:00.924218 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.923991 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" event={"ID":"a85de36b-f8c8-4ade-a468-c114080cb312","Type":"ContainerDied","Data":"b814923153b3fa16975c2e0f656cf351391cfd1f88e77b0e2c6fe56e40626e67"} Oct 30 13:13:00.924218 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.924031 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg" event={"ID":"a85de36b-f8c8-4ade-a468-c114080cb312","Type":"ContainerDied","Data":"13000c8adfdc1ca90c644548d10c381c9080dd4aa972ea8dd0d70c6e92a0987c"} Oct 30 13:13:00.924218 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.924049 5922 scope.go:122] "RemoveContainer" containerID="b814923153b3fa16975c2e0f656cf351391cfd1f88e77b0e2c6fe56e40626e67" Oct 30 13:13:00.925846 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.925821 5922 generic.go:356] "Generic (PLEG): container finished" podID="ef454a96-712b-4ece-b4d8-b116964867a5" containerID="eb6bae3ed9ff47988290782ac4816999a680baf832d88b2a74e4563e5c9ac15c" exitCode=0 Oct 30 13:13:00.925960 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.925865 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" Oct 30 13:13:00.925960 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.925913 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" event={"ID":"ef454a96-712b-4ece-b4d8-b116964867a5","Type":"ContainerDied","Data":"eb6bae3ed9ff47988290782ac4816999a680baf832d88b2a74e4563e5c9ac15c"} Oct 30 13:13:00.926053 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.925967 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd" event={"ID":"ef454a96-712b-4ece-b4d8-b116964867a5","Type":"ContainerDied","Data":"e06aa1d1fc90d682469277a6deede54cd10406feb35cbb1870643a20c52ea5d5"} Oct 30 13:13:00.937455 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.937439 5922 scope.go:122] "RemoveContainer" containerID="b814923153b3fa16975c2e0f656cf351391cfd1f88e77b0e2c6fe56e40626e67" Oct 30 13:13:00.937719 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:13:00.937701 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b814923153b3fa16975c2e0f656cf351391cfd1f88e77b0e2c6fe56e40626e67\": container with ID starting with b814923153b3fa16975c2e0f656cf351391cfd1f88e77b0e2c6fe56e40626e67 not found: ID does not exist" containerID="b814923153b3fa16975c2e0f656cf351391cfd1f88e77b0e2c6fe56e40626e67" Oct 30 13:13:00.937803 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.937724 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b814923153b3fa16975c2e0f656cf351391cfd1f88e77b0e2c6fe56e40626e67"} err="failed to get container status \"b814923153b3fa16975c2e0f656cf351391cfd1f88e77b0e2c6fe56e40626e67\": rpc error: code = NotFound desc = could not find container \"b814923153b3fa16975c2e0f656cf351391cfd1f88e77b0e2c6fe56e40626e67\": container with ID starting with b814923153b3fa16975c2e0f656cf351391cfd1f88e77b0e2c6fe56e40626e67 not found: ID does not exist" Oct 30 13:13:00.937803 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.937739 5922 scope.go:122] "RemoveContainer" containerID="eb6bae3ed9ff47988290782ac4816999a680baf832d88b2a74e4563e5c9ac15c" Oct 30 13:13:00.951315 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.951294 5922 scope.go:122] "RemoveContainer" containerID="eb6bae3ed9ff47988290782ac4816999a680baf832d88b2a74e4563e5c9ac15c" Oct 30 13:13:00.951603 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:13:00.951586 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb6bae3ed9ff47988290782ac4816999a680baf832d88b2a74e4563e5c9ac15c\": container with ID starting with eb6bae3ed9ff47988290782ac4816999a680baf832d88b2a74e4563e5c9ac15c not found: ID does not exist" containerID="eb6bae3ed9ff47988290782ac4816999a680baf832d88b2a74e4563e5c9ac15c" Oct 30 13:13:00.951681 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.951608 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb6bae3ed9ff47988290782ac4816999a680baf832d88b2a74e4563e5c9ac15c"} err="failed to get container status \"eb6bae3ed9ff47988290782ac4816999a680baf832d88b2a74e4563e5c9ac15c\": rpc error: code = NotFound desc = could not find container \"eb6bae3ed9ff47988290782ac4816999a680baf832d88b2a74e4563e5c9ac15c\": container with ID starting with eb6bae3ed9ff47988290782ac4816999a680baf832d88b2a74e4563e5c9ac15c not found: ID does not exist" Oct 30 13:13:00.992848 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.992825 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg"] Oct 30 13:13:00.997545 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:00.997508 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-bb71e2/premium-w2-bb71e2-kserve-7b6c59b97c-b9ngg"] Oct 30 13:13:01.036348 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:01.036314 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd"] Oct 30 13:13:01.042275 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:01.042250 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-sv4sd"] Oct 30 13:13:01.644775 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:01.644739 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a85de36b-f8c8-4ade-a468-c114080cb312" path="/var/lib/kubelet/pods/a85de36b-f8c8-4ade-a468-c114080cb312/volumes" Oct 30 13:13:01.645212 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:01.645197 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef454a96-712b-4ece-b4d8-b116964867a5" path="/var/lib/kubelet/pods/ef454a96-712b-4ece-b4d8-b116964867a5/volumes" Oct 30 13:13:02.356599 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:02.356558 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:02.480060 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:02.480022 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:02.614043 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:02.613955 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:02.662321 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:02.662286 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd"] Oct 30 13:13:02.662754 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:02.662494 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" podUID="e5fb05c2-4a10-4c85-8464-75e247feae67" containerName="istio-proxy" containerID="cri-o://b0ed688211bcc67268db4556a64ca2a9883c7b63106739c7381c6df918864c9c" gracePeriod=30 Oct 30 13:13:02.674649 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:02.674606 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2"] Oct 30 13:13:02.674885 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:02.674860 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" podUID="5813c167-a1c1-49ca-b93c-50fb1ece0b64" containerName="istio-proxy" containerID="cri-o://d5cb6b7d0aca9cface57705be1dfc1fa8aeb2daa1aab7c6c6eded7b2f25e3080" gracePeriod=30 Oct 30 13:13:02.678595 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:02.678420 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk"] Oct 30 13:13:02.678747 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:02.678658 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" podUID="03433bd3-844d-42ca-b3b2-158f1a1f678d" containerName="istio-proxy" containerID="cri-o://2ed3debba84d1f6b67506d240bcdc230131bf906720e8debef4e93c8f2112afe" gracePeriod=30 Oct 30 13:13:02.772078 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:02.766596 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:02.951916 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:02.951885 5922 generic.go:356] "Generic (PLEG): container finished" podID="ac077401-1be7-4461-a1e5-bc4f186c9a94" containerID="fc5b24620981b6fad12282a9e22d5812e805032b934d1e30da4d192be0d4b6be" exitCode=0 Oct 30 13:13:02.952151 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:02.952070 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-54994c-75d97544d7-lvpzs" event={"ID":"ac077401-1be7-4461-a1e5-bc4f186c9a94","Type":"ContainerDied","Data":"fc5b24620981b6fad12282a9e22d5812e805032b934d1e30da4d192be0d4b6be"} Oct 30 13:13:02.954198 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:02.954173 5922 generic.go:356] "Generic (PLEG): container finished" podID="727a0820-c1f2-469e-827d-3f623e9df963" containerID="c316cebabc1a6f07a6e8b66c0b4e6cd81ef845b3954ca7962283e92872d1781a" exitCode=0 Oct 30 13:13:02.954339 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:02.954261 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-54994c-f69fc5d86-68fv5" event={"ID":"727a0820-c1f2-469e-827d-3f623e9df963","Type":"ContainerDied","Data":"c316cebabc1a6f07a6e8b66c0b4e6cd81ef845b3954ca7962283e92872d1781a"} Oct 30 13:13:03.004665 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:03.004607 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-54994c-f69fc5d86-68fv5" Oct 30 13:13:03.033847 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:03.033819 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/727a0820-c1f2-469e-827d-3f623e9df963-kube-api-access-7bxsg\" (UniqueName: \"kubernetes.io/projected/727a0820-c1f2-469e-827d-3f623e9df963-kube-api-access-7bxsg\") pod \"727a0820-c1f2-469e-827d-3f623e9df963\" (UID: \"727a0820-c1f2-469e-827d-3f623e9df963\") " Oct 30 13:13:03.034010 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:03.033910 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/727a0820-c1f2-469e-827d-3f623e9df963-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/727a0820-c1f2-469e-827d-3f623e9df963-plugins-config-volume\") pod \"727a0820-c1f2-469e-827d-3f623e9df963\" (UID: \"727a0820-c1f2-469e-827d-3f623e9df963\") " Oct 30 13:13:03.034397 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:03.034363 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/727a0820-c1f2-469e-827d-3f623e9df963-plugins-config-volume" pod "727a0820-c1f2-469e-827d-3f623e9df963" (UID: "727a0820-c1f2-469e-827d-3f623e9df963"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:13:03.035941 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:03.035918 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/727a0820-c1f2-469e-827d-3f623e9df963-kube-api-access-7bxsg" pod "727a0820-c1f2-469e-827d-3f623e9df963" (UID: "727a0820-c1f2-469e-827d-3f623e9df963"). InnerVolumeSpecName "kube-api-access-7bxsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:03.063756 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:03.063700 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-54994c-75d97544d7-lvpzs" Oct 30 13:13:03.137416 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:03.136864 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac077401-1be7-4461-a1e5-bc4f186c9a94-kube-api-access-mrzj8\" (UniqueName: \"kubernetes.io/projected/ac077401-1be7-4461-a1e5-bc4f186c9a94-kube-api-access-mrzj8\") pod \"ac077401-1be7-4461-a1e5-bc4f186c9a94\" (UID: \"ac077401-1be7-4461-a1e5-bc4f186c9a94\") " Oct 30 13:13:03.137416 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:03.136940 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac077401-1be7-4461-a1e5-bc4f186c9a94-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac077401-1be7-4461-a1e5-bc4f186c9a94-plugins-config-volume\") pod \"ac077401-1be7-4461-a1e5-bc4f186c9a94\" (UID: \"ac077401-1be7-4461-a1e5-bc4f186c9a94\") " Oct 30 13:13:03.137676 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:03.137516 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bxsg\" (UniqueName: \"kubernetes.io/projected/727a0820-c1f2-469e-827d-3f623e9df963-kube-api-access-7bxsg\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:03.137676 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:03.137536 5922 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/727a0820-c1f2-469e-827d-3f623e9df963-plugins-config-volume\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:03.140139 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:03.139852 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac077401-1be7-4461-a1e5-bc4f186c9a94-plugins-config-volume" pod "ac077401-1be7-4461-a1e5-bc4f186c9a94" (UID: "ac077401-1be7-4461-a1e5-bc4f186c9a94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:13:03.141693 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:03.141669 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac077401-1be7-4461-a1e5-bc4f186c9a94-kube-api-access-mrzj8" pod "ac077401-1be7-4461-a1e5-bc4f186c9a94" (UID: "ac077401-1be7-4461-a1e5-bc4f186c9a94"). InnerVolumeSpecName "kube-api-access-mrzj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:03.215855 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:03.215826 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:03.239171 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:03.239150 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrzj8\" (UniqueName: \"kubernetes.io/projected/ac077401-1be7-4461-a1e5-bc4f186c9a94-kube-api-access-mrzj8\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:03.239171 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:03.239171 5922 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac077401-1be7-4461-a1e5-bc4f186c9a94-plugins-config-volume\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:03.427185 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:03.427149 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:13:03.500260 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:03.500184 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:03.753526 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:03.753448 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:03.874200 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:03.874162 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:03.969461 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:03.969431 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-54994c-f69fc5d86-68fv5" Oct 30 13:13:03.969650 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:03.969438 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-54994c-f69fc5d86-68fv5" event={"ID":"727a0820-c1f2-469e-827d-3f623e9df963","Type":"ContainerDied","Data":"9f3fd069eaa512bfcba8a0b0e5aa09a3ca98917e4360ea1be9265d2211be760a"} Oct 30 13:13:03.969650 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:03.969574 5922 scope.go:122] "RemoveContainer" containerID="c316cebabc1a6f07a6e8b66c0b4e6cd81ef845b3954ca7962283e92872d1781a" Oct 30 13:13:03.971458 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:03.971439 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-54994c-75d97544d7-lvpzs" event={"ID":"ac077401-1be7-4461-a1e5-bc4f186c9a94","Type":"ContainerDied","Data":"5571a137a07f5b446beec8d0214346f54075c563083352fbdc497cc133c4d99d"} Oct 30 13:13:03.971538 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:03.971470 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-54994c-75d97544d7-lvpzs" Oct 30 13:13:03.989552 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:03.989532 5922 scope.go:122] "RemoveContainer" containerID="fc5b24620981b6fad12282a9e22d5812e805032b934d1e30da4d192be0d4b6be" Oct 30 13:13:04.029578 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:04.029508 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-54994c-75d97544d7-lvpzs"] Oct 30 13:13:04.042443 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:04.042412 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-54994c-75d97544d7-lvpzs"] Oct 30 13:13:04.085888 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:04.082985 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-54994c-f69fc5d86-68fv5"] Oct 30 13:13:04.089388 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:04.089355 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-54994c-f69fc5d86-68fv5"] Oct 30 13:13:04.137302 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:04.136085 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f" Oct 30 13:13:04.255231 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:04.255201 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19779cce-d1a1-429b-9ece-85f9932374c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19779cce-d1a1-429b-9ece-85f9932374c5-credential-socket\") pod \"19779cce-d1a1-429b-9ece-85f9932374c5\" (UID: \"19779cce-d1a1-429b-9ece-85f9932374c5\") " Oct 30 13:13:04.255404 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:04.255252 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19779cce-d1a1-429b-9ece-85f9932374c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19779cce-d1a1-429b-9ece-85f9932374c5-istio-data\") pod \"19779cce-d1a1-429b-9ece-85f9932374c5\" (UID: \"19779cce-d1a1-429b-9ece-85f9932374c5\") " Oct 30 13:13:04.255404 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:04.255285 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19779cce-d1a1-429b-9ece-85f9932374c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19779cce-d1a1-429b-9ece-85f9932374c5-istiod-ca-cert\") pod \"19779cce-d1a1-429b-9ece-85f9932374c5\" (UID: \"19779cce-d1a1-429b-9ece-85f9932374c5\") " Oct 30 13:13:04.255404 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:04.255358 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19779cce-d1a1-429b-9ece-85f9932374c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19779cce-d1a1-429b-9ece-85f9932374c5-istio-envoy\") pod \"19779cce-d1a1-429b-9ece-85f9932374c5\" (UID: \"19779cce-d1a1-429b-9ece-85f9932374c5\") " Oct 30 13:13:04.255404 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:04.255388 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19779cce-d1a1-429b-9ece-85f9932374c5-istio-token\" (UniqueName: \"kubernetes.io/projected/19779cce-d1a1-429b-9ece-85f9932374c5-istio-token\") pod \"19779cce-d1a1-429b-9ece-85f9932374c5\" (UID: \"19779cce-d1a1-429b-9ece-85f9932374c5\") " Oct 30 13:13:04.255607 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:04.255436 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19779cce-d1a1-429b-9ece-85f9932374c5-kube-api-access-z2tcp\" (UniqueName: \"kubernetes.io/projected/19779cce-d1a1-429b-9ece-85f9932374c5-kube-api-access-z2tcp\") pod \"19779cce-d1a1-429b-9ece-85f9932374c5\" (UID: \"19779cce-d1a1-429b-9ece-85f9932374c5\") " Oct 30 13:13:04.255607 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:04.255495 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19779cce-d1a1-429b-9ece-85f9932374c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19779cce-d1a1-429b-9ece-85f9932374c5-istio-podinfo\") pod \"19779cce-d1a1-429b-9ece-85f9932374c5\" (UID: \"19779cce-d1a1-429b-9ece-85f9932374c5\") " Oct 30 13:13:04.255607 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:04.255498 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19779cce-d1a1-429b-9ece-85f9932374c5-credential-socket" pod "19779cce-d1a1-429b-9ece-85f9932374c5" (UID: "19779cce-d1a1-429b-9ece-85f9932374c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:04.255607 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:04.255530 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19779cce-d1a1-429b-9ece-85f9932374c5-istio-data" pod "19779cce-d1a1-429b-9ece-85f9932374c5" (UID: "19779cce-d1a1-429b-9ece-85f9932374c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:04.255607 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:04.255565 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19779cce-d1a1-429b-9ece-85f9932374c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19779cce-d1a1-429b-9ece-85f9932374c5-workload-socket\") pod \"19779cce-d1a1-429b-9ece-85f9932374c5\" (UID: \"19779cce-d1a1-429b-9ece-85f9932374c5\") " Oct 30 13:13:04.255927 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:04.255660 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19779cce-d1a1-429b-9ece-85f9932374c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19779cce-d1a1-429b-9ece-85f9932374c5-workload-certs\") pod \"19779cce-d1a1-429b-9ece-85f9932374c5\" (UID: \"19779cce-d1a1-429b-9ece-85f9932374c5\") " Oct 30 13:13:04.255984 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:04.255841 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19779cce-d1a1-429b-9ece-85f9932374c5-istiod-ca-cert" pod "19779cce-d1a1-429b-9ece-85f9932374c5" (UID: "19779cce-d1a1-429b-9ece-85f9932374c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:13:04.255984 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:04.255923 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19779cce-d1a1-429b-9ece-85f9932374c5-workload-socket" pod "19779cce-d1a1-429b-9ece-85f9932374c5" (UID: "19779cce-d1a1-429b-9ece-85f9932374c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:04.256139 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:04.256116 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19779cce-d1a1-429b-9ece-85f9932374c5-workload-certs" pod "19779cce-d1a1-429b-9ece-85f9932374c5" (UID: "19779cce-d1a1-429b-9ece-85f9932374c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:04.256181 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:04.256138 5922 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19779cce-d1a1-429b-9ece-85f9932374c5-workload-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:04.256181 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:04.256161 5922 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19779cce-d1a1-429b-9ece-85f9932374c5-credential-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:04.256181 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:04.256177 5922 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19779cce-d1a1-429b-9ece-85f9932374c5-istio-data\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:04.256283 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:04.256192 5922 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19779cce-d1a1-429b-9ece-85f9932374c5-istiod-ca-cert\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:04.257791 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:04.257759 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19779cce-d1a1-429b-9ece-85f9932374c5-kube-api-access-z2tcp" pod "19779cce-d1a1-429b-9ece-85f9932374c5" (UID: "19779cce-d1a1-429b-9ece-85f9932374c5"). InnerVolumeSpecName "kube-api-access-z2tcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:04.258011 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:04.257987 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19779cce-d1a1-429b-9ece-85f9932374c5-istio-podinfo" pod "19779cce-d1a1-429b-9ece-85f9932374c5" (UID: "19779cce-d1a1-429b-9ece-85f9932374c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:13:04.258117 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:04.258096 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19779cce-d1a1-429b-9ece-85f9932374c5-istio-token" pod "19779cce-d1a1-429b-9ece-85f9932374c5" (UID: "19779cce-d1a1-429b-9ece-85f9932374c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:04.258184 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:04.258134 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19779cce-d1a1-429b-9ece-85f9932374c5-istio-envoy" pod "19779cce-d1a1-429b-9ece-85f9932374c5" (UID: "19779cce-d1a1-429b-9ece-85f9932374c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:04.357388 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:04.357321 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2tcp\" (UniqueName: \"kubernetes.io/projected/19779cce-d1a1-429b-9ece-85f9932374c5-kube-api-access-z2tcp\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:04.357388 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:04.357354 5922 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19779cce-d1a1-429b-9ece-85f9932374c5-istio-podinfo\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:04.357388 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:04.357363 5922 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19779cce-d1a1-429b-9ece-85f9932374c5-workload-certs\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:04.357388 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:04.357372 5922 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19779cce-d1a1-429b-9ece-85f9932374c5-istio-envoy\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:04.357388 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:04.357382 5922 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19779cce-d1a1-429b-9ece-85f9932374c5-istio-token\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:04.517036 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:04.517006 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:04.645761 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:04.645657 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:04.980382 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:04.980348 5922 generic.go:356] "Generic (PLEG): container finished" podID="19779cce-d1a1-429b-9ece-85f9932374c5" containerID="3732aeaed83dcf345f3bacac79c90ea328eff4f4fc5652c8dd1bfe3f8257b377" exitCode=0 Oct 30 13:13:04.980838 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:04.980398 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f" Oct 30 13:13:04.980838 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:04.980430 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f" event={"ID":"19779cce-d1a1-429b-9ece-85f9932374c5","Type":"ContainerDied","Data":"3732aeaed83dcf345f3bacac79c90ea328eff4f4fc5652c8dd1bfe3f8257b377"} Oct 30 13:13:04.980838 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:04.980481 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f" event={"ID":"19779cce-d1a1-429b-9ece-85f9932374c5","Type":"ContainerDied","Data":"0dfaf7ba784492ba1474c7f389ca11f95f9adbeb6bd5cdc6325631deaa1ad464"} Oct 30 13:13:04.980838 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:04.980504 5922 scope.go:122] "RemoveContainer" containerID="3732aeaed83dcf345f3bacac79c90ea328eff4f4fc5652c8dd1bfe3f8257b377" Oct 30 13:13:04.991531 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:04.991512 5922 scope.go:122] "RemoveContainer" containerID="3732aeaed83dcf345f3bacac79c90ea328eff4f4fc5652c8dd1bfe3f8257b377" Oct 30 13:13:04.991816 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:13:04.991796 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3732aeaed83dcf345f3bacac79c90ea328eff4f4fc5652c8dd1bfe3f8257b377\": container with ID starting with 3732aeaed83dcf345f3bacac79c90ea328eff4f4fc5652c8dd1bfe3f8257b377 not found: ID does not exist" containerID="3732aeaed83dcf345f3bacac79c90ea328eff4f4fc5652c8dd1bfe3f8257b377" Oct 30 13:13:04.991866 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:04.991825 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3732aeaed83dcf345f3bacac79c90ea328eff4f4fc5652c8dd1bfe3f8257b377"} err="failed to get container status \"3732aeaed83dcf345f3bacac79c90ea328eff4f4fc5652c8dd1bfe3f8257b377\": rpc error: code = NotFound desc = could not find container \"3732aeaed83dcf345f3bacac79c90ea328eff4f4fc5652c8dd1bfe3f8257b377\": container with ID starting with 3732aeaed83dcf345f3bacac79c90ea328eff4f4fc5652c8dd1bfe3f8257b377 not found: ID does not exist" Oct 30 13:13:05.068672 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:05.068649 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f"] Oct 30 13:13:05.080886 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:05.080857 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-xf59f"] Oct 30 13:13:05.644863 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:05.644832 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19779cce-d1a1-429b-9ece-85f9932374c5" path="/var/lib/kubelet/pods/19779cce-d1a1-429b-9ece-85f9932374c5/volumes" Oct 30 13:13:05.645234 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:05.645219 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="727a0820-c1f2-469e-827d-3f623e9df963" path="/var/lib/kubelet/pods/727a0820-c1f2-469e-827d-3f623e9df963/volumes" Oct 30 13:13:05.645564 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:05.645552 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac077401-1be7-4461-a1e5-bc4f186c9a94" path="/var/lib/kubelet/pods/ac077401-1be7-4461-a1e5-bc4f186c9a94/volumes" Oct 30 13:13:06.167997 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:06.167961 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:06.727968 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:06.727928 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:06.848886 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:06.848848 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:06.975324 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:06.975289 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:07.388626 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:07.388589 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:07.948114 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:07.948080 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" Oct 30 13:13:07.966415 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:07.966383 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" Oct 30 13:13:07.996200 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:07.996172 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5fb05c2-4a10-4c85-8464-75e247feae67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5fb05c2-4a10-4c85-8464-75e247feae67-istio-podinfo\") pod \"e5fb05c2-4a10-4c85-8464-75e247feae67\" (UID: \"e5fb05c2-4a10-4c85-8464-75e247feae67\") " Oct 30 13:13:07.996339 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:07.996215 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5fb05c2-4a10-4c85-8464-75e247feae67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5fb05c2-4a10-4c85-8464-75e247feae67-istiod-ca-cert\") pod \"e5fb05c2-4a10-4c85-8464-75e247feae67\" (UID: \"e5fb05c2-4a10-4c85-8464-75e247feae67\") " Oct 30 13:13:07.996339 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:07.996302 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5fb05c2-4a10-4c85-8464-75e247feae67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5fb05c2-4a10-4c85-8464-75e247feae67-istio-envoy\") pod \"e5fb05c2-4a10-4c85-8464-75e247feae67\" (UID: \"e5fb05c2-4a10-4c85-8464-75e247feae67\") " Oct 30 13:13:07.996457 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:07.996343 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5fb05c2-4a10-4c85-8464-75e247feae67-istio-token\" (UniqueName: \"kubernetes.io/projected/e5fb05c2-4a10-4c85-8464-75e247feae67-istio-token\") pod \"e5fb05c2-4a10-4c85-8464-75e247feae67\" (UID: \"e5fb05c2-4a10-4c85-8464-75e247feae67\") " Oct 30 13:13:07.996457 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:07.996397 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5fb05c2-4a10-4c85-8464-75e247feae67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5fb05c2-4a10-4c85-8464-75e247feae67-istio-data\") pod \"e5fb05c2-4a10-4c85-8464-75e247feae67\" (UID: \"e5fb05c2-4a10-4c85-8464-75e247feae67\") " Oct 30 13:13:07.996560 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:07.996457 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5fb05c2-4a10-4c85-8464-75e247feae67-kube-api-access-jgvxj\" (UniqueName: \"kubernetes.io/projected/e5fb05c2-4a10-4c85-8464-75e247feae67-kube-api-access-jgvxj\") pod \"e5fb05c2-4a10-4c85-8464-75e247feae67\" (UID: \"e5fb05c2-4a10-4c85-8464-75e247feae67\") " Oct 30 13:13:07.996560 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:07.996494 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5fb05c2-4a10-4c85-8464-75e247feae67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5fb05c2-4a10-4c85-8464-75e247feae67-credential-socket\") pod \"e5fb05c2-4a10-4c85-8464-75e247feae67\" (UID: \"e5fb05c2-4a10-4c85-8464-75e247feae67\") " Oct 30 13:13:07.996560 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:07.996529 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5fb05c2-4a10-4c85-8464-75e247feae67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5fb05c2-4a10-4c85-8464-75e247feae67-workload-certs\") pod \"e5fb05c2-4a10-4c85-8464-75e247feae67\" (UID: \"e5fb05c2-4a10-4c85-8464-75e247feae67\") " Oct 30 13:13:07.996723 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:07.996650 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5fb05c2-4a10-4c85-8464-75e247feae67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5fb05c2-4a10-4c85-8464-75e247feae67-workload-socket\") pod \"e5fb05c2-4a10-4c85-8464-75e247feae67\" (UID: \"e5fb05c2-4a10-4c85-8464-75e247feae67\") " Oct 30 13:13:07.996723 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:07.996676 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5fb05c2-4a10-4c85-8464-75e247feae67-istiod-ca-cert" pod "e5fb05c2-4a10-4c85-8464-75e247feae67" (UID: "e5fb05c2-4a10-4c85-8464-75e247feae67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:13:07.996886 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:07.996858 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5fb05c2-4a10-4c85-8464-75e247feae67-credential-socket" pod "e5fb05c2-4a10-4c85-8464-75e247feae67" (UID: "e5fb05c2-4a10-4c85-8464-75e247feae67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:07.996999 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:07.996975 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5fb05c2-4a10-4c85-8464-75e247feae67-workload-certs" pod "e5fb05c2-4a10-4c85-8464-75e247feae67" (UID: "e5fb05c2-4a10-4c85-8464-75e247feae67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:07.997136 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:07.997114 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5fb05c2-4a10-4c85-8464-75e247feae67-workload-socket" pod "e5fb05c2-4a10-4c85-8464-75e247feae67" (UID: "e5fb05c2-4a10-4c85-8464-75e247feae67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:07.997235 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:07.997115 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5fb05c2-4a10-4c85-8464-75e247feae67-istio-data" pod "e5fb05c2-4a10-4c85-8464-75e247feae67" (UID: "e5fb05c2-4a10-4c85-8464-75e247feae67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:07.997355 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:07.997336 5922 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5fb05c2-4a10-4c85-8464-75e247feae67-credential-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:07.997424 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:07.997359 5922 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5fb05c2-4a10-4c85-8464-75e247feae67-workload-certs\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:07.997424 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:07.997375 5922 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5fb05c2-4a10-4c85-8464-75e247feae67-workload-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:07.997424 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:07.997389 5922 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5fb05c2-4a10-4c85-8464-75e247feae67-istiod-ca-cert\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:07.997424 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:07.997399 5922 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5fb05c2-4a10-4c85-8464-75e247feae67-istio-data\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:07.998925 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:07.998902 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5fb05c2-4a10-4c85-8464-75e247feae67-kube-api-access-jgvxj" pod "e5fb05c2-4a10-4c85-8464-75e247feae67" (UID: "e5fb05c2-4a10-4c85-8464-75e247feae67"). InnerVolumeSpecName "kube-api-access-jgvxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:07.999016 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:07.998951 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5fb05c2-4a10-4c85-8464-75e247feae67-istio-podinfo" pod "e5fb05c2-4a10-4c85-8464-75e247feae67" (UID: "e5fb05c2-4a10-4c85-8464-75e247feae67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:13:07.999016 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:07.998951 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5fb05c2-4a10-4c85-8464-75e247feae67-istio-envoy" pod "e5fb05c2-4a10-4c85-8464-75e247feae67" (UID: "e5fb05c2-4a10-4c85-8464-75e247feae67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:07.999016 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:07.998977 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5fb05c2-4a10-4c85-8464-75e247feae67-istio-token" pod "e5fb05c2-4a10-4c85-8464-75e247feae67" (UID: "e5fb05c2-4a10-4c85-8464-75e247feae67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:08.024588 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.024557 5922 generic.go:356] "Generic (PLEG): container finished" podID="03433bd3-844d-42ca-b3b2-158f1a1f678d" containerID="2ed3debba84d1f6b67506d240bcdc230131bf906720e8debef4e93c8f2112afe" exitCode=0 Oct 30 13:13:08.024765 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.024736 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" event={"ID":"03433bd3-844d-42ca-b3b2-158f1a1f678d","Type":"ContainerDied","Data":"2ed3debba84d1f6b67506d240bcdc230131bf906720e8debef4e93c8f2112afe"} Oct 30 13:13:08.027077 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.027052 5922 generic.go:356] "Generic (PLEG): container finished" podID="5813c167-a1c1-49ca-b93c-50fb1ece0b64" containerID="d5cb6b7d0aca9cface57705be1dfc1fa8aeb2daa1aab7c6c6eded7b2f25e3080" exitCode=0 Oct 30 13:13:08.027255 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.027240 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" Oct 30 13:13:08.027403 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.027371 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" event={"ID":"5813c167-a1c1-49ca-b93c-50fb1ece0b64","Type":"ContainerDied","Data":"d5cb6b7d0aca9cface57705be1dfc1fa8aeb2daa1aab7c6c6eded7b2f25e3080"} Oct 30 13:13:08.027455 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.027416 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2" event={"ID":"5813c167-a1c1-49ca-b93c-50fb1ece0b64","Type":"ContainerDied","Data":"48b81290c1a3a8b9d8acb4db69cc6388b980e2229519d4eb265759f9644be7f0"} Oct 30 13:13:08.027502 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.027458 5922 scope.go:122] "RemoveContainer" containerID="d5cb6b7d0aca9cface57705be1dfc1fa8aeb2daa1aab7c6c6eded7b2f25e3080" Oct 30 13:13:08.028898 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.028879 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" Oct 30 13:13:08.029222 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.029195 5922 generic.go:356] "Generic (PLEG): container finished" podID="e5fb05c2-4a10-4c85-8464-75e247feae67" containerID="b0ed688211bcc67268db4556a64ca2a9883c7b63106739c7381c6df918864c9c" exitCode=0 Oct 30 13:13:08.029320 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.029238 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" event={"ID":"e5fb05c2-4a10-4c85-8464-75e247feae67","Type":"ContainerDied","Data":"b0ed688211bcc67268db4556a64ca2a9883c7b63106739c7381c6df918864c9c"} Oct 30 13:13:08.029320 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.029275 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" event={"ID":"e5fb05c2-4a10-4c85-8464-75e247feae67","Type":"ContainerDied","Data":"6a9e02ff6f39c888bd7317c041b629355916ea80f202c0ead00d23c629c68f2e"} Oct 30 13:13:08.029446 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.029351 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd" Oct 30 13:13:08.038818 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.038793 5922 scope.go:122] "RemoveContainer" containerID="d5cb6b7d0aca9cface57705be1dfc1fa8aeb2daa1aab7c6c6eded7b2f25e3080" Oct 30 13:13:08.039117 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:13:08.039097 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5cb6b7d0aca9cface57705be1dfc1fa8aeb2daa1aab7c6c6eded7b2f25e3080\": container with ID starting with d5cb6b7d0aca9cface57705be1dfc1fa8aeb2daa1aab7c6c6eded7b2f25e3080 not found: ID does not exist" containerID="d5cb6b7d0aca9cface57705be1dfc1fa8aeb2daa1aab7c6c6eded7b2f25e3080" Oct 30 13:13:08.039209 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.039128 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5cb6b7d0aca9cface57705be1dfc1fa8aeb2daa1aab7c6c6eded7b2f25e3080"} err="failed to get container status \"d5cb6b7d0aca9cface57705be1dfc1fa8aeb2daa1aab7c6c6eded7b2f25e3080\": rpc error: code = NotFound desc = could not find container \"d5cb6b7d0aca9cface57705be1dfc1fa8aeb2daa1aab7c6c6eded7b2f25e3080\": container with ID starting with d5cb6b7d0aca9cface57705be1dfc1fa8aeb2daa1aab7c6c6eded7b2f25e3080 not found: ID does not exist" Oct 30 13:13:08.039209 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.039152 5922 scope.go:122] "RemoveContainer" containerID="b0ed688211bcc67268db4556a64ca2a9883c7b63106739c7381c6df918864c9c" Oct 30 13:13:08.052859 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.052837 5922 scope.go:122] "RemoveContainer" containerID="b0ed688211bcc67268db4556a64ca2a9883c7b63106739c7381c6df918864c9c" Oct 30 13:13:08.053146 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:13:08.053127 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0ed688211bcc67268db4556a64ca2a9883c7b63106739c7381c6df918864c9c\": container with ID starting with b0ed688211bcc67268db4556a64ca2a9883c7b63106739c7381c6df918864c9c not found: ID does not exist" containerID="b0ed688211bcc67268db4556a64ca2a9883c7b63106739c7381c6df918864c9c" Oct 30 13:13:08.053228 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.053150 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0ed688211bcc67268db4556a64ca2a9883c7b63106739c7381c6df918864c9c"} err="failed to get container status \"b0ed688211bcc67268db4556a64ca2a9883c7b63106739c7381c6df918864c9c\": rpc error: code = NotFound desc = could not find container \"b0ed688211bcc67268db4556a64ca2a9883c7b63106739c7381c6df918864c9c\": container with ID starting with b0ed688211bcc67268db4556a64ca2a9883c7b63106739c7381c6df918864c9c not found: ID does not exist" Oct 30 13:13:08.098339 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.098316 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03433bd3-844d-42ca-b3b2-158f1a1f678d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03433bd3-844d-42ca-b3b2-158f1a1f678d-istiod-ca-cert\") pod \"03433bd3-844d-42ca-b3b2-158f1a1f678d\" (UID: \"03433bd3-844d-42ca-b3b2-158f1a1f678d\") " Oct 30 13:13:08.098444 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.098351 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03433bd3-844d-42ca-b3b2-158f1a1f678d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03433bd3-844d-42ca-b3b2-158f1a1f678d-istio-data\") pod \"03433bd3-844d-42ca-b3b2-158f1a1f678d\" (UID: \"03433bd3-844d-42ca-b3b2-158f1a1f678d\") " Oct 30 13:13:08.098444 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.098380 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03433bd3-844d-42ca-b3b2-158f1a1f678d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03433bd3-844d-42ca-b3b2-158f1a1f678d-workload-socket\") pod \"03433bd3-844d-42ca-b3b2-158f1a1f678d\" (UID: \"03433bd3-844d-42ca-b3b2-158f1a1f678d\") " Oct 30 13:13:08.098528 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.098493 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03433bd3-844d-42ca-b3b2-158f1a1f678d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03433bd3-844d-42ca-b3b2-158f1a1f678d-credential-socket\") pod \"03433bd3-844d-42ca-b3b2-158f1a1f678d\" (UID: \"03433bd3-844d-42ca-b3b2-158f1a1f678d\") " Oct 30 13:13:08.098604 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.098587 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5813c167-a1c1-49ca-b93c-50fb1ece0b64-istio-token\" (UniqueName: \"kubernetes.io/projected/5813c167-a1c1-49ca-b93c-50fb1ece0b64-istio-token\") pod \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\" (UID: \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\") " Oct 30 13:13:08.098664 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.098644 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03433bd3-844d-42ca-b3b2-158f1a1f678d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03433bd3-844d-42ca-b3b2-158f1a1f678d-workload-certs\") pod \"03433bd3-844d-42ca-b3b2-158f1a1f678d\" (UID: \"03433bd3-844d-42ca-b3b2-158f1a1f678d\") " Oct 30 13:13:08.098718 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.098589 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03433bd3-844d-42ca-b3b2-158f1a1f678d-workload-socket" pod "03433bd3-844d-42ca-b3b2-158f1a1f678d" (UID: "03433bd3-844d-42ca-b3b2-158f1a1f678d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:08.098718 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.098605 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03433bd3-844d-42ca-b3b2-158f1a1f678d-istio-data" pod "03433bd3-844d-42ca-b3b2-158f1a1f678d" (UID: "03433bd3-844d-42ca-b3b2-158f1a1f678d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:08.098834 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.098728 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5813c167-a1c1-49ca-b93c-50fb1ece0b64-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5813c167-a1c1-49ca-b93c-50fb1ece0b64-istio-data\") pod \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\" (UID: \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\") " Oct 30 13:13:08.098834 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.098740 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03433bd3-844d-42ca-b3b2-158f1a1f678d-credential-socket" pod "03433bd3-844d-42ca-b3b2-158f1a1f678d" (UID: "03433bd3-844d-42ca-b3b2-158f1a1f678d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:08.098834 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.098768 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5813c167-a1c1-49ca-b93c-50fb1ece0b64-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5813c167-a1c1-49ca-b93c-50fb1ece0b64-istiod-ca-cert\") pod \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\" (UID: \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\") " Oct 30 13:13:08.098834 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.098771 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03433bd3-844d-42ca-b3b2-158f1a1f678d-istiod-ca-cert" pod "03433bd3-844d-42ca-b3b2-158f1a1f678d" (UID: "03433bd3-844d-42ca-b3b2-158f1a1f678d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:13:08.098834 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.098813 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5813c167-a1c1-49ca-b93c-50fb1ece0b64-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5813c167-a1c1-49ca-b93c-50fb1ece0b64-workload-socket\") pod \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\" (UID: \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\") " Oct 30 13:13:08.099087 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.098846 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03433bd3-844d-42ca-b3b2-158f1a1f678d-kube-api-access-ft2mc\" (UniqueName: \"kubernetes.io/projected/03433bd3-844d-42ca-b3b2-158f1a1f678d-kube-api-access-ft2mc\") pod \"03433bd3-844d-42ca-b3b2-158f1a1f678d\" (UID: \"03433bd3-844d-42ca-b3b2-158f1a1f678d\") " Oct 30 13:13:08.099087 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.098894 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5813c167-a1c1-49ca-b93c-50fb1ece0b64-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5813c167-a1c1-49ca-b93c-50fb1ece0b64-istio-podinfo\") pod \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\" (UID: \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\") " Oct 30 13:13:08.099087 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.098914 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03433bd3-844d-42ca-b3b2-158f1a1f678d-workload-certs" pod "03433bd3-844d-42ca-b3b2-158f1a1f678d" (UID: "03433bd3-844d-42ca-b3b2-158f1a1f678d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:08.099087 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.098938 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5813c167-a1c1-49ca-b93c-50fb1ece0b64-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5813c167-a1c1-49ca-b93c-50fb1ece0b64-istio-envoy\") pod \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\" (UID: \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\") " Oct 30 13:13:08.099087 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.098984 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5813c167-a1c1-49ca-b93c-50fb1ece0b64-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5813c167-a1c1-49ca-b93c-50fb1ece0b64-workload-certs\") pod \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\" (UID: \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\") " Oct 30 13:13:08.099087 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.099000 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5813c167-a1c1-49ca-b93c-50fb1ece0b64-istio-data" pod "5813c167-a1c1-49ca-b93c-50fb1ece0b64" (UID: "5813c167-a1c1-49ca-b93c-50fb1ece0b64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:08.099087 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.099034 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03433bd3-844d-42ca-b3b2-158f1a1f678d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03433bd3-844d-42ca-b3b2-158f1a1f678d-istio-envoy\") pod \"03433bd3-844d-42ca-b3b2-158f1a1f678d\" (UID: \"03433bd3-844d-42ca-b3b2-158f1a1f678d\") " Oct 30 13:13:08.099087 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.099065 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5813c167-a1c1-49ca-b93c-50fb1ece0b64-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5813c167-a1c1-49ca-b93c-50fb1ece0b64-credential-socket\") pod \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\" (UID: \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\") " Oct 30 13:13:08.099515 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.099098 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03433bd3-844d-42ca-b3b2-158f1a1f678d-istio-token\" (UniqueName: \"kubernetes.io/projected/03433bd3-844d-42ca-b3b2-158f1a1f678d-istio-token\") pod \"03433bd3-844d-42ca-b3b2-158f1a1f678d\" (UID: \"03433bd3-844d-42ca-b3b2-158f1a1f678d\") " Oct 30 13:13:08.099515 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.099110 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5813c167-a1c1-49ca-b93c-50fb1ece0b64-workload-socket" pod "5813c167-a1c1-49ca-b93c-50fb1ece0b64" (UID: "5813c167-a1c1-49ca-b93c-50fb1ece0b64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:08.099515 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.099133 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5813c167-a1c1-49ca-b93c-50fb1ece0b64-kube-api-access-d8v6c\" (UniqueName: \"kubernetes.io/projected/5813c167-a1c1-49ca-b93c-50fb1ece0b64-kube-api-access-d8v6c\") pod \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\" (UID: \"5813c167-a1c1-49ca-b93c-50fb1ece0b64\") " Oct 30 13:13:08.099515 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.099169 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03433bd3-844d-42ca-b3b2-158f1a1f678d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03433bd3-844d-42ca-b3b2-158f1a1f678d-istio-podinfo\") pod \"03433bd3-844d-42ca-b3b2-158f1a1f678d\" (UID: \"03433bd3-844d-42ca-b3b2-158f1a1f678d\") " Oct 30 13:13:08.099515 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.099193 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5813c167-a1c1-49ca-b93c-50fb1ece0b64-istiod-ca-cert" pod "5813c167-a1c1-49ca-b93c-50fb1ece0b64" (UID: "5813c167-a1c1-49ca-b93c-50fb1ece0b64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:13:08.099515 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.099317 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5813c167-a1c1-49ca-b93c-50fb1ece0b64-workload-certs" pod "5813c167-a1c1-49ca-b93c-50fb1ece0b64" (UID: "5813c167-a1c1-49ca-b93c-50fb1ece0b64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:08.099515 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.099328 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5813c167-a1c1-49ca-b93c-50fb1ece0b64-credential-socket" pod "5813c167-a1c1-49ca-b93c-50fb1ece0b64" (UID: "5813c167-a1c1-49ca-b93c-50fb1ece0b64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:08.099903 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.099728 5922 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5813c167-a1c1-49ca-b93c-50fb1ece0b64-istiod-ca-cert\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.099903 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.099750 5922 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5813c167-a1c1-49ca-b93c-50fb1ece0b64-workload-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.099903 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.099764 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgvxj\" (UniqueName: \"kubernetes.io/projected/e5fb05c2-4a10-4c85-8464-75e247feae67-kube-api-access-jgvxj\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.099903 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.099780 5922 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5813c167-a1c1-49ca-b93c-50fb1ece0b64-workload-certs\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.099903 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.099794 5922 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5813c167-a1c1-49ca-b93c-50fb1ece0b64-credential-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.099903 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.099808 5922 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03433bd3-844d-42ca-b3b2-158f1a1f678d-istiod-ca-cert\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.099903 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.099820 5922 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03433bd3-844d-42ca-b3b2-158f1a1f678d-istio-data\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.099903 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.099832 5922 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03433bd3-844d-42ca-b3b2-158f1a1f678d-workload-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.099903 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.099845 5922 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03433bd3-844d-42ca-b3b2-158f1a1f678d-credential-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.099903 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.099858 5922 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5fb05c2-4a10-4c85-8464-75e247feae67-istio-podinfo\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.099903 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.099870 5922 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03433bd3-844d-42ca-b3b2-158f1a1f678d-workload-certs\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.099903 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.099882 5922 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5fb05c2-4a10-4c85-8464-75e247feae67-istio-envoy\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.099903 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.099894 5922 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5813c167-a1c1-49ca-b93c-50fb1ece0b64-istio-data\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.099903 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.099906 5922 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5fb05c2-4a10-4c85-8464-75e247feae67-istio-token\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.101209 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.101188 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5813c167-a1c1-49ca-b93c-50fb1ece0b64-istio-token" pod "5813c167-a1c1-49ca-b93c-50fb1ece0b64" (UID: "5813c167-a1c1-49ca-b93c-50fb1ece0b64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:08.101401 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.101385 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5813c167-a1c1-49ca-b93c-50fb1ece0b64-istio-podinfo" pod "5813c167-a1c1-49ca-b93c-50fb1ece0b64" (UID: "5813c167-a1c1-49ca-b93c-50fb1ece0b64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:13:08.101595 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.101572 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03433bd3-844d-42ca-b3b2-158f1a1f678d-kube-api-access-ft2mc" pod "03433bd3-844d-42ca-b3b2-158f1a1f678d" (UID: "03433bd3-844d-42ca-b3b2-158f1a1f678d"). InnerVolumeSpecName "kube-api-access-ft2mc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:08.101814 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.101788 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5813c167-a1c1-49ca-b93c-50fb1ece0b64-kube-api-access-d8v6c" pod "5813c167-a1c1-49ca-b93c-50fb1ece0b64" (UID: "5813c167-a1c1-49ca-b93c-50fb1ece0b64"). InnerVolumeSpecName "kube-api-access-d8v6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:08.101892 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.101839 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03433bd3-844d-42ca-b3b2-158f1a1f678d-istio-token" pod "03433bd3-844d-42ca-b3b2-158f1a1f678d" (UID: "03433bd3-844d-42ca-b3b2-158f1a1f678d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:08.102272 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.102256 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5813c167-a1c1-49ca-b93c-50fb1ece0b64-istio-envoy" pod "5813c167-a1c1-49ca-b93c-50fb1ece0b64" (UID: "5813c167-a1c1-49ca-b93c-50fb1ece0b64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:08.102367 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.102351 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03433bd3-844d-42ca-b3b2-158f1a1f678d-istio-podinfo" pod "03433bd3-844d-42ca-b3b2-158f1a1f678d" (UID: "03433bd3-844d-42ca-b3b2-158f1a1f678d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:13:08.102565 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.102553 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03433bd3-844d-42ca-b3b2-158f1a1f678d-istio-envoy" pod "03433bd3-844d-42ca-b3b2-158f1a1f678d" (UID: "03433bd3-844d-42ca-b3b2-158f1a1f678d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:08.156429 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.156401 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd"] Oct 30 13:13:08.165104 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.165077 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-d5zjd"] Oct 30 13:13:08.200962 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.200935 5922 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5813c167-a1c1-49ca-b93c-50fb1ece0b64-istio-token\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.200962 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.200961 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft2mc\" (UniqueName: \"kubernetes.io/projected/03433bd3-844d-42ca-b3b2-158f1a1f678d-kube-api-access-ft2mc\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.201118 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.200971 5922 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5813c167-a1c1-49ca-b93c-50fb1ece0b64-istio-podinfo\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.201118 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.200982 5922 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5813c167-a1c1-49ca-b93c-50fb1ece0b64-istio-envoy\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.201118 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.200989 5922 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03433bd3-844d-42ca-b3b2-158f1a1f678d-istio-envoy\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.201118 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.200996 5922 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03433bd3-844d-42ca-b3b2-158f1a1f678d-istio-token\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.201118 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.201003 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8v6c\" (UniqueName: \"kubernetes.io/projected/5813c167-a1c1-49ca-b93c-50fb1ece0b64-kube-api-access-d8v6c\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.201118 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.201013 5922 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03433bd3-844d-42ca-b3b2-158f1a1f678d-istio-podinfo\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.378886 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.378854 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2"] Oct 30 13:13:08.384439 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.384413 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-2drs2"] Oct 30 13:13:08.881747 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:08.881712 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:09.045016 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:09.044983 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" Oct 30 13:13:09.045167 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:09.045017 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk" event={"ID":"03433bd3-844d-42ca-b3b2-158f1a1f678d","Type":"ContainerDied","Data":"6ff39cbce3d44ddb3bbe5ca17d0b57a4736063e85ea674cda16a958bea4beb41"} Oct 30 13:13:09.045167 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:09.045078 5922 scope.go:122] "RemoveContainer" containerID="2ed3debba84d1f6b67506d240bcdc230131bf906720e8debef4e93c8f2112afe" Oct 30 13:13:09.106561 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:09.106530 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk"] Oct 30 13:13:09.118448 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:09.118419 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-hj8tk"] Oct 30 13:13:09.644390 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:09.644357 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03433bd3-844d-42ca-b3b2-158f1a1f678d" path="/var/lib/kubelet/pods/03433bd3-844d-42ca-b3b2-158f1a1f678d/volumes" Oct 30 13:13:09.644871 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:09.644855 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5813c167-a1c1-49ca-b93c-50fb1ece0b64" path="/var/lib/kubelet/pods/5813c167-a1c1-49ca-b93c-50fb1ece0b64/volumes" Oct 30 13:13:09.645281 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:09.645268 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5fb05c2-4a10-4c85-8464-75e247feae67" path="/var/lib/kubelet/pods/e5fb05c2-4a10-4c85-8464-75e247feae67/volumes" Oct 30 13:13:10.305601 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:10.305562 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bb71e2-59f79f4f99-xh9lj" podUID="72a0ed5e-1d31-4695-9dc5-92a304c0dc69" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:13:10.712189 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:10.712021 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:10.839821 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:10.839783 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:10.878020 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:10.877987 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-175831-lsz8g"] Oct 30 13:13:10.878467 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:10.878453 5922 cpu_manager.go:414] "containerMap: removing container" podUID="727a0820-c1f2-469e-827d-3f623e9df963" containerName="payload-processing" Oct 30 13:13:10.878520 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:10.878469 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="727a0820-c1f2-469e-827d-3f623e9df963" containerName="payload-processing" Oct 30 13:13:10.878520 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:10.878483 5922 cpu_manager.go:414] "containerMap: removing container" podUID="5813c167-a1c1-49ca-b93c-50fb1ece0b64" containerName="istio-proxy" Oct 30 13:13:10.878520 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:10.878488 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5813c167-a1c1-49ca-b93c-50fb1ece0b64" containerName="istio-proxy" Oct 30 13:13:10.878520 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:10.878495 5922 cpu_manager.go:414] "containerMap: removing container" podUID="19779cce-d1a1-429b-9ece-85f9932374c5" containerName="istio-proxy" Oct 30 13:13:10.878520 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:10.878499 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19779cce-d1a1-429b-9ece-85f9932374c5" containerName="istio-proxy" Oct 30 13:13:10.878717 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:10.878525 5922 cpu_manager.go:414] "containerMap: removing container" podUID="e5fb05c2-4a10-4c85-8464-75e247feae67" containerName="istio-proxy" Oct 30 13:13:10.878717 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:10.878530 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5fb05c2-4a10-4c85-8464-75e247feae67" containerName="istio-proxy" Oct 30 13:13:10.878717 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:10.878537 5922 cpu_manager.go:414] "containerMap: removing container" podUID="ac077401-1be7-4461-a1e5-bc4f186c9a94" containerName="payload-pre-processing" Oct 30 13:13:10.878717 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:10.878542 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac077401-1be7-4461-a1e5-bc4f186c9a94" containerName="payload-pre-processing" Oct 30 13:13:10.878717 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:10.878551 5922 cpu_manager.go:414] "containerMap: removing container" podUID="03433bd3-844d-42ca-b3b2-158f1a1f678d" containerName="istio-proxy" Oct 30 13:13:10.878717 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:10.878556 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03433bd3-844d-42ca-b3b2-158f1a1f678d" containerName="istio-proxy" Oct 30 13:13:10.878717 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:10.878565 5922 cpu_manager.go:414] "containerMap: removing container" podUID="8b126f0c-889e-4485-a4af-4ee23b88ab57" containerName="revoke-keys" Oct 30 13:13:10.878717 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:10.878570 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b126f0c-889e-4485-a4af-4ee23b88ab57" containerName="revoke-keys" Oct 30 13:13:10.878717 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:10.878577 5922 cpu_manager.go:414] "containerMap: removing container" podUID="ef454a96-712b-4ece-b4d8-b116964867a5" containerName="istio-proxy" Oct 30 13:13:10.878717 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:10.878581 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef454a96-712b-4ece-b4d8-b116964867a5" containerName="istio-proxy" Oct 30 13:13:10.878717 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:10.878589 5922 cpu_manager.go:414] "containerMap: removing container" podUID="a85de36b-f8c8-4ade-a468-c114080cb312" containerName="main" Oct 30 13:13:10.878717 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:10.878594 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a85de36b-f8c8-4ade-a468-c114080cb312" containerName="main" Oct 30 13:13:10.878717 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:10.878672 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="19779cce-d1a1-429b-9ece-85f9932374c5" containerName="istio-proxy" Oct 30 13:13:10.878717 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:10.878683 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5fb05c2-4a10-4c85-8464-75e247feae67" containerName="istio-proxy" Oct 30 13:13:10.878717 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:10.878692 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac077401-1be7-4461-a1e5-bc4f186c9a94" containerName="payload-pre-processing" Oct 30 13:13:10.878717 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:10.878697 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="03433bd3-844d-42ca-b3b2-158f1a1f678d" containerName="istio-proxy" Oct 30 13:13:10.878717 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:10.878706 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b126f0c-889e-4485-a4af-4ee23b88ab57" containerName="revoke-keys" Oct 30 13:13:10.878717 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:10.878713 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef454a96-712b-4ece-b4d8-b116964867a5" containerName="istio-proxy" Oct 30 13:13:10.878717 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:10.878719 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="a85de36b-f8c8-4ade-a468-c114080cb312" containerName="main" Oct 30 13:13:10.878717 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:10.878726 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="727a0820-c1f2-469e-827d-3f623e9df963" containerName="payload-processing" Oct 30 13:13:10.879277 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:10.878733 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="5813c167-a1c1-49ca-b93c-50fb1ece0b64" containerName="istio-proxy" Oct 30 13:13:10.884067 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:10.884050 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-175831-lsz8g" Oct 30 13:13:10.902054 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:10.902023 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-175831-lsz8g"] Oct 30 13:13:10.910131 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:10.910109 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:13:10.910254 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:10.910161 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:13:10.910254 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:10.910212 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:13:10.911869 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:10.911840 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:13:11.032890 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:11.032811 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b98661f-1136-483c-b836-85d753238487-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-175831-lsz8g\" (UID: \"1b98661f-1136-483c-b836-85d753238487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-175831-lsz8g" Oct 30 13:13:11.033036 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:11.032896 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sqx9\" (UniqueName: \"kubernetes.io/projected/1b98661f-1136-483c-b836-85d753238487-kube-api-access-9sqx9\") pod \"maas-api-revoke-keys-e2e-worker-w5-175831-lsz8g\" (UID: \"1b98661f-1136-483c-b836-85d753238487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-175831-lsz8g" Oct 30 13:13:11.033036 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:11.033022 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b98661f-1136-483c-b836-85d753238487-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-175831-lsz8g\" (UID: \"1b98661f-1136-483c-b836-85d753238487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-175831-lsz8g" Oct 30 13:13:11.134151 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:11.134119 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b98661f-1136-483c-b836-85d753238487-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-175831-lsz8g\" (UID: \"1b98661f-1136-483c-b836-85d753238487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-175831-lsz8g" Oct 30 13:13:11.134332 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:11.134188 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sqx9\" (UniqueName: \"kubernetes.io/projected/1b98661f-1136-483c-b836-85d753238487-kube-api-access-9sqx9\") pod \"maas-api-revoke-keys-e2e-worker-w5-175831-lsz8g\" (UID: \"1b98661f-1136-483c-b836-85d753238487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-175831-lsz8g" Oct 30 13:13:11.134332 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:11.134279 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b98661f-1136-483c-b836-85d753238487-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-175831-lsz8g\" (UID: \"1b98661f-1136-483c-b836-85d753238487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-175831-lsz8g" Oct 30 13:13:11.134919 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:11.134894 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b98661f-1136-483c-b836-85d753238487-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-175831-lsz8g\" (UID: \"1b98661f-1136-483c-b836-85d753238487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-175831-lsz8g" Oct 30 13:13:11.136388 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:11.136367 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b98661f-1136-483c-b836-85d753238487-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-175831-lsz8g\" (UID: \"1b98661f-1136-483c-b836-85d753238487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-175831-lsz8g" Oct 30 13:13:11.158529 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:11.158499 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sqx9\" (UniqueName: \"kubernetes.io/projected/1b98661f-1136-483c-b836-85d753238487-kube-api-access-9sqx9\") pod \"maas-api-revoke-keys-e2e-worker-w5-175831-lsz8g\" (UID: \"1b98661f-1136-483c-b836-85d753238487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-175831-lsz8g" Oct 30 13:13:11.200549 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:11.200524 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-175831-lsz8g" Oct 30 13:13:11.439580 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:11.439503 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:11.504301 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:11.504268 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-175831-lsz8g"] Oct 30 13:13:11.505944 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:13:11.505917 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b98661f_1136_483c_b836_85d753238487.slice/crio-8ddae6dcfd133b18aeb304bd107f4318981e0409407e34e8623e064e65ac576f WatchSource:0}: Error finding container 8ddae6dcfd133b18aeb304bd107f4318981e0409407e34e8623e064e65ac576f: Status 404 returned error can't find the container with id 8ddae6dcfd133b18aeb304bd107f4318981e0409407e34e8623e064e65ac576f Oct 30 13:13:11.557192 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:11.557161 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:12.082091 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:12.082008 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-175831-lsz8g" event={"ID":"1b98661f-1136-483c-b836-85d753238487","Type":"ContainerStarted","Data":"7442678e5cddbc80064518ab5cef773d84cf01652e392b873f514c7d1d88ddb3"} Oct 30 13:13:12.082091 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:12.082069 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-175831-lsz8g" event={"ID":"1b98661f-1136-483c-b836-85d753238487","Type":"ContainerStarted","Data":"8ddae6dcfd133b18aeb304bd107f4318981e0409407e34e8623e064e65ac576f"} Oct 30 13:13:12.128026 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:12.127983 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-175831-lsz8g" podStartSLOduration=2.127970055 podStartE2EDuration="2.127970055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:12.124079318 +0000 UTC m=+1248.593495062" watchObservedRunningTime="2026-10-30 13:13:12.127970055 +0000 UTC m=+1248.597385799" Oct 30 13:13:12.410489 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:12.410459 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:12.726794 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:12.726756 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:13.052287 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:13.052217 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc9a7925-vnz2x"] Oct 30 13:13:13.064107 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:13.064080 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc9a7925-vnz2x"] Oct 30 13:13:13.091229 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:13.091200 5922 generic.go:356] "Generic (PLEG): container finished" podID="1b98661f-1136-483c-b836-85d753238487" containerID="7442678e5cddbc80064518ab5cef773d84cf01652e392b873f514c7d1d88ddb3" exitCode=0 Oct 30 13:13:13.091337 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:13.091293 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-175831-lsz8g" event={"ID":"1b98661f-1136-483c-b836-85d753238487","Type":"ContainerDied","Data":"7442678e5cddbc80064518ab5cef773d84cf01652e392b873f514c7d1d88ddb3"} Oct 30 13:13:13.644228 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:13.644197 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b985705-3a9a-4ade-8cd5-a09e6d425f6e" path="/var/lib/kubelet/pods/0b985705-3a9a-4ade-8cd5-a09e6d425f6e/volumes" Oct 30 13:13:13.908597 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:13.908520 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:14.038188 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:14.038158 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08d73678-qgj7k"] Oct 30 13:13:14.049993 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:14.049965 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08d73678-qgj7k"] Oct 30 13:13:14.251990 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:14.251917 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-175831-lsz8g" Oct 30 13:13:14.372781 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:14.372753 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b98661f-1136-483c-b836-85d753238487-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b98661f-1136-483c-b836-85d753238487-maas-api-openshift-ca\") pod \"1b98661f-1136-483c-b836-85d753238487\" (UID: \"1b98661f-1136-483c-b836-85d753238487\") " Oct 30 13:13:14.372946 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:14.372815 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b98661f-1136-483c-b836-85d753238487-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b98661f-1136-483c-b836-85d753238487-maas-api-xks-ca\") pod \"1b98661f-1136-483c-b836-85d753238487\" (UID: \"1b98661f-1136-483c-b836-85d753238487\") " Oct 30 13:13:14.372946 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:14.372917 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b98661f-1136-483c-b836-85d753238487-kube-api-access-9sqx9\" (UniqueName: \"kubernetes.io/projected/1b98661f-1136-483c-b836-85d753238487-kube-api-access-9sqx9\") pod \"1b98661f-1136-483c-b836-85d753238487\" (UID: \"1b98661f-1136-483c-b836-85d753238487\") " Oct 30 13:13:14.373207 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:14.373173 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b98661f-1136-483c-b836-85d753238487-maas-api-openshift-ca" pod "1b98661f-1136-483c-b836-85d753238487" (UID: "1b98661f-1136-483c-b836-85d753238487"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:13:14.373377 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:14.373362 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b98661f-1136-483c-b836-85d753238487-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:14.374940 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:14.374923 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b98661f-1136-483c-b836-85d753238487-maas-api-xks-ca" pod "1b98661f-1136-483c-b836-85d753238487" (UID: "1b98661f-1136-483c-b836-85d753238487"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:13:14.375006 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:14.374988 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b98661f-1136-483c-b836-85d753238487-kube-api-access-9sqx9" pod "1b98661f-1136-483c-b836-85d753238487" (UID: "1b98661f-1136-483c-b836-85d753238487"). InnerVolumeSpecName "kube-api-access-9sqx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:14.474201 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:14.474177 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b98661f-1136-483c-b836-85d753238487-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:14.474201 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:14.474201 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sqx9\" (UniqueName: \"kubernetes.io/projected/1b98661f-1136-483c-b836-85d753238487-kube-api-access-9sqx9\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:14.508159 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:14.508094 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:15.116836 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:15.116806 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-175831-lsz8g" Oct 30 13:13:15.117228 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:15.116854 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-175831-lsz8g" event={"ID":"1b98661f-1136-483c-b836-85d753238487","Type":"ContainerDied","Data":"8ddae6dcfd133b18aeb304bd107f4318981e0409407e34e8623e064e65ac576f"} Oct 30 13:13:15.117228 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:15.116945 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ddae6dcfd133b18aeb304bd107f4318981e0409407e34e8623e064e65ac576f" Oct 30 13:13:15.644851 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:15.644818 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11ebdacb-1634-472a-b8d4-2785866325df" path="/var/lib/kubelet/pods/11ebdacb-1634-472a-b8d4-2785866325df/volumes" Oct 30 13:13:15.697092 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:15.697063 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2"] Oct 30 13:13:15.836209 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:15.836177 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:16.048521 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:16.048475 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:16.071729 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:16.071708 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bb71e2-59f79f4f99-xh9lj" Oct 30 13:13:16.127007 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:16.126974 5922 generic.go:356] "Generic (PLEG): container finished" podID="72a0ed5e-1d31-4695-9dc5-92a304c0dc69" containerID="c39f7418cd4472879b43de64fe85b261a2cdd81d409b658f68edd9650c50375f" exitCode=0 Oct 30 13:13:16.127502 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:16.127016 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bb71e2-59f79f4f99-xh9lj" event={"ID":"72a0ed5e-1d31-4695-9dc5-92a304c0dc69","Type":"ContainerDied","Data":"c39f7418cd4472879b43de64fe85b261a2cdd81d409b658f68edd9650c50375f"} Oct 30 13:13:16.127502 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:16.127081 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bb71e2-59f79f4f99-xh9lj" event={"ID":"72a0ed5e-1d31-4695-9dc5-92a304c0dc69","Type":"ContainerDied","Data":"cd3ca241f8386b317f249ae483ef74711a80293ab66e396b6155d4a474c1f175"} Oct 30 13:13:16.127502 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:16.127030 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bb71e2-59f79f4f99-xh9lj" Oct 30 13:13:16.127502 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:16.127114 5922 scope.go:122] "RemoveContainer" containerID="c39f7418cd4472879b43de64fe85b261a2cdd81d409b658f68edd9650c50375f" Oct 30 13:13:16.144926 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:16.139551 5922 scope.go:122] "RemoveContainer" containerID="c39f7418cd4472879b43de64fe85b261a2cdd81d409b658f68edd9650c50375f" Oct 30 13:13:16.145388 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:13:16.145359 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c39f7418cd4472879b43de64fe85b261a2cdd81d409b658f68edd9650c50375f\": container with ID starting with c39f7418cd4472879b43de64fe85b261a2cdd81d409b658f68edd9650c50375f not found: ID does not exist" containerID="c39f7418cd4472879b43de64fe85b261a2cdd81d409b658f68edd9650c50375f" Oct 30 13:13:16.145561 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:16.145533 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c39f7418cd4472879b43de64fe85b261a2cdd81d409b658f68edd9650c50375f"} err="failed to get container status \"c39f7418cd4472879b43de64fe85b261a2cdd81d409b658f68edd9650c50375f\": rpc error: code = NotFound desc = could not find container \"c39f7418cd4472879b43de64fe85b261a2cdd81d409b658f68edd9650c50375f\": container with ID starting with c39f7418cd4472879b43de64fe85b261a2cdd81d409b658f68edd9650c50375f not found: ID does not exist" Oct 30 13:13:16.167277 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:16.167256 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:16.194343 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:16.194320 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72a0ed5e-1d31-4695-9dc5-92a304c0dc69-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72a0ed5e-1d31-4695-9dc5-92a304c0dc69-plugins-config-volume\") pod \"72a0ed5e-1d31-4695-9dc5-92a304c0dc69\" (UID: \"72a0ed5e-1d31-4695-9dc5-92a304c0dc69\") " Oct 30 13:13:16.194441 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:16.194387 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72a0ed5e-1d31-4695-9dc5-92a304c0dc69-kube-api-access-cw8rz\" (UniqueName: \"kubernetes.io/projected/72a0ed5e-1d31-4695-9dc5-92a304c0dc69-kube-api-access-cw8rz\") pod \"72a0ed5e-1d31-4695-9dc5-92a304c0dc69\" (UID: \"72a0ed5e-1d31-4695-9dc5-92a304c0dc69\") " Oct 30 13:13:16.194720 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:16.194694 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72a0ed5e-1d31-4695-9dc5-92a304c0dc69-plugins-config-volume" pod "72a0ed5e-1d31-4695-9dc5-92a304c0dc69" (UID: "72a0ed5e-1d31-4695-9dc5-92a304c0dc69"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:13:16.196535 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:16.196515 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72a0ed5e-1d31-4695-9dc5-92a304c0dc69-kube-api-access-cw8rz" pod "72a0ed5e-1d31-4695-9dc5-92a304c0dc69" (UID: "72a0ed5e-1d31-4695-9dc5-92a304c0dc69"). InnerVolumeSpecName "kube-api-access-cw8rz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:16.295424 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:16.295399 5922 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72a0ed5e-1d31-4695-9dc5-92a304c0dc69-plugins-config-volume\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:16.295552 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:16.295424 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw8rz\" (UniqueName: \"kubernetes.io/projected/72a0ed5e-1d31-4695-9dc5-92a304c0dc69-kube-api-access-cw8rz\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:16.402349 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:16.402276 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24"] Oct 30 13:13:16.403553 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:16.403517 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" podUID="1830fbd9-7c51-40d0-b6df-49a723185784" containerName="istio-proxy" containerID="cri-o://629fe8b00af9f7879c5086af3fa9c9cb0bf1701e99a7af8dd6a41b1a9204ed05" gracePeriod=30 Oct 30 13:13:16.408271 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:16.408242 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb"] Oct 30 13:13:16.408489 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:16.408467 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" podUID="b607f9a7-6ef2-4ea2-bdb4-352b2312966d" containerName="istio-proxy" containerID="cri-o://7905db159264c464bfdd28ddb020ee47a98d62f341fb933cb51740e930840481" gracePeriod=30 Oct 30 13:13:16.414167 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:16.414135 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8"] Oct 30 13:13:16.414366 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:16.414343 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" podUID="a68a48f7-d090-4eb7-af40-df304a84b319" containerName="istio-proxy" containerID="cri-o://170d1cca2c2bcc213a867eb484cd2cc7bbc0897f9156bff5b2620ebc42cbf4d7" gracePeriod=30 Oct 30 13:13:16.414549 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:16.414523 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:16.524323 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:16.524222 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:16.606893 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:16.606857 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-bb71e2-59f79f4f99-xh9lj"] Oct 30 13:13:16.610110 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:16.610084 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-bb71e2-59f79f4f99-xh9lj"] Oct 30 13:13:17.446426 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:17.446396 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:13:17.644302 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:17.644266 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72a0ed5e-1d31-4695-9dc5-92a304c0dc69" path="/var/lib/kubelet/pods/72a0ed5e-1d31-4695-9dc5-92a304c0dc69/volumes" Oct 30 13:13:18.034528 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:18.034487 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d30e8ab1-5ftgc"] Oct 30 13:13:18.046567 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:18.046540 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d30e8ab1-5ftgc"] Oct 30 13:13:18.387318 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:18.387220 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:19.379810 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:19.379770 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:19.644046 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:19.643967 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f106dfed-0dcd-4413-8647-1390c98dcdf8" path="/var/lib/kubelet/pods/f106dfed-0dcd-4413-8647-1390c98dcdf8/volumes" Oct 30 13:13:20.151253 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:20.151220 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:21.418082 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.418046 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-175831-6f46fb6d59-ltklr"] Oct 30 13:13:21.528597 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.528562 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:21.662046 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.661979 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:21.682979 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.682952 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" Oct 30 13:13:21.688670 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.688636 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26235bd2-9qhbp"] Oct 30 13:13:21.689168 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.689152 5922 cpu_manager.go:414] "containerMap: removing container" podUID="1b98661f-1136-483c-b836-85d753238487" containerName="revoke-keys" Oct 30 13:13:21.689218 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.689169 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b98661f-1136-483c-b836-85d753238487" containerName="revoke-keys" Oct 30 13:13:21.689218 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.689190 5922 cpu_manager.go:414] "containerMap: removing container" podUID="1830fbd9-7c51-40d0-b6df-49a723185784" containerName="istio-proxy" Oct 30 13:13:21.689218 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.689195 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1830fbd9-7c51-40d0-b6df-49a723185784" containerName="istio-proxy" Oct 30 13:13:21.689218 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.689206 5922 cpu_manager.go:414] "containerMap: removing container" podUID="72a0ed5e-1d31-4695-9dc5-92a304c0dc69" containerName="payload-pre-processing" Oct 30 13:13:21.689218 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.689211 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72a0ed5e-1d31-4695-9dc5-92a304c0dc69" containerName="payload-pre-processing" Oct 30 13:13:21.689377 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.689299 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b98661f-1136-483c-b836-85d753238487" containerName="revoke-keys" Oct 30 13:13:21.689377 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.689310 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="1830fbd9-7c51-40d0-b6df-49a723185784" containerName="istio-proxy" Oct 30 13:13:21.689377 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.689316 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="72a0ed5e-1d31-4695-9dc5-92a304c0dc69" containerName="payload-pre-processing" Oct 30 13:13:21.692583 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.692564 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26235bd2-9qhbp" Oct 30 13:13:21.713127 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.713100 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26235bd2-9qhbp"] Oct 30 13:13:21.714349 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.714322 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:13:21.714465 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.714409 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:13:21.715000 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.714946 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:13:21.715000 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.714947 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:13:21.718677 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.718655 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" Oct 30 13:13:21.854124 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.854079 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1830fbd9-7c51-40d0-b6df-49a723185784-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1830fbd9-7c51-40d0-b6df-49a723185784-credential-socket\") pod \"1830fbd9-7c51-40d0-b6df-49a723185784\" (UID: \"1830fbd9-7c51-40d0-b6df-49a723185784\") " Oct 30 13:13:21.854124 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.854130 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a68a48f7-d090-4eb7-af40-df304a84b319-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a68a48f7-d090-4eb7-af40-df304a84b319-workload-certs\") pod \"a68a48f7-d090-4eb7-af40-df304a84b319\" (UID: \"a68a48f7-d090-4eb7-af40-df304a84b319\") " Oct 30 13:13:21.854389 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.854169 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a68a48f7-d090-4eb7-af40-df304a84b319-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a68a48f7-d090-4eb7-af40-df304a84b319-workload-socket\") pod \"a68a48f7-d090-4eb7-af40-df304a84b319\" (UID: \"a68a48f7-d090-4eb7-af40-df304a84b319\") " Oct 30 13:13:21.854389 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.854205 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1830fbd9-7c51-40d0-b6df-49a723185784-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1830fbd9-7c51-40d0-b6df-49a723185784-workload-certs\") pod \"1830fbd9-7c51-40d0-b6df-49a723185784\" (UID: \"1830fbd9-7c51-40d0-b6df-49a723185784\") " Oct 30 13:13:21.854389 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.854236 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a68a48f7-d090-4eb7-af40-df304a84b319-istio-token\" (UniqueName: \"kubernetes.io/projected/a68a48f7-d090-4eb7-af40-df304a84b319-istio-token\") pod \"a68a48f7-d090-4eb7-af40-df304a84b319\" (UID: \"a68a48f7-d090-4eb7-af40-df304a84b319\") " Oct 30 13:13:21.854389 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.854292 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1830fbd9-7c51-40d0-b6df-49a723185784-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1830fbd9-7c51-40d0-b6df-49a723185784-workload-socket\") pod \"1830fbd9-7c51-40d0-b6df-49a723185784\" (UID: \"1830fbd9-7c51-40d0-b6df-49a723185784\") " Oct 30 13:13:21.854389 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.854362 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1830fbd9-7c51-40d0-b6df-49a723185784-kube-api-access-2frl6\" (UniqueName: \"kubernetes.io/projected/1830fbd9-7c51-40d0-b6df-49a723185784-kube-api-access-2frl6\") pod \"1830fbd9-7c51-40d0-b6df-49a723185784\" (UID: \"1830fbd9-7c51-40d0-b6df-49a723185784\") " Oct 30 13:13:21.854707 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.854381 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1830fbd9-7c51-40d0-b6df-49a723185784-credential-socket" pod "1830fbd9-7c51-40d0-b6df-49a723185784" (UID: "1830fbd9-7c51-40d0-b6df-49a723185784"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:21.854707 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.854404 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1830fbd9-7c51-40d0-b6df-49a723185784-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1830fbd9-7c51-40d0-b6df-49a723185784-istio-podinfo\") pod \"1830fbd9-7c51-40d0-b6df-49a723185784\" (UID: \"1830fbd9-7c51-40d0-b6df-49a723185784\") " Oct 30 13:13:21.854707 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.854416 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a68a48f7-d090-4eb7-af40-df304a84b319-workload-socket" pod "a68a48f7-d090-4eb7-af40-df304a84b319" (UID: "a68a48f7-d090-4eb7-af40-df304a84b319"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:21.854707 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.854444 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a68a48f7-d090-4eb7-af40-df304a84b319-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a68a48f7-d090-4eb7-af40-df304a84b319-istio-data\") pod \"a68a48f7-d090-4eb7-af40-df304a84b319\" (UID: \"a68a48f7-d090-4eb7-af40-df304a84b319\") " Oct 30 13:13:21.854707 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.854505 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a68a48f7-d090-4eb7-af40-df304a84b319-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a68a48f7-d090-4eb7-af40-df304a84b319-credential-socket\") pod \"a68a48f7-d090-4eb7-af40-df304a84b319\" (UID: \"a68a48f7-d090-4eb7-af40-df304a84b319\") " Oct 30 13:13:21.854707 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.854521 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1830fbd9-7c51-40d0-b6df-49a723185784-workload-certs" pod "1830fbd9-7c51-40d0-b6df-49a723185784" (UID: "1830fbd9-7c51-40d0-b6df-49a723185784"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:21.854707 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.854536 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a68a48f7-d090-4eb7-af40-df304a84b319-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a68a48f7-d090-4eb7-af40-df304a84b319-istio-envoy\") pod \"a68a48f7-d090-4eb7-af40-df304a84b319\" (UID: \"a68a48f7-d090-4eb7-af40-df304a84b319\") " Oct 30 13:13:21.854707 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.854572 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1830fbd9-7c51-40d0-b6df-49a723185784-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1830fbd9-7c51-40d0-b6df-49a723185784-istiod-ca-cert\") pod \"1830fbd9-7c51-40d0-b6df-49a723185784\" (UID: \"1830fbd9-7c51-40d0-b6df-49a723185784\") " Oct 30 13:13:21.854707 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.854603 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1830fbd9-7c51-40d0-b6df-49a723185784-workload-socket" pod "1830fbd9-7c51-40d0-b6df-49a723185784" (UID: "1830fbd9-7c51-40d0-b6df-49a723185784"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:21.854707 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.854630 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1830fbd9-7c51-40d0-b6df-49a723185784-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1830fbd9-7c51-40d0-b6df-49a723185784-istio-data\") pod \"1830fbd9-7c51-40d0-b6df-49a723185784\" (UID: \"1830fbd9-7c51-40d0-b6df-49a723185784\") " Oct 30 13:13:21.854707 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.854678 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1830fbd9-7c51-40d0-b6df-49a723185784-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1830fbd9-7c51-40d0-b6df-49a723185784-istio-envoy\") pod \"1830fbd9-7c51-40d0-b6df-49a723185784\" (UID: \"1830fbd9-7c51-40d0-b6df-49a723185784\") " Oct 30 13:13:21.855310 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.854743 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a68a48f7-d090-4eb7-af40-df304a84b319-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a68a48f7-d090-4eb7-af40-df304a84b319-istio-podinfo\") pod \"a68a48f7-d090-4eb7-af40-df304a84b319\" (UID: \"a68a48f7-d090-4eb7-af40-df304a84b319\") " Oct 30 13:13:21.855310 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.854750 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a68a48f7-d090-4eb7-af40-df304a84b319-credential-socket" pod "a68a48f7-d090-4eb7-af40-df304a84b319" (UID: "a68a48f7-d090-4eb7-af40-df304a84b319"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:21.855310 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.854904 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a68a48f7-d090-4eb7-af40-df304a84b319-istio-data" pod "a68a48f7-d090-4eb7-af40-df304a84b319" (UID: "a68a48f7-d090-4eb7-af40-df304a84b319"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:21.855310 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.854924 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1830fbd9-7c51-40d0-b6df-49a723185784-istio-data" pod "1830fbd9-7c51-40d0-b6df-49a723185784" (UID: "1830fbd9-7c51-40d0-b6df-49a723185784"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:21.855310 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.854935 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1830fbd9-7c51-40d0-b6df-49a723185784-istio-token\" (UniqueName: \"kubernetes.io/projected/1830fbd9-7c51-40d0-b6df-49a723185784-istio-token\") pod \"1830fbd9-7c51-40d0-b6df-49a723185784\" (UID: \"1830fbd9-7c51-40d0-b6df-49a723185784\") " Oct 30 13:13:21.855310 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.854987 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a68a48f7-d090-4eb7-af40-df304a84b319-kube-api-access-rqbv7\" (UniqueName: \"kubernetes.io/projected/a68a48f7-d090-4eb7-af40-df304a84b319-kube-api-access-rqbv7\") pod \"a68a48f7-d090-4eb7-af40-df304a84b319\" (UID: \"a68a48f7-d090-4eb7-af40-df304a84b319\") " Oct 30 13:13:21.855310 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.855208 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a68a48f7-d090-4eb7-af40-df304a84b319-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a68a48f7-d090-4eb7-af40-df304a84b319-istiod-ca-cert\") pod \"a68a48f7-d090-4eb7-af40-df304a84b319\" (UID: \"a68a48f7-d090-4eb7-af40-df304a84b319\") " Oct 30 13:13:21.855728 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.855526 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlgtg\" (UniqueName: \"kubernetes.io/projected/f661cb7b-0cb2-42c2-a0a9-96dc71421c3e-kube-api-access-hlgtg\") pod \"maas-api-revoke-sub-26235bd2-9qhbp\" (UID: \"f661cb7b-0cb2-42c2-a0a9-96dc71421c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26235bd2-9qhbp" Oct 30 13:13:21.855793 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.855732 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f661cb7b-0cb2-42c2-a0a9-96dc71421c3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26235bd2-9qhbp\" (UID: \"f661cb7b-0cb2-42c2-a0a9-96dc71421c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26235bd2-9qhbp" Oct 30 13:13:21.855878 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.855857 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f661cb7b-0cb2-42c2-a0a9-96dc71421c3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26235bd2-9qhbp\" (UID: \"f661cb7b-0cb2-42c2-a0a9-96dc71421c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26235bd2-9qhbp" Oct 30 13:13:21.855983 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.855960 5922 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1830fbd9-7c51-40d0-b6df-49a723185784-credential-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:21.855983 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.855977 5922 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a68a48f7-d090-4eb7-af40-df304a84b319-workload-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:21.856101 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.855992 5922 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1830fbd9-7c51-40d0-b6df-49a723185784-workload-certs\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:21.856101 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.856005 5922 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1830fbd9-7c51-40d0-b6df-49a723185784-workload-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:21.856101 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.855209 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1830fbd9-7c51-40d0-b6df-49a723185784-istiod-ca-cert" pod "1830fbd9-7c51-40d0-b6df-49a723185784" (UID: "1830fbd9-7c51-40d0-b6df-49a723185784"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:13:21.856101 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.855677 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a68a48f7-d090-4eb7-af40-df304a84b319-workload-certs" pod "a68a48f7-d090-4eb7-af40-df304a84b319" (UID: "a68a48f7-d090-4eb7-af40-df304a84b319"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:21.856101 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.855978 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a68a48f7-d090-4eb7-af40-df304a84b319-istiod-ca-cert" pod "a68a48f7-d090-4eb7-af40-df304a84b319" (UID: "a68a48f7-d090-4eb7-af40-df304a84b319"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:13:21.856101 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.856069 5922 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a68a48f7-d090-4eb7-af40-df304a84b319-istio-data\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:21.856101 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.856094 5922 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a68a48f7-d090-4eb7-af40-df304a84b319-credential-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:21.856445 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.856109 5922 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1830fbd9-7c51-40d0-b6df-49a723185784-istio-data\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:21.857649 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.857581 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1830fbd9-7c51-40d0-b6df-49a723185784-kube-api-access-2frl6" pod "1830fbd9-7c51-40d0-b6df-49a723185784" (UID: "1830fbd9-7c51-40d0-b6df-49a723185784"). InnerVolumeSpecName "kube-api-access-2frl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:21.857926 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.857903 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a68a48f7-d090-4eb7-af40-df304a84b319-istio-envoy" pod "a68a48f7-d090-4eb7-af40-df304a84b319" (UID: "a68a48f7-d090-4eb7-af40-df304a84b319"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:21.858354 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.858234 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1830fbd9-7c51-40d0-b6df-49a723185784-istio-token" pod "1830fbd9-7c51-40d0-b6df-49a723185784" (UID: "1830fbd9-7c51-40d0-b6df-49a723185784"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:21.858354 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.858345 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1830fbd9-7c51-40d0-b6df-49a723185784-istio-envoy" pod "1830fbd9-7c51-40d0-b6df-49a723185784" (UID: "1830fbd9-7c51-40d0-b6df-49a723185784"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:21.858738 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.858713 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a68a48f7-d090-4eb7-af40-df304a84b319-kube-api-access-rqbv7" pod "a68a48f7-d090-4eb7-af40-df304a84b319" (UID: "a68a48f7-d090-4eb7-af40-df304a84b319"). InnerVolumeSpecName "kube-api-access-rqbv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:21.859276 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.859256 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a68a48f7-d090-4eb7-af40-df304a84b319-istio-token" pod "a68a48f7-d090-4eb7-af40-df304a84b319" (UID: "a68a48f7-d090-4eb7-af40-df304a84b319"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:21.859962 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.859935 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1830fbd9-7c51-40d0-b6df-49a723185784-istio-podinfo" pod "1830fbd9-7c51-40d0-b6df-49a723185784" (UID: "1830fbd9-7c51-40d0-b6df-49a723185784"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:13:21.859962 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.859943 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a68a48f7-d090-4eb7-af40-df304a84b319-istio-podinfo" pod "a68a48f7-d090-4eb7-af40-df304a84b319" (UID: "a68a48f7-d090-4eb7-af40-df304a84b319"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:13:21.956679 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.956650 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlgtg\" (UniqueName: \"kubernetes.io/projected/f661cb7b-0cb2-42c2-a0a9-96dc71421c3e-kube-api-access-hlgtg\") pod \"maas-api-revoke-sub-26235bd2-9qhbp\" (UID: \"f661cb7b-0cb2-42c2-a0a9-96dc71421c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26235bd2-9qhbp" Oct 30 13:13:21.956964 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.956941 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f661cb7b-0cb2-42c2-a0a9-96dc71421c3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26235bd2-9qhbp\" (UID: \"f661cb7b-0cb2-42c2-a0a9-96dc71421c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26235bd2-9qhbp" Oct 30 13:13:21.957202 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.957187 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f661cb7b-0cb2-42c2-a0a9-96dc71421c3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26235bd2-9qhbp\" (UID: \"f661cb7b-0cb2-42c2-a0a9-96dc71421c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26235bd2-9qhbp" Oct 30 13:13:21.957484 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.957466 5922 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1830fbd9-7c51-40d0-b6df-49a723185784-istio-envoy\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:21.957639 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.957607 5922 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a68a48f7-d090-4eb7-af40-df304a84b319-istio-podinfo\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:21.957752 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.957740 5922 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1830fbd9-7c51-40d0-b6df-49a723185784-istio-token\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:21.957855 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.957842 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqbv7\" (UniqueName: \"kubernetes.io/projected/a68a48f7-d090-4eb7-af40-df304a84b319-kube-api-access-rqbv7\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:21.957952 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.957940 5922 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a68a48f7-d090-4eb7-af40-df304a84b319-istiod-ca-cert\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:21.958031 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.958020 5922 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a68a48f7-d090-4eb7-af40-df304a84b319-workload-certs\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:21.958095 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.957947 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f661cb7b-0cb2-42c2-a0a9-96dc71421c3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26235bd2-9qhbp\" (UID: \"f661cb7b-0cb2-42c2-a0a9-96dc71421c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26235bd2-9qhbp" Oct 30 13:13:21.958158 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.958090 5922 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a68a48f7-d090-4eb7-af40-df304a84b319-istio-token\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:21.958212 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.958205 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2frl6\" (UniqueName: \"kubernetes.io/projected/1830fbd9-7c51-40d0-b6df-49a723185784-kube-api-access-2frl6\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:21.958263 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.958256 5922 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1830fbd9-7c51-40d0-b6df-49a723185784-istio-podinfo\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:21.958318 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.958312 5922 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a68a48f7-d090-4eb7-af40-df304a84b319-istio-envoy\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:21.958369 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.958363 5922 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1830fbd9-7c51-40d0-b6df-49a723185784-istiod-ca-cert\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:21.961792 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.961763 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f661cb7b-0cb2-42c2-a0a9-96dc71421c3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26235bd2-9qhbp\" (UID: \"f661cb7b-0cb2-42c2-a0a9-96dc71421c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26235bd2-9qhbp" Oct 30 13:13:21.981271 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:21.981248 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlgtg\" (UniqueName: \"kubernetes.io/projected/f661cb7b-0cb2-42c2-a0a9-96dc71421c3e-kube-api-access-hlgtg\") pod \"maas-api-revoke-sub-26235bd2-9qhbp\" (UID: \"f661cb7b-0cb2-42c2-a0a9-96dc71421c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26235bd2-9qhbp" Oct 30 13:13:22.008369 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.008098 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26235bd2-9qhbp" Oct 30 13:13:22.048177 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.048108 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06776799-9gfcb"] Oct 30 13:13:22.063496 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.063425 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06776799-9gfcb"] Oct 30 13:13:22.105347 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.105317 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" Oct 30 13:13:22.169091 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.169049 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-workload-socket\") pod \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\" (UID: \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\") " Oct 30 13:13:22.169292 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.169119 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-istio-podinfo\") pod \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\" (UID: \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\") " Oct 30 13:13:22.169292 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.169148 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-workload-certs\") pod \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\" (UID: \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\") " Oct 30 13:13:22.169292 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.169251 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-istiod-ca-cert\") pod \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\" (UID: \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\") " Oct 30 13:13:22.169292 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.169284 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-istio-envoy\") pod \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\" (UID: \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\") " Oct 30 13:13:22.169479 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.169327 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-istio-token\" (UniqueName: \"kubernetes.io/projected/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-istio-token\") pod \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\" (UID: \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\") " Oct 30 13:13:22.169479 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.169358 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-credential-socket\") pod \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\" (UID: \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\") " Oct 30 13:13:22.169479 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.169391 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-kube-api-access-jmrmn\" (UniqueName: \"kubernetes.io/projected/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-kube-api-access-jmrmn\") pod \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\" (UID: \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\") " Oct 30 13:13:22.169479 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.169409 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-istio-data\") pod \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\" (UID: \"b607f9a7-6ef2-4ea2-bdb4-352b2312966d\") " Oct 30 13:13:22.169674 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.169558 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-workload-certs" pod "b607f9a7-6ef2-4ea2-bdb4-352b2312966d" (UID: "b607f9a7-6ef2-4ea2-bdb4-352b2312966d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:22.173362 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.169760 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-istio-data" pod "b607f9a7-6ef2-4ea2-bdb4-352b2312966d" (UID: "b607f9a7-6ef2-4ea2-bdb4-352b2312966d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:22.173362 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.169827 5922 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-workload-certs\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:22.173362 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.169965 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-istiod-ca-cert" pod "b607f9a7-6ef2-4ea2-bdb4-352b2312966d" (UID: "b607f9a7-6ef2-4ea2-bdb4-352b2312966d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:13:22.173362 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.170193 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-workload-socket" pod "b607f9a7-6ef2-4ea2-bdb4-352b2312966d" (UID: "b607f9a7-6ef2-4ea2-bdb4-352b2312966d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:22.173362 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.170443 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-credential-socket" pod "b607f9a7-6ef2-4ea2-bdb4-352b2312966d" (UID: "b607f9a7-6ef2-4ea2-bdb4-352b2312966d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:22.173362 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.172685 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-istio-token" pod "b607f9a7-6ef2-4ea2-bdb4-352b2312966d" (UID: "b607f9a7-6ef2-4ea2-bdb4-352b2312966d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:22.173362 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.172943 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-kube-api-access-jmrmn" pod "b607f9a7-6ef2-4ea2-bdb4-352b2312966d" (UID: "b607f9a7-6ef2-4ea2-bdb4-352b2312966d"). InnerVolumeSpecName "kube-api-access-jmrmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:22.174028 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.173856 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-istio-envoy" pod "b607f9a7-6ef2-4ea2-bdb4-352b2312966d" (UID: "b607f9a7-6ef2-4ea2-bdb4-352b2312966d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:22.175401 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.175360 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-istio-podinfo" pod "b607f9a7-6ef2-4ea2-bdb4-352b2312966d" (UID: "b607f9a7-6ef2-4ea2-bdb4-352b2312966d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:13:22.176975 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.176951 5922 generic.go:356] "Generic (PLEG): container finished" podID="b607f9a7-6ef2-4ea2-bdb4-352b2312966d" containerID="7905db159264c464bfdd28ddb020ee47a98d62f341fb933cb51740e930840481" exitCode=0 Oct 30 13:13:22.177069 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.177014 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" Oct 30 13:13:22.177069 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.177039 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" event={"ID":"b607f9a7-6ef2-4ea2-bdb4-352b2312966d","Type":"ContainerDied","Data":"7905db159264c464bfdd28ddb020ee47a98d62f341fb933cb51740e930840481"} Oct 30 13:13:22.177170 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.177135 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb" event={"ID":"b607f9a7-6ef2-4ea2-bdb4-352b2312966d","Type":"ContainerDied","Data":"5fb19caa85339fb4ec8e6ebe82d456484cc9b386f339201021ff911b97ff4c7c"} Oct 30 13:13:22.177170 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.177167 5922 scope.go:122] "RemoveContainer" containerID="7905db159264c464bfdd28ddb020ee47a98d62f341fb933cb51740e930840481" Oct 30 13:13:22.178879 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.178847 5922 generic.go:356] "Generic (PLEG): container finished" podID="1830fbd9-7c51-40d0-b6df-49a723185784" containerID="629fe8b00af9f7879c5086af3fa9c9cb0bf1701e99a7af8dd6a41b1a9204ed05" exitCode=0 Oct 30 13:13:22.178984 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.178911 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" Oct 30 13:13:22.178984 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.178941 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" event={"ID":"1830fbd9-7c51-40d0-b6df-49a723185784","Type":"ContainerDied","Data":"629fe8b00af9f7879c5086af3fa9c9cb0bf1701e99a7af8dd6a41b1a9204ed05"} Oct 30 13:13:22.178984 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.178980 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24" event={"ID":"1830fbd9-7c51-40d0-b6df-49a723185784","Type":"ContainerDied","Data":"cbfa5e1a2db9048521eecfbd94901f28693d8cff184d8b2472230b78d5cf1b2d"} Oct 30 13:13:22.182985 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.182933 5922 generic.go:356] "Generic (PLEG): container finished" podID="a68a48f7-d090-4eb7-af40-df304a84b319" containerID="170d1cca2c2bcc213a867eb484cd2cc7bbc0897f9156bff5b2620ebc42cbf4d7" exitCode=0 Oct 30 13:13:22.183085 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.183016 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" event={"ID":"a68a48f7-d090-4eb7-af40-df304a84b319","Type":"ContainerDied","Data":"170d1cca2c2bcc213a867eb484cd2cc7bbc0897f9156bff5b2620ebc42cbf4d7"} Oct 30 13:13:22.183085 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.183077 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" event={"ID":"a68a48f7-d090-4eb7-af40-df304a84b319","Type":"ContainerDied","Data":"35ec412147d64b21fd80049f3fbea856135a8b313e2235dedd48bc1c3f0bb5c7"} Oct 30 13:13:22.189003 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.183282 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8" Oct 30 13:13:22.189003 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.188949 5922 scope.go:122] "RemoveContainer" containerID="7905db159264c464bfdd28ddb020ee47a98d62f341fb933cb51740e930840481" Oct 30 13:13:22.189246 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:13:22.189228 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7905db159264c464bfdd28ddb020ee47a98d62f341fb933cb51740e930840481\": container with ID starting with 7905db159264c464bfdd28ddb020ee47a98d62f341fb933cb51740e930840481 not found: ID does not exist" containerID="7905db159264c464bfdd28ddb020ee47a98d62f341fb933cb51740e930840481" Oct 30 13:13:22.189288 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.189252 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7905db159264c464bfdd28ddb020ee47a98d62f341fb933cb51740e930840481"} err="failed to get container status \"7905db159264c464bfdd28ddb020ee47a98d62f341fb933cb51740e930840481\": rpc error: code = NotFound desc = could not find container \"7905db159264c464bfdd28ddb020ee47a98d62f341fb933cb51740e930840481\": container with ID starting with 7905db159264c464bfdd28ddb020ee47a98d62f341fb933cb51740e930840481 not found: ID does not exist" Oct 30 13:13:22.189288 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.189268 5922 scope.go:122] "RemoveContainer" containerID="629fe8b00af9f7879c5086af3fa9c9cb0bf1701e99a7af8dd6a41b1a9204ed05" Oct 30 13:13:22.199839 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.199821 5922 scope.go:122] "RemoveContainer" containerID="629fe8b00af9f7879c5086af3fa9c9cb0bf1701e99a7af8dd6a41b1a9204ed05" Oct 30 13:13:22.200122 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:13:22.200102 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"629fe8b00af9f7879c5086af3fa9c9cb0bf1701e99a7af8dd6a41b1a9204ed05\": container with ID starting with 629fe8b00af9f7879c5086af3fa9c9cb0bf1701e99a7af8dd6a41b1a9204ed05 not found: ID does not exist" containerID="629fe8b00af9f7879c5086af3fa9c9cb0bf1701e99a7af8dd6a41b1a9204ed05" Oct 30 13:13:22.200202 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.200126 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"629fe8b00af9f7879c5086af3fa9c9cb0bf1701e99a7af8dd6a41b1a9204ed05"} err="failed to get container status \"629fe8b00af9f7879c5086af3fa9c9cb0bf1701e99a7af8dd6a41b1a9204ed05\": rpc error: code = NotFound desc = could not find container \"629fe8b00af9f7879c5086af3fa9c9cb0bf1701e99a7af8dd6a41b1a9204ed05\": container with ID starting with 629fe8b00af9f7879c5086af3fa9c9cb0bf1701e99a7af8dd6a41b1a9204ed05 not found: ID does not exist" Oct 30 13:13:22.200202 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.200151 5922 scope.go:122] "RemoveContainer" containerID="170d1cca2c2bcc213a867eb484cd2cc7bbc0897f9156bff5b2620ebc42cbf4d7" Oct 30 13:13:22.211899 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.211878 5922 scope.go:122] "RemoveContainer" containerID="933f6ee0846072e68eb36ab006c8cc87cea85081333069c8fb6816d7f3650b59" Oct 30 13:13:22.221051 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.221033 5922 scope.go:122] "RemoveContainer" containerID="170d1cca2c2bcc213a867eb484cd2cc7bbc0897f9156bff5b2620ebc42cbf4d7" Oct 30 13:13:22.221284 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:13:22.221266 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"170d1cca2c2bcc213a867eb484cd2cc7bbc0897f9156bff5b2620ebc42cbf4d7\": container with ID starting with 170d1cca2c2bcc213a867eb484cd2cc7bbc0897f9156bff5b2620ebc42cbf4d7 not found: ID does not exist" containerID="170d1cca2c2bcc213a867eb484cd2cc7bbc0897f9156bff5b2620ebc42cbf4d7" Oct 30 13:13:22.221336 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.221289 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"170d1cca2c2bcc213a867eb484cd2cc7bbc0897f9156bff5b2620ebc42cbf4d7"} err="failed to get container status \"170d1cca2c2bcc213a867eb484cd2cc7bbc0897f9156bff5b2620ebc42cbf4d7\": rpc error: code = NotFound desc = could not find container \"170d1cca2c2bcc213a867eb484cd2cc7bbc0897f9156bff5b2620ebc42cbf4d7\": container with ID starting with 170d1cca2c2bcc213a867eb484cd2cc7bbc0897f9156bff5b2620ebc42cbf4d7 not found: ID does not exist" Oct 30 13:13:22.221336 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.221306 5922 scope.go:122] "RemoveContainer" containerID="933f6ee0846072e68eb36ab006c8cc87cea85081333069c8fb6816d7f3650b59" Oct 30 13:13:22.221510 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:13:22.221497 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"933f6ee0846072e68eb36ab006c8cc87cea85081333069c8fb6816d7f3650b59\": container with ID starting with 933f6ee0846072e68eb36ab006c8cc87cea85081333069c8fb6816d7f3650b59 not found: ID does not exist" containerID="933f6ee0846072e68eb36ab006c8cc87cea85081333069c8fb6816d7f3650b59" Oct 30 13:13:22.221554 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.221512 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"933f6ee0846072e68eb36ab006c8cc87cea85081333069c8fb6816d7f3650b59"} err="failed to get container status \"933f6ee0846072e68eb36ab006c8cc87cea85081333069c8fb6816d7f3650b59\": rpc error: code = NotFound desc = could not find container \"933f6ee0846072e68eb36ab006c8cc87cea85081333069c8fb6816d7f3650b59\": container with ID starting with 933f6ee0846072e68eb36ab006c8cc87cea85081333069c8fb6816d7f3650b59 not found: ID does not exist" Oct 30 13:13:22.255535 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.255511 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24"] Oct 30 13:13:22.261385 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.261364 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-nnz24"] Oct 30 13:13:22.270990 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.270969 5922 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-istiod-ca-cert\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:22.270990 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.270990 5922 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-istio-envoy\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:22.271121 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.270999 5922 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-istio-token\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:22.271121 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.271007 5922 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-credential-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:22.271121 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.271018 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmrmn\" (UniqueName: \"kubernetes.io/projected/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-kube-api-access-jmrmn\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:22.271121 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.271030 5922 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-istio-data\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:22.271121 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.271043 5922 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-workload-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:22.271121 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.271054 5922 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b607f9a7-6ef2-4ea2-bdb4-352b2312966d-istio-podinfo\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:22.302133 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.302111 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb"] Oct 30 13:13:22.307926 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.307904 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-6pwdb"] Oct 30 13:13:22.350056 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.350031 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8"] Oct 30 13:13:22.370104 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.370079 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-d77t8"] Oct 30 13:13:22.390631 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:22.390598 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26235bd2-9qhbp"] Oct 30 13:13:22.392291 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:13:22.392265 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf661cb7b_0cb2_42c2_a0a9_96dc71421c3e.slice/crio-8bd4c0f972eefba612b57b810aaf18f41a00257fcb53f07955b8447b85b12745 WatchSource:0}: Error finding container 8bd4c0f972eefba612b57b810aaf18f41a00257fcb53f07955b8447b85b12745: Status 404 returned error can't find the container with id 8bd4c0f972eefba612b57b810aaf18f41a00257fcb53f07955b8447b85b12745 Oct 30 13:13:23.193060 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:23.193023 5922 generic.go:356] "Generic (PLEG): container finished" podID="f661cb7b-0cb2-42c2-a0a9-96dc71421c3e" containerID="cec58b9d2860dfe8ec21d2e3b6596249a65d1ba9f5a76b2ad2c4ae542c61fcf0" exitCode=0 Oct 30 13:13:23.193454 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:23.193059 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26235bd2-9qhbp" event={"ID":"f661cb7b-0cb2-42c2-a0a9-96dc71421c3e","Type":"ContainerDied","Data":"cec58b9d2860dfe8ec21d2e3b6596249a65d1ba9f5a76b2ad2c4ae542c61fcf0"} Oct 30 13:13:23.193454 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:23.193120 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26235bd2-9qhbp" event={"ID":"f661cb7b-0cb2-42c2-a0a9-96dc71421c3e","Type":"ContainerStarted","Data":"8bd4c0f972eefba612b57b810aaf18f41a00257fcb53f07955b8447b85b12745"} Oct 30 13:13:23.552173 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:13:23.552131 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:13:23.644325 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:23.644294 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1830fbd9-7c51-40d0-b6df-49a723185784" path="/var/lib/kubelet/pods/1830fbd9-7c51-40d0-b6df-49a723185784/volumes" Oct 30 13:13:23.644740 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:23.644726 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a68a48f7-d090-4eb7-af40-df304a84b319" path="/var/lib/kubelet/pods/a68a48f7-d090-4eb7-af40-df304a84b319/volumes" Oct 30 13:13:23.645140 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:23.645127 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b607f9a7-6ef2-4ea2-bdb4-352b2312966d" path="/var/lib/kubelet/pods/b607f9a7-6ef2-4ea2-bdb4-352b2312966d/volumes" Oct 30 13:13:23.645839 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:23.645822 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4deafea-cb6a-4c4a-b820-fcc454879ffa" path="/var/lib/kubelet/pods/c4deafea-cb6a-4c4a-b820-fcc454879ffa/volumes" Oct 30 13:13:24.288984 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:24.288959 5922 scope.go:122] "RemoveContainer" containerID="6da806f8a77b0e89131b12ece6fe4d5faff08b0c0afe4df77527b4623e77bb07" Oct 30 13:13:24.303056 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:24.303006 5922 scope.go:122] "RemoveContainer" containerID="05f3b66efdb1445b8c49c480c917130f001150a198662694f29edd2f8c234823" Oct 30 13:13:24.313512 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:24.313488 5922 scope.go:122] "RemoveContainer" containerID="dbd4f36d944fd7fa66bba1fdb3699f5f7656ac7f12afae4dc4e201a99f1b4560" Oct 30 13:13:24.325354 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:24.325322 5922 scope.go:122] "RemoveContainer" containerID="a981deff70dc47bde8f69479c6412db86f286d6ec6bd893da134fa0f03c33421" Oct 30 13:13:24.337048 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:24.337022 5922 scope.go:122] "RemoveContainer" containerID="a2b6a02a9de636e02e42af6069b9525e97760c746e4dc9c188d09a08e2b7e4c9" Oct 30 13:13:24.345918 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:24.345900 5922 scope.go:122] "RemoveContainer" containerID="afd8d18a813e42506379cef3857a06d5e147b9dbcc79c40a25c918ca279828dd" Oct 30 13:13:24.368847 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:24.368832 5922 scope.go:122] "RemoveContainer" containerID="9834914807a62848a58c7ecec261a51ff8e0619cfd1e27a2c7d161bf53eff8be" Oct 30 13:13:24.373744 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:24.373723 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26235bd2-9qhbp" Oct 30 13:13:24.391156 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:24.391135 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f661cb7b-0cb2-42c2-a0a9-96dc71421c3e-kube-api-access-hlgtg\" (UniqueName: \"kubernetes.io/projected/f661cb7b-0cb2-42c2-a0a9-96dc71421c3e-kube-api-access-hlgtg\") pod \"f661cb7b-0cb2-42c2-a0a9-96dc71421c3e\" (UID: \"f661cb7b-0cb2-42c2-a0a9-96dc71421c3e\") " Oct 30 13:13:24.391247 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:24.391224 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f661cb7b-0cb2-42c2-a0a9-96dc71421c3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f661cb7b-0cb2-42c2-a0a9-96dc71421c3e-maas-api-xks-ca\") pod \"f661cb7b-0cb2-42c2-a0a9-96dc71421c3e\" (UID: \"f661cb7b-0cb2-42c2-a0a9-96dc71421c3e\") " Oct 30 13:13:24.391286 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:24.391268 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f661cb7b-0cb2-42c2-a0a9-96dc71421c3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f661cb7b-0cb2-42c2-a0a9-96dc71421c3e-maas-api-openshift-ca\") pod \"f661cb7b-0cb2-42c2-a0a9-96dc71421c3e\" (UID: \"f661cb7b-0cb2-42c2-a0a9-96dc71421c3e\") " Oct 30 13:13:24.391649 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:24.391607 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f661cb7b-0cb2-42c2-a0a9-96dc71421c3e-maas-api-openshift-ca" pod "f661cb7b-0cb2-42c2-a0a9-96dc71421c3e" (UID: "f661cb7b-0cb2-42c2-a0a9-96dc71421c3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:13:24.393262 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:24.393239 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f661cb7b-0cb2-42c2-a0a9-96dc71421c3e-maas-api-xks-ca" pod "f661cb7b-0cb2-42c2-a0a9-96dc71421c3e" (UID: "f661cb7b-0cb2-42c2-a0a9-96dc71421c3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:13:24.393352 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:24.393332 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f661cb7b-0cb2-42c2-a0a9-96dc71421c3e-kube-api-access-hlgtg" pod "f661cb7b-0cb2-42c2-a0a9-96dc71421c3e" (UID: "f661cb7b-0cb2-42c2-a0a9-96dc71421c3e"). InnerVolumeSpecName "kube-api-access-hlgtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:24.495333 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:24.493290 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlgtg\" (UniqueName: \"kubernetes.io/projected/f661cb7b-0cb2-42c2-a0a9-96dc71421c3e-kube-api-access-hlgtg\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:24.495333 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:24.493327 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f661cb7b-0cb2-42c2-a0a9-96dc71421c3e-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:24.495333 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:24.493343 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f661cb7b-0cb2-42c2-a0a9-96dc71421c3e-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:24.525090 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:24.525060 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:24.643118 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:24.643072 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:25.046747 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:25.046711 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ea6e5c7-gwnpb"] Oct 30 13:13:25.061341 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:25.061305 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ea6e5c7-gwnpb"] Oct 30 13:13:25.212316 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:25.212285 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26235bd2-9qhbp" Oct 30 13:13:25.212487 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:25.212321 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26235bd2-9qhbp" event={"ID":"f661cb7b-0cb2-42c2-a0a9-96dc71421c3e","Type":"ContainerDied","Data":"8bd4c0f972eefba612b57b810aaf18f41a00257fcb53f07955b8447b85b12745"} Oct 30 13:13:25.212487 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:25.212401 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bd4c0f972eefba612b57b810aaf18f41a00257fcb53f07955b8447b85b12745" Oct 30 13:13:25.644165 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:25.644133 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25935958-aa4d-4c13-96de-3532eaec0422" path="/var/lib/kubelet/pods/25935958-aa4d-4c13-96de-3532eaec0422/volumes" Oct 30 13:13:26.030665 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:26.030603 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:26.418987 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:26.418910 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-175831-6f46fb6d59-ltklr" podUID="81a0354c-9716-4cc3-a902-b81908879ab3" containerName="payload-pre-processing" containerID="cri-o://991f0f1fdb622d1e6103d8df2c36e47883f508b34bed8c4da8a818f6a5451191" gracePeriod=25 Oct 30 13:13:27.043870 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:27.043835 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-688053e2-gtvtz"] Oct 30 13:13:27.055812 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:27.055785 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-688053e2-gtvtz"] Oct 30 13:13:27.643683 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:27.643651 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ce92897-0aa3-4c92-a50d-3a231fae4165" path="/var/lib/kubelet/pods/5ce92897-0aa3-4c92-a50d-3a231fae4165/volumes" Oct 30 13:13:29.036979 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:29.036945 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbfecdd2-kw7hg"] Oct 30 13:13:29.048995 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:29.048967 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbfecdd2-kw7hg"] Oct 30 13:13:29.648807 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:29.648767 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3b9423e-208d-45c0-bd24-bec761499e36" path="/var/lib/kubelet/pods/b3b9423e-208d-45c0-bd24-bec761499e36/volumes" Oct 30 13:13:30.035381 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:30.035351 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8462ff11-htt87"] Oct 30 13:13:30.047395 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:30.047368 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8462ff11-htt87"] Oct 30 13:13:30.717122 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:30.717084 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" podUID="47517373-759d-4eeb-ab5e-6150b7e82bb7" containerName="main" containerID="cri-o://2150b5f29f2bbc8413a20cba99ce41eeef399d503c0c3629db635a9cbb6edc9c" gracePeriod=45 Oct 30 13:13:30.979037 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:30.978979 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" Oct 30 13:13:31.046776 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:31.046744 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-925dbcef-gx9pd"] Oct 30 13:13:31.051452 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:31.051420 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-925dbcef-gx9pd"] Oct 30 13:13:31.055079 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:31.055048 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47517373-759d-4eeb-ab5e-6150b7e82bb7-kube-api-access-vjkbr\" (UniqueName: \"kubernetes.io/projected/47517373-759d-4eeb-ab5e-6150b7e82bb7-kube-api-access-vjkbr\") pod \"47517373-759d-4eeb-ab5e-6150b7e82bb7\" (UID: \"47517373-759d-4eeb-ab5e-6150b7e82bb7\") " Oct 30 13:13:31.055207 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:31.055116 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47517373-759d-4eeb-ab5e-6150b7e82bb7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47517373-759d-4eeb-ab5e-6150b7e82bb7-tmp-dir\") pod \"47517373-759d-4eeb-ab5e-6150b7e82bb7\" (UID: \"47517373-759d-4eeb-ab5e-6150b7e82bb7\") " Oct 30 13:13:31.055207 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:31.055163 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47517373-759d-4eeb-ab5e-6150b7e82bb7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47517373-759d-4eeb-ab5e-6150b7e82bb7-model-cache\") pod \"47517373-759d-4eeb-ab5e-6150b7e82bb7\" (UID: \"47517373-759d-4eeb-ab5e-6150b7e82bb7\") " Oct 30 13:13:31.055283 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:31.055241 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47517373-759d-4eeb-ab5e-6150b7e82bb7-home\" (UniqueName: \"kubernetes.io/empty-dir/47517373-759d-4eeb-ab5e-6150b7e82bb7-home\") pod \"47517373-759d-4eeb-ab5e-6150b7e82bb7\" (UID: \"47517373-759d-4eeb-ab5e-6150b7e82bb7\") " Oct 30 13:13:31.055359 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:31.055337 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47517373-759d-4eeb-ab5e-6150b7e82bb7-tls-certs\" (UniqueName: \"kubernetes.io/secret/47517373-759d-4eeb-ab5e-6150b7e82bb7-tls-certs\") pod \"47517373-759d-4eeb-ab5e-6150b7e82bb7\" (UID: \"47517373-759d-4eeb-ab5e-6150b7e82bb7\") " Oct 30 13:13:31.055500 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:31.055388 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47517373-759d-4eeb-ab5e-6150b7e82bb7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/47517373-759d-4eeb-ab5e-6150b7e82bb7-dshm\") pod \"47517373-759d-4eeb-ab5e-6150b7e82bb7\" (UID: \"47517373-759d-4eeb-ab5e-6150b7e82bb7\") " Oct 30 13:13:31.056368 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:31.056346 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47517373-759d-4eeb-ab5e-6150b7e82bb7-model-cache" pod "47517373-759d-4eeb-ab5e-6150b7e82bb7" (UID: "47517373-759d-4eeb-ab5e-6150b7e82bb7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:31.056518 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:31.056395 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47517373-759d-4eeb-ab5e-6150b7e82bb7-home" pod "47517373-759d-4eeb-ab5e-6150b7e82bb7" (UID: "47517373-759d-4eeb-ab5e-6150b7e82bb7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:31.057322 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:31.057296 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47517373-759d-4eeb-ab5e-6150b7e82bb7-tmp-dir" pod "47517373-759d-4eeb-ab5e-6150b7e82bb7" (UID: "47517373-759d-4eeb-ab5e-6150b7e82bb7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:31.058967 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:31.058942 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47517373-759d-4eeb-ab5e-6150b7e82bb7-kube-api-access-vjkbr" pod "47517373-759d-4eeb-ab5e-6150b7e82bb7" (UID: "47517373-759d-4eeb-ab5e-6150b7e82bb7"). InnerVolumeSpecName "kube-api-access-vjkbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:31.059526 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:31.059497 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47517373-759d-4eeb-ab5e-6150b7e82bb7-tls-certs" pod "47517373-759d-4eeb-ab5e-6150b7e82bb7" (UID: "47517373-759d-4eeb-ab5e-6150b7e82bb7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:13:31.061545 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:31.061520 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47517373-759d-4eeb-ab5e-6150b7e82bb7-dshm" pod "47517373-759d-4eeb-ab5e-6150b7e82bb7" (UID: "47517373-759d-4eeb-ab5e-6150b7e82bb7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:31.156805 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:31.156770 5922 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47517373-759d-4eeb-ab5e-6150b7e82bb7-tls-certs\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:31.156805 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:31.156799 5922 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47517373-759d-4eeb-ab5e-6150b7e82bb7-dshm\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:31.156805 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:31.156810 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjkbr\" (UniqueName: \"kubernetes.io/projected/47517373-759d-4eeb-ab5e-6150b7e82bb7-kube-api-access-vjkbr\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:31.157023 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:31.156819 5922 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47517373-759d-4eeb-ab5e-6150b7e82bb7-tmp-dir\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:31.157023 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:31.156828 5922 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47517373-759d-4eeb-ab5e-6150b7e82bb7-model-cache\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:31.157023 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:31.156835 5922 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47517373-759d-4eeb-ab5e-6150b7e82bb7-home\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:31.264292 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:31.264208 5922 generic.go:356] "Generic (PLEG): container finished" podID="47517373-759d-4eeb-ab5e-6150b7e82bb7" containerID="2150b5f29f2bbc8413a20cba99ce41eeef399d503c0c3629db635a9cbb6edc9c" exitCode=0 Oct 30 13:13:31.264292 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:31.264269 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" Oct 30 13:13:31.264489 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:31.264292 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" event={"ID":"47517373-759d-4eeb-ab5e-6150b7e82bb7","Type":"ContainerDied","Data":"2150b5f29f2bbc8413a20cba99ce41eeef399d503c0c3629db635a9cbb6edc9c"} Oct 30 13:13:31.264489 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:31.264333 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" event={"ID":"47517373-759d-4eeb-ab5e-6150b7e82bb7","Type":"ContainerDied","Data":"bd3b756465a6d07daa10c1d4ca5f346bf13a93240ddc93ac7c0b3cc38fd6d250"} Oct 30 13:13:31.264489 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:31.264351 5922 scope.go:122] "RemoveContainer" containerID="2150b5f29f2bbc8413a20cba99ce41eeef399d503c0c3629db635a9cbb6edc9c" Oct 30 13:13:31.274948 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:31.274933 5922 scope.go:122] "RemoveContainer" containerID="2150b5f29f2bbc8413a20cba99ce41eeef399d503c0c3629db635a9cbb6edc9c" Oct 30 13:13:31.275204 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:13:31.275186 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2150b5f29f2bbc8413a20cba99ce41eeef399d503c0c3629db635a9cbb6edc9c\": container with ID starting with 2150b5f29f2bbc8413a20cba99ce41eeef399d503c0c3629db635a9cbb6edc9c not found: ID does not exist" containerID="2150b5f29f2bbc8413a20cba99ce41eeef399d503c0c3629db635a9cbb6edc9c" Oct 30 13:13:31.275253 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:31.275213 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2150b5f29f2bbc8413a20cba99ce41eeef399d503c0c3629db635a9cbb6edc9c"} err="failed to get container status \"2150b5f29f2bbc8413a20cba99ce41eeef399d503c0c3629db635a9cbb6edc9c\": rpc error: code = NotFound desc = could not find container \"2150b5f29f2bbc8413a20cba99ce41eeef399d503c0c3629db635a9cbb6edc9c\": container with ID starting with 2150b5f29f2bbc8413a20cba99ce41eeef399d503c0c3629db635a9cbb6edc9c not found: ID does not exist" Oct 30 13:13:31.310465 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:31.310440 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-175831-6f46fb6d59-ltklr" podUID="81a0354c-9716-4cc3-a902-b81908879ab3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:13:31.319225 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:31.319206 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2"] Oct 30 13:13:31.321948 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:31.321922 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2"] Oct 30 13:13:31.644215 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:31.644156 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="055fc516-1042-4221-aff3-b07b975bec3d" path="/var/lib/kubelet/pods/055fc516-1042-4221-aff3-b07b975bec3d/volumes" Oct 30 13:13:31.644559 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:31.644546 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1eee749f-1f8c-4dc8-80b8-e0288d52e2b1" path="/var/lib/kubelet/pods/1eee749f-1f8c-4dc8-80b8-e0288d52e2b1/volumes" Oct 30 13:13:31.644972 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:31.644957 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47517373-759d-4eeb-ab5e-6150b7e82bb7" path="/var/lib/kubelet/pods/47517373-759d-4eeb-ab5e-6150b7e82bb7/volumes" Oct 30 13:13:31.926663 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:31.926559 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-2-w5-175831-kserve-64db498f4b-d76n2" podUID="47517373-759d-4eeb-ab5e-6150b7e82bb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:13:32.582947 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:32.582911 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:33.035092 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:33.035062 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5101680f-4lrqb"] Oct 30 13:13:33.047178 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:33.047152 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5101680f-4lrqb"] Oct 30 13:13:33.606464 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:33.606426 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:33.643701 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:33.643675 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="348fae6c-e0b1-47de-a4c5-c2db9e1cd49e" path="/var/lib/kubelet/pods/348fae6c-e0b1-47de-a4c5-c2db9e1cd49e/volumes" Oct 30 13:13:34.034314 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:34.034284 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d66ae31-rhpxq"] Oct 30 13:13:34.045876 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:34.045850 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d66ae31-rhpxq"] Oct 30 13:13:34.528197 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:34.528170 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:13:35.646926 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:35.646885 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1e8049c-5832-4f22-b078-e29a51557bfb" path="/var/lib/kubelet/pods/d1e8049c-5832-4f22-b078-e29a51557bfb/volumes" Oct 30 13:13:36.897428 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:36.897382 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:37.034240 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:37.034213 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08bacf2d-st7nj"] Oct 30 13:13:37.046068 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:37.046045 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08bacf2d-st7nj"] Oct 30 13:13:37.409571 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:37.409538 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:37.526511 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:37.526480 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-175831-lsz8g"] Oct 30 13:13:37.534526 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:37.534465 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:37.540722 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:37.540685 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-175831-lsz8g"] Oct 30 13:13:37.643892 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:37.643862 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b98661f-1136-483c-b836-85d753238487" path="/var/lib/kubelet/pods/1b98661f-1136-483c-b836-85d753238487/volumes" Oct 30 13:13:37.644300 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:37.644287 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="207be5eb-d83a-477b-af9a-550a8abe5e73" path="/var/lib/kubelet/pods/207be5eb-d83a-477b-af9a-550a8abe5e73/volumes" Oct 30 13:13:38.032247 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:38.032213 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:41.310309 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:41.310274 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-175831-6f46fb6d59-ltklr" podUID="81a0354c-9716-4cc3-a902-b81908879ab3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:13:44.442564 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:44.442529 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:13:44.749988 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:44.749916 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:44.885884 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:44.885843 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:45.070507 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:45.070428 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66fb3c85-7v8nn"] Oct 30 13:13:45.087368 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:45.087330 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66fb3c85-7v8nn"] Oct 30 13:13:45.094342 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:45.094303 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn"] Oct 30 13:13:45.094567 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:45.094516 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" podUID="c61d2f25-754d-4538-b154-b9095bffe96f" containerName="istio-proxy" containerID="cri-o://2185979abda3a74fd942f13b3aa4f6a5289a25f8a8812fe823a9390e92563364" gracePeriod=30 Oct 30 13:13:45.111477 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:45.111448 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k"] Oct 30 13:13:45.111691 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:45.111670 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" podUID="aced22f0-8827-4cad-b04b-85ddb56490af" containerName="istio-proxy" containerID="cri-o://9ccb1060f3bde50596ab6c0df7e12a58730d5a310da1f4299aa84a3a67358581" gracePeriod=30 Oct 30 13:13:45.135745 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:45.135714 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf"] Oct 30 13:13:45.135937 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:45.135912 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" podUID="aee561c5-3aa7-4021-9d3e-05e7636d19be" containerName="istio-proxy" containerID="cri-o://c46065deda8846da87056fdbf8225588e3a49592466cad9ef3c9931d49bb883f" gracePeriod=30 Oct 30 13:13:45.137904 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:45.137878 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:45.643906 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:45.643871 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e420d739-6af6-408d-bb35-b5af95de2c5f" path="/var/lib/kubelet/pods/e420d739-6af6-408d-bb35-b5af95de2c5f/volumes" Oct 30 13:13:46.444522 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:46.444493 5922 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:13:46.444662 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:46.444564 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" podUID="c61d2f25-754d-4538-b154-b9095bffe96f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:13:46.685815 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:46.685759 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-175831-6f46fb6d59-ltklr" Oct 30 13:13:46.829561 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:46.829526 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81a0354c-9716-4cc3-a902-b81908879ab3-kube-api-access-cwjww\" (UniqueName: \"kubernetes.io/projected/81a0354c-9716-4cc3-a902-b81908879ab3-kube-api-access-cwjww\") pod \"81a0354c-9716-4cc3-a902-b81908879ab3\" (UID: \"81a0354c-9716-4cc3-a902-b81908879ab3\") " Oct 30 13:13:46.829884 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:46.829568 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81a0354c-9716-4cc3-a902-b81908879ab3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81a0354c-9716-4cc3-a902-b81908879ab3-plugins-config-volume\") pod \"81a0354c-9716-4cc3-a902-b81908879ab3\" (UID: \"81a0354c-9716-4cc3-a902-b81908879ab3\") " Oct 30 13:13:46.829956 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:46.829923 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81a0354c-9716-4cc3-a902-b81908879ab3-plugins-config-volume" pod "81a0354c-9716-4cc3-a902-b81908879ab3" (UID: "81a0354c-9716-4cc3-a902-b81908879ab3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:13:46.830054 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:46.830034 5922 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81a0354c-9716-4cc3-a902-b81908879ab3-plugins-config-volume\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:46.831550 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:46.831525 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81a0354c-9716-4cc3-a902-b81908879ab3-kube-api-access-cwjww" pod "81a0354c-9716-4cc3-a902-b81908879ab3" (UID: "81a0354c-9716-4cc3-a902-b81908879ab3"). InnerVolumeSpecName "kube-api-access-cwjww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:46.930752 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:46.930728 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwjww\" (UniqueName: \"kubernetes.io/projected/81a0354c-9716-4cc3-a902-b81908879ab3-kube-api-access-cwjww\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:47.033972 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:47.033943 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be022317-9k5dl"] Oct 30 13:13:47.045603 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:47.045578 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be022317-9k5dl"] Oct 30 13:13:47.421716 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:47.421629 5922 generic.go:356] "Generic (PLEG): container finished" podID="81a0354c-9716-4cc3-a902-b81908879ab3" containerID="991f0f1fdb622d1e6103d8df2c36e47883f508b34bed8c4da8a818f6a5451191" exitCode=0 Oct 30 13:13:47.421716 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:47.421662 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-175831-6f46fb6d59-ltklr" event={"ID":"81a0354c-9716-4cc3-a902-b81908879ab3","Type":"ContainerDied","Data":"991f0f1fdb622d1e6103d8df2c36e47883f508b34bed8c4da8a818f6a5451191"} Oct 30 13:13:47.421716 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:47.421687 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-175831-6f46fb6d59-ltklr" Oct 30 13:13:47.421988 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:47.421734 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-175831-6f46fb6d59-ltklr" event={"ID":"81a0354c-9716-4cc3-a902-b81908879ab3","Type":"ContainerDied","Data":"f7acf4bb2de15bede613705a5ebdafe17b0e4be7d9f47a6f0d88ca4bf786068c"} Oct 30 13:13:47.421988 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:47.421766 5922 scope.go:122] "RemoveContainer" containerID="991f0f1fdb622d1e6103d8df2c36e47883f508b34bed8c4da8a818f6a5451191" Oct 30 13:13:47.432453 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:47.432434 5922 scope.go:122] "RemoveContainer" containerID="991f0f1fdb622d1e6103d8df2c36e47883f508b34bed8c4da8a818f6a5451191" Oct 30 13:13:47.432735 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:13:47.432713 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"991f0f1fdb622d1e6103d8df2c36e47883f508b34bed8c4da8a818f6a5451191\": container with ID starting with 991f0f1fdb622d1e6103d8df2c36e47883f508b34bed8c4da8a818f6a5451191 not found: ID does not exist" containerID="991f0f1fdb622d1e6103d8df2c36e47883f508b34bed8c4da8a818f6a5451191" Oct 30 13:13:47.432809 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:47.432742 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"991f0f1fdb622d1e6103d8df2c36e47883f508b34bed8c4da8a818f6a5451191"} err="failed to get container status \"991f0f1fdb622d1e6103d8df2c36e47883f508b34bed8c4da8a818f6a5451191\": rpc error: code = NotFound desc = could not find container \"991f0f1fdb622d1e6103d8df2c36e47883f508b34bed8c4da8a818f6a5451191\": container with ID starting with 991f0f1fdb622d1e6103d8df2c36e47883f508b34bed8c4da8a818f6a5451191 not found: ID does not exist" Oct 30 13:13:47.476603 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:47.476576 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-175831-6f46fb6d59-ltklr"] Oct 30 13:13:47.488573 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:47.488549 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-175831-6f46fb6d59-ltklr"] Oct 30 13:13:47.643724 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:47.643692 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ddaea9f-d694-4294-bbc3-3d718ae8abd1" path="/var/lib/kubelet/pods/2ddaea9f-d694-4294-bbc3-3d718ae8abd1/volumes" Oct 30 13:13:47.644157 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:47.644134 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81a0354c-9716-4cc3-a902-b81908879ab3" path="/var/lib/kubelet/pods/81a0354c-9716-4cc3-a902-b81908879ab3/volumes" Oct 30 13:13:47.970060 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:47.970026 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:49.034465 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:49.034432 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56c47c05-d7wdp"] Oct 30 13:13:49.046316 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:49.046293 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56c47c05-d7wdp"] Oct 30 13:13:49.643737 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:49.643705 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c69ddfca-b346-4241-b6f6-5a08c31db00b" path="/var/lib/kubelet/pods/c69ddfca-b346-4241-b6f6-5a08c31db00b/volumes" Oct 30 13:13:50.416064 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.416038 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" Oct 30 13:13:50.425698 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.425591 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" Oct 30 13:13:50.450754 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.450542 5922 generic.go:356] "Generic (PLEG): container finished" podID="aced22f0-8827-4cad-b04b-85ddb56490af" containerID="9ccb1060f3bde50596ab6c0df7e12a58730d5a310da1f4299aa84a3a67358581" exitCode=0 Oct 30 13:13:50.450754 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.450599 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" Oct 30 13:13:50.450754 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.450648 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" event={"ID":"aced22f0-8827-4cad-b04b-85ddb56490af","Type":"ContainerDied","Data":"9ccb1060f3bde50596ab6c0df7e12a58730d5a310da1f4299aa84a3a67358581"} Oct 30 13:13:50.450754 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.450700 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k" event={"ID":"aced22f0-8827-4cad-b04b-85ddb56490af","Type":"ContainerDied","Data":"bafb68be112012c37e6e86b30c59964d6f555e02150bbfde796f4be7a44323db"} Oct 30 13:13:50.450754 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.450730 5922 scope.go:122] "RemoveContainer" containerID="9ccb1060f3bde50596ab6c0df7e12a58730d5a310da1f4299aa84a3a67358581" Oct 30 13:13:50.453202 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.453034 5922 generic.go:356] "Generic (PLEG): container finished" podID="c61d2f25-754d-4538-b154-b9095bffe96f" containerID="2185979abda3a74fd942f13b3aa4f6a5289a25f8a8812fe823a9390e92563364" exitCode=0 Oct 30 13:13:50.453202 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.453075 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" Oct 30 13:13:50.453202 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.453131 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" event={"ID":"c61d2f25-754d-4538-b154-b9095bffe96f","Type":"ContainerDied","Data":"2185979abda3a74fd942f13b3aa4f6a5289a25f8a8812fe823a9390e92563364"} Oct 30 13:13:50.453202 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.453166 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn" event={"ID":"c61d2f25-754d-4538-b154-b9095bffe96f","Type":"ContainerDied","Data":"36952ec869368eef6359d65005d5f58fec9c3575ad3471f33cadce7a8827fe1b"} Oct 30 13:13:50.454939 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.454920 5922 generic.go:356] "Generic (PLEG): container finished" podID="aee561c5-3aa7-4021-9d3e-05e7636d19be" containerID="c46065deda8846da87056fdbf8225588e3a49592466cad9ef3c9931d49bb883f" exitCode=0 Oct 30 13:13:50.455070 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.455011 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" event={"ID":"aee561c5-3aa7-4021-9d3e-05e7636d19be","Type":"ContainerDied","Data":"c46065deda8846da87056fdbf8225588e3a49592466cad9ef3c9931d49bb883f"} Oct 30 13:13:50.468398 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.466314 5922 scope.go:122] "RemoveContainer" containerID="622569e4d5eaebd803a4ee6adfb75551939cb04face002ebb9dd9a900633ff33" Oct 30 13:13:50.490008 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.489915 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" Oct 30 13:13:50.494095 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.494020 5922 scope.go:122] "RemoveContainer" containerID="9ccb1060f3bde50596ab6c0df7e12a58730d5a310da1f4299aa84a3a67358581" Oct 30 13:13:50.497471 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:13:50.495923 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ccb1060f3bde50596ab6c0df7e12a58730d5a310da1f4299aa84a3a67358581\": container with ID starting with 9ccb1060f3bde50596ab6c0df7e12a58730d5a310da1f4299aa84a3a67358581 not found: ID does not exist" containerID="9ccb1060f3bde50596ab6c0df7e12a58730d5a310da1f4299aa84a3a67358581" Oct 30 13:13:50.497471 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.495955 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ccb1060f3bde50596ab6c0df7e12a58730d5a310da1f4299aa84a3a67358581"} err="failed to get container status \"9ccb1060f3bde50596ab6c0df7e12a58730d5a310da1f4299aa84a3a67358581\": rpc error: code = NotFound desc = could not find container \"9ccb1060f3bde50596ab6c0df7e12a58730d5a310da1f4299aa84a3a67358581\": container with ID starting with 9ccb1060f3bde50596ab6c0df7e12a58730d5a310da1f4299aa84a3a67358581 not found: ID does not exist" Oct 30 13:13:50.497471 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.495973 5922 scope.go:122] "RemoveContainer" containerID="622569e4d5eaebd803a4ee6adfb75551939cb04face002ebb9dd9a900633ff33" Oct 30 13:13:50.497471 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:13:50.496271 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"622569e4d5eaebd803a4ee6adfb75551939cb04face002ebb9dd9a900633ff33\": container with ID starting with 622569e4d5eaebd803a4ee6adfb75551939cb04face002ebb9dd9a900633ff33 not found: ID does not exist" containerID="622569e4d5eaebd803a4ee6adfb75551939cb04face002ebb9dd9a900633ff33" Oct 30 13:13:50.497471 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.496323 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"622569e4d5eaebd803a4ee6adfb75551939cb04face002ebb9dd9a900633ff33"} err="failed to get container status \"622569e4d5eaebd803a4ee6adfb75551939cb04face002ebb9dd9a900633ff33\": rpc error: code = NotFound desc = could not find container \"622569e4d5eaebd803a4ee6adfb75551939cb04face002ebb9dd9a900633ff33\": container with ID starting with 622569e4d5eaebd803a4ee6adfb75551939cb04face002ebb9dd9a900633ff33 not found: ID does not exist" Oct 30 13:13:50.497471 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.496340 5922 scope.go:122] "RemoveContainer" containerID="2185979abda3a74fd942f13b3aa4f6a5289a25f8a8812fe823a9390e92563364" Oct 30 13:13:50.507968 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.507943 5922 scope.go:122] "RemoveContainer" containerID="2185979abda3a74fd942f13b3aa4f6a5289a25f8a8812fe823a9390e92563364" Oct 30 13:13:50.508337 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:13:50.508313 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2185979abda3a74fd942f13b3aa4f6a5289a25f8a8812fe823a9390e92563364\": container with ID starting with 2185979abda3a74fd942f13b3aa4f6a5289a25f8a8812fe823a9390e92563364 not found: ID does not exist" containerID="2185979abda3a74fd942f13b3aa4f6a5289a25f8a8812fe823a9390e92563364" Oct 30 13:13:50.508455 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.508341 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2185979abda3a74fd942f13b3aa4f6a5289a25f8a8812fe823a9390e92563364"} err="failed to get container status \"2185979abda3a74fd942f13b3aa4f6a5289a25f8a8812fe823a9390e92563364\": rpc error: code = NotFound desc = could not find container \"2185979abda3a74fd942f13b3aa4f6a5289a25f8a8812fe823a9390e92563364\": container with ID starting with 2185979abda3a74fd942f13b3aa4f6a5289a25f8a8812fe823a9390e92563364 not found: ID does not exist" Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.564424 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aee561c5-3aa7-4021-9d3e-05e7636d19be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee561c5-3aa7-4021-9d3e-05e7636d19be-workload-socket\") pod \"aee561c5-3aa7-4021-9d3e-05e7636d19be\" (UID: \"aee561c5-3aa7-4021-9d3e-05e7636d19be\") " Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.564485 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c61d2f25-754d-4538-b154-b9095bffe96f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c61d2f25-754d-4538-b154-b9095bffe96f-istio-envoy\") pod \"c61d2f25-754d-4538-b154-b9095bffe96f\" (UID: \"c61d2f25-754d-4538-b154-b9095bffe96f\") " Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.564526 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aced22f0-8827-4cad-b04b-85ddb56490af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aced22f0-8827-4cad-b04b-85ddb56490af-istiod-ca-cert\") pod \"aced22f0-8827-4cad-b04b-85ddb56490af\" (UID: \"aced22f0-8827-4cad-b04b-85ddb56490af\") " Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.564562 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aced22f0-8827-4cad-b04b-85ddb56490af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aced22f0-8827-4cad-b04b-85ddb56490af-istio-podinfo\") pod \"aced22f0-8827-4cad-b04b-85ddb56490af\" (UID: \"aced22f0-8827-4cad-b04b-85ddb56490af\") " Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.564592 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c61d2f25-754d-4538-b154-b9095bffe96f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c61d2f25-754d-4538-b154-b9095bffe96f-workload-certs\") pod \"c61d2f25-754d-4538-b154-b9095bffe96f\" (UID: \"c61d2f25-754d-4538-b154-b9095bffe96f\") " Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.564656 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aee561c5-3aa7-4021-9d3e-05e7636d19be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aee561c5-3aa7-4021-9d3e-05e7636d19be-istiod-ca-cert\") pod \"aee561c5-3aa7-4021-9d3e-05e7636d19be\" (UID: \"aee561c5-3aa7-4021-9d3e-05e7636d19be\") " Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.564697 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aee561c5-3aa7-4021-9d3e-05e7636d19be-kube-api-access-8sm57\" (UniqueName: \"kubernetes.io/projected/aee561c5-3aa7-4021-9d3e-05e7636d19be-kube-api-access-8sm57\") pod \"aee561c5-3aa7-4021-9d3e-05e7636d19be\" (UID: \"aee561c5-3aa7-4021-9d3e-05e7636d19be\") " Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.564759 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aee561c5-3aa7-4021-9d3e-05e7636d19be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aee561c5-3aa7-4021-9d3e-05e7636d19be-istio-podinfo\") pod \"aee561c5-3aa7-4021-9d3e-05e7636d19be\" (UID: \"aee561c5-3aa7-4021-9d3e-05e7636d19be\") " Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.564814 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aced22f0-8827-4cad-b04b-85ddb56490af-istio-token\" (UniqueName: \"kubernetes.io/projected/aced22f0-8827-4cad-b04b-85ddb56490af-istio-token\") pod \"aced22f0-8827-4cad-b04b-85ddb56490af\" (UID: \"aced22f0-8827-4cad-b04b-85ddb56490af\") " Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.564847 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aee561c5-3aa7-4021-9d3e-05e7636d19be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aee561c5-3aa7-4021-9d3e-05e7636d19be-istio-envoy\") pod \"aee561c5-3aa7-4021-9d3e-05e7636d19be\" (UID: \"aee561c5-3aa7-4021-9d3e-05e7636d19be\") " Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.564866 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c61d2f25-754d-4538-b154-b9095bffe96f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c61d2f25-754d-4538-b154-b9095bffe96f-istio-data\") pod \"c61d2f25-754d-4538-b154-b9095bffe96f\" (UID: \"c61d2f25-754d-4538-b154-b9095bffe96f\") " Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.564896 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aced22f0-8827-4cad-b04b-85ddb56490af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aced22f0-8827-4cad-b04b-85ddb56490af-workload-certs\") pod \"aced22f0-8827-4cad-b04b-85ddb56490af\" (UID: \"aced22f0-8827-4cad-b04b-85ddb56490af\") " Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.564934 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aee561c5-3aa7-4021-9d3e-05e7636d19be-istio-token\" (UniqueName: \"kubernetes.io/projected/aee561c5-3aa7-4021-9d3e-05e7636d19be-istio-token\") pod \"aee561c5-3aa7-4021-9d3e-05e7636d19be\" (UID: \"aee561c5-3aa7-4021-9d3e-05e7636d19be\") " Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.564953 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c61d2f25-754d-4538-b154-b9095bffe96f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c61d2f25-754d-4538-b154-b9095bffe96f-istiod-ca-cert\") pod \"c61d2f25-754d-4538-b154-b9095bffe96f\" (UID: \"c61d2f25-754d-4538-b154-b9095bffe96f\") " Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.564986 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c61d2f25-754d-4538-b154-b9095bffe96f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c61d2f25-754d-4538-b154-b9095bffe96f-credential-socket\") pod \"c61d2f25-754d-4538-b154-b9095bffe96f\" (UID: \"c61d2f25-754d-4538-b154-b9095bffe96f\") " Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.565012 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c61d2f25-754d-4538-b154-b9095bffe96f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c61d2f25-754d-4538-b154-b9095bffe96f-workload-socket\") pod \"c61d2f25-754d-4538-b154-b9095bffe96f\" (UID: \"c61d2f25-754d-4538-b154-b9095bffe96f\") " Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.565031 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aced22f0-8827-4cad-b04b-85ddb56490af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aced22f0-8827-4cad-b04b-85ddb56490af-istio-data\") pod \"aced22f0-8827-4cad-b04b-85ddb56490af\" (UID: \"aced22f0-8827-4cad-b04b-85ddb56490af\") " Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.565055 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aee561c5-3aa7-4021-9d3e-05e7636d19be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aee561c5-3aa7-4021-9d3e-05e7636d19be-workload-certs\") pod \"aee561c5-3aa7-4021-9d3e-05e7636d19be\" (UID: \"aee561c5-3aa7-4021-9d3e-05e7636d19be\") " Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.565080 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aee561c5-3aa7-4021-9d3e-05e7636d19be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee561c5-3aa7-4021-9d3e-05e7636d19be-credential-socket\") pod \"aee561c5-3aa7-4021-9d3e-05e7636d19be\" (UID: \"aee561c5-3aa7-4021-9d3e-05e7636d19be\") " Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.565101 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c61d2f25-754d-4538-b154-b9095bffe96f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c61d2f25-754d-4538-b154-b9095bffe96f-istio-podinfo\") pod \"c61d2f25-754d-4538-b154-b9095bffe96f\" (UID: \"c61d2f25-754d-4538-b154-b9095bffe96f\") " Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.565122 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aced22f0-8827-4cad-b04b-85ddb56490af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aced22f0-8827-4cad-b04b-85ddb56490af-istio-envoy\") pod \"aced22f0-8827-4cad-b04b-85ddb56490af\" (UID: \"aced22f0-8827-4cad-b04b-85ddb56490af\") " Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.565161 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c61d2f25-754d-4538-b154-b9095bffe96f-istio-token\" (UniqueName: \"kubernetes.io/projected/c61d2f25-754d-4538-b154-b9095bffe96f-istio-token\") pod \"c61d2f25-754d-4538-b154-b9095bffe96f\" (UID: \"c61d2f25-754d-4538-b154-b9095bffe96f\") " Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.565179 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aced22f0-8827-4cad-b04b-85ddb56490af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aced22f0-8827-4cad-b04b-85ddb56490af-credential-socket\") pod \"aced22f0-8827-4cad-b04b-85ddb56490af\" (UID: \"aced22f0-8827-4cad-b04b-85ddb56490af\") " Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.565197 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aee561c5-3aa7-4021-9d3e-05e7636d19be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aee561c5-3aa7-4021-9d3e-05e7636d19be-istio-data\") pod \"aee561c5-3aa7-4021-9d3e-05e7636d19be\" (UID: \"aee561c5-3aa7-4021-9d3e-05e7636d19be\") " Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.565224 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c61d2f25-754d-4538-b154-b9095bffe96f-kube-api-access-jbvm7\" (UniqueName: \"kubernetes.io/projected/c61d2f25-754d-4538-b154-b9095bffe96f-kube-api-access-jbvm7\") pod \"c61d2f25-754d-4538-b154-b9095bffe96f\" (UID: \"c61d2f25-754d-4538-b154-b9095bffe96f\") " Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.565252 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aced22f0-8827-4cad-b04b-85ddb56490af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aced22f0-8827-4cad-b04b-85ddb56490af-workload-socket\") pod \"aced22f0-8827-4cad-b04b-85ddb56490af\" (UID: \"aced22f0-8827-4cad-b04b-85ddb56490af\") " Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.565281 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aced22f0-8827-4cad-b04b-85ddb56490af-kube-api-access-r7mrp\" (UniqueName: \"kubernetes.io/projected/aced22f0-8827-4cad-b04b-85ddb56490af-kube-api-access-r7mrp\") pod \"aced22f0-8827-4cad-b04b-85ddb56490af\" (UID: \"aced22f0-8827-4cad-b04b-85ddb56490af\") " Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.566541 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aced22f0-8827-4cad-b04b-85ddb56490af-credential-socket" pod "aced22f0-8827-4cad-b04b-85ddb56490af" (UID: "aced22f0-8827-4cad-b04b-85ddb56490af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.567347 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aced22f0-8827-4cad-b04b-85ddb56490af-workload-socket" pod "aced22f0-8827-4cad-b04b-85ddb56490af" (UID: "aced22f0-8827-4cad-b04b-85ddb56490af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.568421 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee561c5-3aa7-4021-9d3e-05e7636d19be-istio-data" pod "aee561c5-3aa7-4021-9d3e-05e7636d19be" (UID: "aee561c5-3aa7-4021-9d3e-05e7636d19be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.568988 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aee561c5-3aa7-4021-9d3e-05e7636d19be-istiod-ca-cert" pod "aee561c5-3aa7-4021-9d3e-05e7636d19be" (UID: "aee561c5-3aa7-4021-9d3e-05e7636d19be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.569390 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aced22f0-8827-4cad-b04b-85ddb56490af-istiod-ca-cert" pod "aced22f0-8827-4cad-b04b-85ddb56490af" (UID: "aced22f0-8827-4cad-b04b-85ddb56490af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.569835 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee561c5-3aa7-4021-9d3e-05e7636d19be-workload-socket" pod "aee561c5-3aa7-4021-9d3e-05e7636d19be" (UID: "aee561c5-3aa7-4021-9d3e-05e7636d19be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.570391 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c61d2f25-754d-4538-b154-b9095bffe96f-workload-certs" pod "c61d2f25-754d-4538-b154-b9095bffe96f" (UID: "c61d2f25-754d-4538-b154-b9095bffe96f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.570580 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c61d2f25-754d-4538-b154-b9095bffe96f-istio-data" pod "c61d2f25-754d-4538-b154-b9095bffe96f" (UID: "c61d2f25-754d-4538-b154-b9095bffe96f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.572547 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c61d2f25-754d-4538-b154-b9095bffe96f-istio-token" pod "c61d2f25-754d-4538-b154-b9095bffe96f" (UID: "c61d2f25-754d-4538-b154-b9095bffe96f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.572620 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aced22f0-8827-4cad-b04b-85ddb56490af-istio-podinfo" pod "aced22f0-8827-4cad-b04b-85ddb56490af" (UID: "aced22f0-8827-4cad-b04b-85ddb56490af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.572861 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aced22f0-8827-4cad-b04b-85ddb56490af-workload-certs" pod "aced22f0-8827-4cad-b04b-85ddb56490af" (UID: "aced22f0-8827-4cad-b04b-85ddb56490af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.573051 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c61d2f25-754d-4538-b154-b9095bffe96f-workload-socket" pod "c61d2f25-754d-4538-b154-b9095bffe96f" (UID: "c61d2f25-754d-4538-b154-b9095bffe96f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.573407 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c61d2f25-754d-4538-b154-b9095bffe96f-istiod-ca-cert" pod "c61d2f25-754d-4538-b154-b9095bffe96f" (UID: "c61d2f25-754d-4538-b154-b9095bffe96f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.573713 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c61d2f25-754d-4538-b154-b9095bffe96f-credential-socket" pod "c61d2f25-754d-4538-b154-b9095bffe96f" (UID: "c61d2f25-754d-4538-b154-b9095bffe96f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.573896 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aced22f0-8827-4cad-b04b-85ddb56490af-istio-data" pod "aced22f0-8827-4cad-b04b-85ddb56490af" (UID: "aced22f0-8827-4cad-b04b-85ddb56490af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.574034 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee561c5-3aa7-4021-9d3e-05e7636d19be-workload-certs" pod "aee561c5-3aa7-4021-9d3e-05e7636d19be" (UID: "aee561c5-3aa7-4021-9d3e-05e7636d19be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.574210 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee561c5-3aa7-4021-9d3e-05e7636d19be-credential-socket" pod "aee561c5-3aa7-4021-9d3e-05e7636d19be" (UID: "aee561c5-3aa7-4021-9d3e-05e7636d19be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.576885 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aee561c5-3aa7-4021-9d3e-05e7636d19be-istio-podinfo" pod "aee561c5-3aa7-4021-9d3e-05e7636d19be" (UID: "aee561c5-3aa7-4021-9d3e-05e7636d19be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.579926 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aced22f0-8827-4cad-b04b-85ddb56490af-istio-envoy" pod "aced22f0-8827-4cad-b04b-85ddb56490af" (UID: "aced22f0-8827-4cad-b04b-85ddb56490af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.579960 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aee561c5-3aa7-4021-9d3e-05e7636d19be-kube-api-access-8sm57" pod "aee561c5-3aa7-4021-9d3e-05e7636d19be" (UID: "aee561c5-3aa7-4021-9d3e-05e7636d19be"). InnerVolumeSpecName "kube-api-access-8sm57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.579981 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aee561c5-3aa7-4021-9d3e-05e7636d19be-istio-token" pod "aee561c5-3aa7-4021-9d3e-05e7636d19be" (UID: "aee561c5-3aa7-4021-9d3e-05e7636d19be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.580058 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aced22f0-8827-4cad-b04b-85ddb56490af-istio-token" pod "aced22f0-8827-4cad-b04b-85ddb56490af" (UID: "aced22f0-8827-4cad-b04b-85ddb56490af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.580072 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c61d2f25-754d-4538-b154-b9095bffe96f-istio-envoy" pod "c61d2f25-754d-4538-b154-b9095bffe96f" (UID: "c61d2f25-754d-4538-b154-b9095bffe96f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:50.580637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.580569 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee561c5-3aa7-4021-9d3e-05e7636d19be-istio-envoy" pod "aee561c5-3aa7-4021-9d3e-05e7636d19be" (UID: "aee561c5-3aa7-4021-9d3e-05e7636d19be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:50.583757 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.581493 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c61d2f25-754d-4538-b154-b9095bffe96f-kube-api-access-jbvm7" pod "c61d2f25-754d-4538-b154-b9095bffe96f" (UID: "c61d2f25-754d-4538-b154-b9095bffe96f"). InnerVolumeSpecName "kube-api-access-jbvm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:50.583757 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.582765 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aced22f0-8827-4cad-b04b-85ddb56490af-kube-api-access-r7mrp" pod "aced22f0-8827-4cad-b04b-85ddb56490af" (UID: "aced22f0-8827-4cad-b04b-85ddb56490af"). InnerVolumeSpecName "kube-api-access-r7mrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:50.583757 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.582771 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c61d2f25-754d-4538-b154-b9095bffe96f-istio-podinfo" pod "c61d2f25-754d-4538-b154-b9095bffe96f" (UID: "c61d2f25-754d-4538-b154-b9095bffe96f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:13:50.666851 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.666764 5922 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aee561c5-3aa7-4021-9d3e-05e7636d19be-istio-podinfo\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.666851 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.666791 5922 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aced22f0-8827-4cad-b04b-85ddb56490af-istio-token\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.666851 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.666799 5922 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aee561c5-3aa7-4021-9d3e-05e7636d19be-istio-envoy\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.666851 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.666818 5922 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c61d2f25-754d-4538-b154-b9095bffe96f-istio-data\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.666851 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.666831 5922 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aced22f0-8827-4cad-b04b-85ddb56490af-workload-certs\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.666851 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.666842 5922 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aee561c5-3aa7-4021-9d3e-05e7636d19be-istio-token\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.666851 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.666854 5922 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c61d2f25-754d-4538-b154-b9095bffe96f-istiod-ca-cert\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.667352 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.666863 5922 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c61d2f25-754d-4538-b154-b9095bffe96f-credential-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.667352 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.666871 5922 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c61d2f25-754d-4538-b154-b9095bffe96f-workload-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.667352 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.666880 5922 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aced22f0-8827-4cad-b04b-85ddb56490af-istio-data\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.667352 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.666888 5922 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aee561c5-3aa7-4021-9d3e-05e7636d19be-workload-certs\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.667352 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.666897 5922 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee561c5-3aa7-4021-9d3e-05e7636d19be-credential-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.667352 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.666904 5922 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c61d2f25-754d-4538-b154-b9095bffe96f-istio-podinfo\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.667352 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.666913 5922 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aced22f0-8827-4cad-b04b-85ddb56490af-istio-envoy\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.667352 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.666920 5922 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c61d2f25-754d-4538-b154-b9095bffe96f-istio-token\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.667352 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.666927 5922 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aced22f0-8827-4cad-b04b-85ddb56490af-credential-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.667352 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.666935 5922 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aee561c5-3aa7-4021-9d3e-05e7636d19be-istio-data\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.667352 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.666943 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbvm7\" (UniqueName: \"kubernetes.io/projected/c61d2f25-754d-4538-b154-b9095bffe96f-kube-api-access-jbvm7\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.667352 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.666951 5922 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aced22f0-8827-4cad-b04b-85ddb56490af-workload-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.667352 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.666960 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7mrp\" (UniqueName: \"kubernetes.io/projected/aced22f0-8827-4cad-b04b-85ddb56490af-kube-api-access-r7mrp\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.667352 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.666969 5922 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee561c5-3aa7-4021-9d3e-05e7636d19be-workload-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.667352 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.666978 5922 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c61d2f25-754d-4538-b154-b9095bffe96f-istio-envoy\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.667352 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.666986 5922 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aced22f0-8827-4cad-b04b-85ddb56490af-istiod-ca-cert\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.667352 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.666994 5922 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aced22f0-8827-4cad-b04b-85ddb56490af-istio-podinfo\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.667352 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.667001 5922 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c61d2f25-754d-4538-b154-b9095bffe96f-workload-certs\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.667352 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.667010 5922 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aee561c5-3aa7-4021-9d3e-05e7636d19be-istiod-ca-cert\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.667352 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.667017 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sm57\" (UniqueName: \"kubernetes.io/projected/aee561c5-3aa7-4021-9d3e-05e7636d19be-kube-api-access-8sm57\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.817440 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.817404 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k"] Oct 30 13:13:50.821436 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.821406 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-h4b7k"] Oct 30 13:13:50.858964 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.858935 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn"] Oct 30 13:13:50.865365 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:50.865341 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-fqxwn"] Oct 30 13:13:51.516637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:51.513193 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" event={"ID":"aee561c5-3aa7-4021-9d3e-05e7636d19be","Type":"ContainerDied","Data":"9b70974994632fb095a8defd4430013ec523854e530f5a0f4f71a52f3e27c7c0"} Oct 30 13:13:51.516637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:51.513304 5922 scope.go:122] "RemoveContainer" containerID="c46065deda8846da87056fdbf8225588e3a49592466cad9ef3c9931d49bb883f" Oct 30 13:13:51.516637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:51.513468 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf" Oct 30 13:13:51.633805 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:51.633700 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf"] Oct 30 13:13:51.635083 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:51.635055 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hmbwf"] Oct 30 13:13:51.646766 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:51.646744 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aced22f0-8827-4cad-b04b-85ddb56490af" path="/var/lib/kubelet/pods/aced22f0-8827-4cad-b04b-85ddb56490af/volumes" Oct 30 13:13:51.647228 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:51.647215 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aee561c5-3aa7-4021-9d3e-05e7636d19be" path="/var/lib/kubelet/pods/aee561c5-3aa7-4021-9d3e-05e7636d19be/volumes" Oct 30 13:13:51.647653 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:51.647597 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c61d2f25-754d-4538-b154-b9095bffe96f" path="/var/lib/kubelet/pods/c61d2f25-754d-4538-b154-b9095bffe96f/volumes" Oct 30 13:13:52.049534 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:52.049506 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71e7d561-h96hh"] Oct 30 13:13:52.051845 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:52.051822 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71e7d561-h96hh"] Oct 30 13:13:52.448357 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:52.448318 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:52.752563 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:52.752394 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:53.018300 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:53.018212 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:53.045481 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:53.045447 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-555d7dd0-vtkp6"] Oct 30 13:13:53.054849 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:53.054821 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-555d7dd0-vtkp6"] Oct 30 13:13:53.227741 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:53.227697 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:53.553384 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:13:53.553349 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:13:53.643572 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:53.643544 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16ba72c8-5f1d-492a-8140-ee4171654b96" path="/var/lib/kubelet/pods/16ba72c8-5f1d-492a-8140-ee4171654b96/volumes" Oct 30 13:13:53.643997 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:53.643983 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fefc8aa7-6d48-40b2-8d64-44a1ad767c1a" path="/var/lib/kubelet/pods/fefc8aa7-6d48-40b2-8d64-44a1ad767c1a/volumes" Oct 30 13:13:55.427091 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:13:55.427056 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:00.044505 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:14:00.044470 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e3e4f23-5wnb5"] Oct 30 13:14:00.056400 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:14:00.056368 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e3e4f23-5wnb5"] Oct 30 13:14:00.747603 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:14:00.747547 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:00.859705 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:14:00.859672 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:01.643004 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:14:01.642974 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4931dc41-6e44-4365-ad2d-f0c6ae9a44d5" path="/var/lib/kubelet/pods/4931dc41-6e44-4365-ad2d-f0c6ae9a44d5/volumes" Oct 30 13:14:01.652275 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:14:01.652233 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:16.895919 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:14:16.895872 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:20.035552 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:14:20.035506 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93b6dd98-6nxm4"] Oct 30 13:14:20.047416 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:14:20.047387 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93b6dd98-6nxm4"] Oct 30 13:14:21.642899 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:14:21.642863 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47c3d395-2a73-4415-901c-a007b2d167ff" path="/var/lib/kubelet/pods/47c3d395-2a73-4415-901c-a007b2d167ff/volumes" Oct 30 13:14:22.922589 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:14:22.922525 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:23.551314 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:14:23.551279 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:14:24.553204 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:14:24.553175 5922 scope.go:122] "RemoveContainer" containerID="cc1794b035ca076e37a5542c22c52ce23336d89682c9a77a5db422bdcc88eb67" Oct 30 13:14:24.562622 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:14:24.562592 5922 scope.go:122] "RemoveContainer" containerID="806aa4480d659f24a872a769eb5d0e866fe714dd4f2086809e70d4091fcd6515" Oct 30 13:14:24.571363 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:14:24.571335 5922 scope.go:122] "RemoveContainer" containerID="a91d664160ce147cdbf0d8b2deb4c0cfd89355b3f2013e79192459f3aa4eb73b" Oct 30 13:14:24.579709 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:14:24.579693 5922 scope.go:122] "RemoveContainer" containerID="45b1a552fa78db38a1c997c438e902831f2b2a6412c31492c5484324b418710e" Oct 30 13:14:24.588574 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:14:24.588557 5922 scope.go:122] "RemoveContainer" containerID="c4c8ca3e9357ec8664ac8de299bcb171d9f7aa51db2a0cccca03468b026afb90" Oct 30 13:14:24.596622 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:14:24.596590 5922 scope.go:122] "RemoveContainer" containerID="80b183e3c01a9d4bf53c9646794d1830a9a483e0ce59758cf500b7d748003844" Oct 30 13:14:24.604360 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:14:24.604344 5922 scope.go:122] "RemoveContainer" containerID="20896b15adbf980433327fe263fac620de8abab6eff0004b0d5a0b4f3fdf6f91" Oct 30 13:14:24.612126 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:14:24.612109 5922 scope.go:122] "RemoveContainer" containerID="e3f9f578c707db53651a403ee001f6b32e904ebd02a19c94c9fb82342bc9463c" Oct 30 13:14:24.620226 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:14:24.620212 5922 scope.go:122] "RemoveContainer" containerID="c3ada22ba5c2915a0332b29b037517c2437df1036cfe42e14823324979659d72" Oct 30 13:14:24.628666 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:14:24.628585 5922 scope.go:122] "RemoveContainer" containerID="5f1644984bcd48f7c34ca25f5e12b91575fa35dc87a25139c7c7a8b82c2b2abc" Oct 30 13:14:24.636769 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:14:24.636744 5922 scope.go:122] "RemoveContainer" containerID="6565ef857cce9880b37d7d74e4d3199ac01756395112e2f1fbf6429ec77602a2" Oct 30 13:14:24.644547 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:14:24.644528 5922 scope.go:122] "RemoveContainer" containerID="66bfe534c8e9d2e3560bda1d20002abcf93ca6b7e703e5e4328e132895234ea0" Oct 30 13:14:24.652273 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:14:24.652250 5922 scope.go:122] "RemoveContainer" containerID="4f44d264324f44b0238491c456cf8a5eff3f65b9bcd21abec22b0d52d7bf0639" Oct 30 13:14:24.786510 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:14:24.786476 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:33.040801 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:14:33.040761 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-470f78b0-ncbmp"] Oct 30 13:14:33.052707 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:14:33.052679 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-470f78b0-ncbmp"] Oct 30 13:14:33.643100 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:14:33.643067 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="954859db-438e-4108-b8fd-e4b0535b1998" path="/var/lib/kubelet/pods/954859db-438e-4108-b8fd-e4b0535b1998/volumes" Oct 30 13:14:33.728481 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:14:33.728449 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:53.552348 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:14:53.552310 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:14:56.006330 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:14:56.006295 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:15:01.778391 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:01.778360 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cplvq"] Oct 30 13:15:01.778789 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:01.778771 5922 cpu_manager.go:414] "containerMap: removing container" podUID="47517373-759d-4eeb-ab5e-6150b7e82bb7" containerName="main" Oct 30 13:15:01.778789 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:01.778784 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47517373-759d-4eeb-ab5e-6150b7e82bb7" containerName="main" Oct 30 13:15:01.778868 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:01.778804 5922 cpu_manager.go:414] "containerMap: removing container" podUID="a68a48f7-d090-4eb7-af40-df304a84b319" containerName="istio-proxy" Oct 30 13:15:01.778868 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:01.778810 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a68a48f7-d090-4eb7-af40-df304a84b319" containerName="istio-proxy" Oct 30 13:15:01.778868 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:01.778825 5922 cpu_manager.go:414] "containerMap: removing container" podUID="aee561c5-3aa7-4021-9d3e-05e7636d19be" containerName="istio-proxy" Oct 30 13:15:01.778868 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:01.778831 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aee561c5-3aa7-4021-9d3e-05e7636d19be" containerName="istio-proxy" Oct 30 13:15:01.778868 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:01.778843 5922 cpu_manager.go:414] "containerMap: removing container" podUID="f661cb7b-0cb2-42c2-a0a9-96dc71421c3e" containerName="revoke-keys" Oct 30 13:15:01.778868 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:01.778848 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f661cb7b-0cb2-42c2-a0a9-96dc71421c3e" containerName="revoke-keys" Oct 30 13:15:01.778868 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:01.778856 5922 cpu_manager.go:414] "containerMap: removing container" podUID="c61d2f25-754d-4538-b154-b9095bffe96f" containerName="istio-proxy" Oct 30 13:15:01.778868 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:01.778861 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c61d2f25-754d-4538-b154-b9095bffe96f" containerName="istio-proxy" Oct 30 13:15:01.778868 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:01.778868 5922 cpu_manager.go:414] "containerMap: removing container" podUID="b607f9a7-6ef2-4ea2-bdb4-352b2312966d" containerName="istio-proxy" Oct 30 13:15:01.779167 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:01.778873 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b607f9a7-6ef2-4ea2-bdb4-352b2312966d" containerName="istio-proxy" Oct 30 13:15:01.779167 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:01.778886 5922 cpu_manager.go:414] "containerMap: removing container" podUID="aced22f0-8827-4cad-b04b-85ddb56490af" containerName="istio-proxy" Oct 30 13:15:01.779167 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:01.778891 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aced22f0-8827-4cad-b04b-85ddb56490af" containerName="istio-proxy" Oct 30 13:15:01.779167 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:01.778897 5922 cpu_manager.go:414] "containerMap: removing container" podUID="81a0354c-9716-4cc3-a902-b81908879ab3" containerName="payload-pre-processing" Oct 30 13:15:01.779167 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:01.778902 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81a0354c-9716-4cc3-a902-b81908879ab3" containerName="payload-pre-processing" Oct 30 13:15:01.779167 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:01.778960 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="f661cb7b-0cb2-42c2-a0a9-96dc71421c3e" containerName="revoke-keys" Oct 30 13:15:01.779167 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:01.778968 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="c61d2f25-754d-4538-b154-b9095bffe96f" containerName="istio-proxy" Oct 30 13:15:01.779167 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:01.778975 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="b607f9a7-6ef2-4ea2-bdb4-352b2312966d" containerName="istio-proxy" Oct 30 13:15:01.779167 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:01.778983 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="aced22f0-8827-4cad-b04b-85ddb56490af" containerName="istio-proxy" Oct 30 13:15:01.779167 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:01.778989 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="81a0354c-9716-4cc3-a902-b81908879ab3" containerName="payload-pre-processing" Oct 30 13:15:01.779167 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:01.778995 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="a68a48f7-d090-4eb7-af40-df304a84b319" containerName="istio-proxy" Oct 30 13:15:01.779167 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:01.779001 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="aced22f0-8827-4cad-b04b-85ddb56490af" containerName="istio-proxy" Oct 30 13:15:01.779167 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:01.779006 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="47517373-759d-4eeb-ab5e-6150b7e82bb7" containerName="main" Oct 30 13:15:01.779167 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:01.779015 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="aee561c5-3aa7-4021-9d3e-05e7636d19be" containerName="istio-proxy" Oct 30 13:15:01.782532 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:01.782515 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cplvq" Oct 30 13:15:01.801225 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:01.801200 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cplvq"] Oct 30 13:15:01.810377 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:01.810350 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:15:01.810377 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:01.810353 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:15:01.810533 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:01.810386 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:15:01.810533 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:01.810478 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:15:01.874664 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:01.874640 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b511576b-152c-41a8-ad55-e5341f42de26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-cplvq\" (UID: \"b511576b-152c-41a8-ad55-e5341f42de26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cplvq" Oct 30 13:15:01.874795 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:01.874671 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b511576b-152c-41a8-ad55-e5341f42de26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-cplvq\" (UID: \"b511576b-152c-41a8-ad55-e5341f42de26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cplvq" Oct 30 13:15:01.874795 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:01.874737 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf6w7\" (UniqueName: \"kubernetes.io/projected/b511576b-152c-41a8-ad55-e5341f42de26-kube-api-access-pf6w7\") pod \"maas-api-revoke-sub-c1e271a9-cplvq\" (UID: \"b511576b-152c-41a8-ad55-e5341f42de26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cplvq" Oct 30 13:15:01.976020 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:01.975995 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b511576b-152c-41a8-ad55-e5341f42de26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-cplvq\" (UID: \"b511576b-152c-41a8-ad55-e5341f42de26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cplvq" Oct 30 13:15:01.976151 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:01.976026 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b511576b-152c-41a8-ad55-e5341f42de26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-cplvq\" (UID: \"b511576b-152c-41a8-ad55-e5341f42de26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cplvq" Oct 30 13:15:01.976151 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:01.976047 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf6w7\" (UniqueName: \"kubernetes.io/projected/b511576b-152c-41a8-ad55-e5341f42de26-kube-api-access-pf6w7\") pod \"maas-api-revoke-sub-c1e271a9-cplvq\" (UID: \"b511576b-152c-41a8-ad55-e5341f42de26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cplvq" Oct 30 13:15:01.976799 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:01.976775 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b511576b-152c-41a8-ad55-e5341f42de26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-cplvq\" (UID: \"b511576b-152c-41a8-ad55-e5341f42de26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cplvq" Oct 30 13:15:01.978249 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:01.978231 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b511576b-152c-41a8-ad55-e5341f42de26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-cplvq\" (UID: \"b511576b-152c-41a8-ad55-e5341f42de26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cplvq" Oct 30 13:15:02.000844 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:02.000818 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf6w7\" (UniqueName: \"kubernetes.io/projected/b511576b-152c-41a8-ad55-e5341f42de26-kube-api-access-pf6w7\") pod \"maas-api-revoke-sub-c1e271a9-cplvq\" (UID: \"b511576b-152c-41a8-ad55-e5341f42de26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cplvq" Oct 30 13:15:02.094696 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:02.094645 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cplvq" Oct 30 13:15:02.408778 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:02.408753 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cplvq"] Oct 30 13:15:02.411479 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:15:02.411442 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb511576b_152c_41a8_ad55_e5341f42de26.slice/crio-128c2c176cabb3d66f96fbbcd75e34608d5a006892086dc17a4dbab1cdd1726d WatchSource:0}: Error finding container 128c2c176cabb3d66f96fbbcd75e34608d5a006892086dc17a4dbab1cdd1726d: Status 404 returned error can't find the container with id 128c2c176cabb3d66f96fbbcd75e34608d5a006892086dc17a4dbab1cdd1726d Oct 30 13:15:03.025271 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:03.025237 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cplvq" event={"ID":"b511576b-152c-41a8-ad55-e5341f42de26","Type":"ContainerStarted","Data":"3918853fc00b71be007238a190583174d846d1bd92a72d8e72fdec3024fbf8d3"} Oct 30 13:15:03.025702 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:03.025302 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cplvq" event={"ID":"b511576b-152c-41a8-ad55-e5341f42de26","Type":"ContainerStarted","Data":"128c2c176cabb3d66f96fbbcd75e34608d5a006892086dc17a4dbab1cdd1726d"} Oct 30 13:15:03.069463 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:03.069405 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cplvq" podStartSLOduration=2.069392556 podStartE2EDuration="2.069392556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:03.067467439 +0000 UTC m=+1359.536883185" watchObservedRunningTime="2026-10-30 13:15:03.069392556 +0000 UTC m=+1359.538808300" Oct 30 13:15:04.033591 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:04.033550 5922 generic.go:356] "Generic (PLEG): container finished" podID="b511576b-152c-41a8-ad55-e5341f42de26" containerID="3918853fc00b71be007238a190583174d846d1bd92a72d8e72fdec3024fbf8d3" exitCode=0 Oct 30 13:15:04.034011 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:04.033641 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cplvq" event={"ID":"b511576b-152c-41a8-ad55-e5341f42de26","Type":"ContainerDied","Data":"3918853fc00b71be007238a190583174d846d1bd92a72d8e72fdec3024fbf8d3"} Oct 30 13:15:05.173235 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:05.173211 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cplvq" Oct 30 13:15:05.205570 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:05.205544 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b511576b-152c-41a8-ad55-e5341f42de26-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b511576b-152c-41a8-ad55-e5341f42de26-maas-api-openshift-ca\") pod \"b511576b-152c-41a8-ad55-e5341f42de26\" (UID: \"b511576b-152c-41a8-ad55-e5341f42de26\") " Oct 30 13:15:05.205695 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:05.205582 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b511576b-152c-41a8-ad55-e5341f42de26-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b511576b-152c-41a8-ad55-e5341f42de26-maas-api-xks-ca\") pod \"b511576b-152c-41a8-ad55-e5341f42de26\" (UID: \"b511576b-152c-41a8-ad55-e5341f42de26\") " Oct 30 13:15:05.205695 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:05.205655 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b511576b-152c-41a8-ad55-e5341f42de26-kube-api-access-pf6w7\" (UniqueName: \"kubernetes.io/projected/b511576b-152c-41a8-ad55-e5341f42de26-kube-api-access-pf6w7\") pod \"b511576b-152c-41a8-ad55-e5341f42de26\" (UID: \"b511576b-152c-41a8-ad55-e5341f42de26\") " Oct 30 13:15:05.205929 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:05.205906 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b511576b-152c-41a8-ad55-e5341f42de26-maas-api-openshift-ca" pod "b511576b-152c-41a8-ad55-e5341f42de26" (UID: "b511576b-152c-41a8-ad55-e5341f42de26"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:15:05.207502 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:05.207473 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b511576b-152c-41a8-ad55-e5341f42de26-maas-api-xks-ca" pod "b511576b-152c-41a8-ad55-e5341f42de26" (UID: "b511576b-152c-41a8-ad55-e5341f42de26"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:15:05.207657 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:05.207629 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b511576b-152c-41a8-ad55-e5341f42de26-kube-api-access-pf6w7" pod "b511576b-152c-41a8-ad55-e5341f42de26" (UID: "b511576b-152c-41a8-ad55-e5341f42de26"). InnerVolumeSpecName "kube-api-access-pf6w7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:15:05.306778 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:05.306709 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf6w7\" (UniqueName: \"kubernetes.io/projected/b511576b-152c-41a8-ad55-e5341f42de26-kube-api-access-pf6w7\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:15:05.306778 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:05.306733 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b511576b-152c-41a8-ad55-e5341f42de26-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:15:05.306778 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:05.306743 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b511576b-152c-41a8-ad55-e5341f42de26-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:15:06.048415 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:06.048391 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cplvq" Oct 30 13:15:06.048572 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:06.048431 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cplvq" event={"ID":"b511576b-152c-41a8-ad55-e5341f42de26","Type":"ContainerDied","Data":"128c2c176cabb3d66f96fbbcd75e34608d5a006892086dc17a4dbab1cdd1726d"} Oct 30 13:15:06.048572 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:06.048479 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="128c2c176cabb3d66f96fbbcd75e34608d5a006892086dc17a4dbab1cdd1726d" Oct 30 13:15:12.931432 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:12.931400 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:15:18.146882 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:18.146852 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:15:21.089316 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:21.089278 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d33f9772-6rtmj"] Oct 30 13:15:21.108195 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:21.108159 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d33f9772-6rtmj"] Oct 30 13:15:21.643255 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:21.643223 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0faf6428-6b88-40c6-b3a2-bc963b9f3bc0" path="/var/lib/kubelet/pods/0faf6428-6b88-40c6-b3a2-bc963b9f3bc0/volumes" Oct 30 13:15:23.552376 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:15:23.552341 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:15:24.751702 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:24.751674 5922 scope.go:122] "RemoveContainer" containerID="25aea5083d07e431a1d25029628b8f0bef171d531588bfe128d98ce740c95d3d" Oct 30 13:15:24.760965 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:24.760946 5922 scope.go:122] "RemoveContainer" containerID="f16b093f50b8156f881db45962bb451bf6f51865439580e0c15efb06c760ccf6" Oct 30 13:15:24.769153 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:24.769131 5922 scope.go:122] "RemoveContainer" containerID="d297df8dcf181a776ef5f89ee39585a0dd2cb3c85129face9e4737fce5406c5c" Oct 30 13:15:24.777054 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:24.777033 5922 scope.go:122] "RemoveContainer" containerID="ac633b60600bdbe934660e775f143f26d146dd66870e397db9fbcef5861a1223" Oct 30 13:15:24.785185 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:24.785169 5922 scope.go:122] "RemoveContainer" containerID="60744197d6e360df5966c56420fec4d883da73fcca17c47fd9dcad6b23348335" Oct 30 13:15:24.793266 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:24.793250 5922 scope.go:122] "RemoveContainer" containerID="36c01587692285970bbb3344ea9205c08a95e6fee15a2d58c822cb5376f092a9" Oct 30 13:15:24.800927 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:24.800913 5922 scope.go:122] "RemoveContainer" containerID="5e300b0795401d3e348e75077417cac184169b31b32e4605e2dc933525ae03b1" Oct 30 13:15:24.808404 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:24.808388 5922 scope.go:122] "RemoveContainer" containerID="5b8f3b32480a71efef16c562ccabc8572deca9250a9d340aeb959b20b3412a87" Oct 30 13:15:24.816017 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:24.816000 5922 scope.go:122] "RemoveContainer" containerID="65af167f59e75105a08570ed9d8cb2925c74a29ec11a666fe77f819610f8b6c9" Oct 30 13:15:24.826268 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:24.826251 5922 scope.go:122] "RemoveContainer" containerID="82e57b9fe02b47147dfbf945e45868f8faeeaf2150b54b69a09b616df7b24112" Oct 30 13:15:24.834365 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:24.834345 5922 scope.go:122] "RemoveContainer" containerID="6086d058ca6acbc00bb7475ab07d68a2eccfda3d4256ce486c01678cf03a9195" Oct 30 13:15:24.842234 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:24.842215 5922 scope.go:122] "RemoveContainer" containerID="3637756e4f5e53072f89a762b3d88e136366aeb954218b78199f7a271b5a7d25" Oct 30 13:15:24.849785 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:24.849759 5922 scope.go:122] "RemoveContainer" containerID="c721f00e5b7d6b2121217cc7a7a193da8ff6c4a0f36e83cca00b2d1186b44297" Oct 30 13:15:31.035913 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:31.035877 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-999b8bdc-fmt85"] Oct 30 13:15:31.048285 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:31.048254 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-999b8bdc-fmt85"] Oct 30 13:15:31.643582 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:31.643545 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f904654e-bf69-45ec-9293-a555868acb57" path="/var/lib/kubelet/pods/f904654e-bf69-45ec-9293-a555868acb57/volumes" Oct 30 13:15:35.289919 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:35.289880 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:36.235608 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:36.235574 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:15:44.037144 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:44.037111 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea4b8cb1-tl947"] Oct 30 13:15:44.050365 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:44.050336 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea4b8cb1-tl947"] Oct 30 13:15:45.642932 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:45.642901 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f3c3956-902f-46f0-be07-e4d57765ff4a" path="/var/lib/kubelet/pods/1f3c3956-902f-46f0-be07-e4d57765ff4a/volumes" Oct 30 13:15:51.033009 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:51.032973 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a976c65-zdf47"] Oct 30 13:15:51.045404 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:51.045378 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a976c65-zdf47"] Oct 30 13:15:51.643957 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:51.643923 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6b7cc9a-3532-4b96-b523-fb5eeb9f6258" path="/var/lib/kubelet/pods/d6b7cc9a-3532-4b96-b523-fb5eeb9f6258/volumes" Oct 30 13:15:51.806139 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:51.806108 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:15:53.554761 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:15:53.554725 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:15:54.036367 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:54.035919 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-800c8ce2-ls4mh"] Oct 30 13:15:54.046258 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:54.046226 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-800c8ce2-ls4mh"] Oct 30 13:15:55.643310 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:15:55.643274 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4f503a4-2366-439a-9603-e2e49b8c823a" path="/var/lib/kubelet/pods/d4f503a4-2366-439a-9603-e2e49b8c823a/volumes" Oct 30 13:16:10.040831 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:16:10.040798 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d1aa41c-tt4zt"] Oct 30 13:16:10.053203 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:16:10.053174 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe3da6ff-bq57s"] Oct 30 13:16:10.061317 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:16:10.061277 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d1aa41c-tt4zt"] Oct 30 13:16:10.061473 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:16:10.061355 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe3da6ff-bq57s"] Oct 30 13:16:11.642858 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:16:11.642825 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="123b434c-7da6-49b6-9f2f-801c92f68c59" path="/var/lib/kubelet/pods/123b434c-7da6-49b6-9f2f-801c92f68c59/volumes" Oct 30 13:16:11.643357 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:16:11.643336 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ce4cf03-e6a2-442f-8065-9a3f541e84c1" path="/var/lib/kubelet/pods/4ce4cf03-e6a2-442f-8065-9a3f541e84c1/volumes" Oct 30 13:16:16.019298 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:16:16.019264 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:16:18.619575 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:16:18.619540 5922 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-cxwwl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:16:18.620182 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:16:18.620152 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cxwwl" podUID="c29858c0-d396-4092-9e19-b76069383a82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:16:23.552082 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:16:23.552045 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:16:24.919889 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:16:24.919863 5922 scope.go:122] "RemoveContainer" containerID="7ce43e29216ea6847c30d16aed2700d7d6d584c3b8eaa0d6cd18949dfdc90313" Oct 30 13:16:24.928412 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:16:24.928391 5922 scope.go:122] "RemoveContainer" containerID="68d5dd32bdc2d1b092bf92165ddb66d1b4be0f7dd9e0af71a99a35f730df0757" Oct 30 13:16:24.936135 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:16:24.936119 5922 scope.go:122] "RemoveContainer" containerID="35f1cda2c709c6dd298c22aba5749904bbdd03a3156a5fba08bd70d4759fce11" Oct 30 13:16:24.943810 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:16:24.943791 5922 scope.go:122] "RemoveContainer" containerID="cb9f276c1c16ddcf5712aea5471d1a07a01c35542d0ab9a5adfb48b2308ca603" Oct 30 13:16:26.407959 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:16:26.407926 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:16:29.034045 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:16:29.034012 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f45f73ac-pvcfw"] Oct 30 13:16:29.046897 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:16:29.046868 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f5ef83c-26bfl"] Oct 30 13:16:29.048522 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:16:29.048500 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f45f73ac-pvcfw"] Oct 30 13:16:29.057016 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:16:29.056993 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f5ef83c-26bfl"] Oct 30 13:16:29.642986 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:16:29.642952 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="175cab09-e4aa-424b-9eaa-01c9d32d2669" path="/var/lib/kubelet/pods/175cab09-e4aa-424b-9eaa-01c9d32d2669/volumes" Oct 30 13:16:29.643364 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:16:29.643351 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b8f7b30-787c-451b-a41f-a5973359c9d6" path="/var/lib/kubelet/pods/8b8f7b30-787c-451b-a41f-a5973359c9d6/volumes" Oct 30 13:16:30.730914 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:16:30.730882 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:16:33.973898 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:16:33.973857 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:48.944772 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:16:48.944733 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:16:53.554671 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:16:53.554636 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:16:54.348046 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:16:54.348015 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:17:02.046860 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:17:02.046829 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88b5cc45-jzt8s"] Oct 30 13:17:02.059180 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:17:02.059150 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5da9845-7bwfw"] Oct 30 13:17:02.061003 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:17:02.060982 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88b5cc45-jzt8s"] Oct 30 13:17:02.062313 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:17:02.062288 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5da9845-7bwfw"] Oct 30 13:17:03.643903 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:17:03.643875 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07" path="/var/lib/kubelet/pods/76a1b5bb-f7a5-4f5a-a663-c76c3cfe0c07/volumes" Oct 30 13:17:03.644286 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:17:03.644267 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d02e9835-5ada-40e8-ba81-13f9f8fe6541" path="/var/lib/kubelet/pods/d02e9835-5ada-40e8-ba81-13f9f8fe6541/volumes" Oct 30 13:17:09.346165 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:17:09.346132 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:17:11.034853 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:17:11.034813 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bdd6e80-w87p7"] Oct 30 13:17:11.048449 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:17:11.048416 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bdd6e80-w87p7"] Oct 30 13:17:11.644106 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:17:11.644066 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dba2c02d-ec04-41d1-a34a-1929af79ad19" path="/var/lib/kubelet/pods/dba2c02d-ec04-41d1-a34a-1929af79ad19/volumes" Oct 30 13:17:13.743458 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:17:13.743421 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:17:20.397959 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:17:20.397920 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:23.551139 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:17:23.551098 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:17:24.978052 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:17:24.978022 5922 scope.go:122] "RemoveContainer" containerID="255bfd3614b2c603399dcccd55d2e96858fa2b2c04fb1b1e9ba746c8db7a3574" Oct 30 13:17:24.986687 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:17:24.986667 5922 scope.go:122] "RemoveContainer" containerID="88d0bb0714b430a483c165a29e6a401d62a2fbd0e515a34aa7e89c5bc94cfc92" Oct 30 13:17:24.994547 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:17:24.994474 5922 scope.go:122] "RemoveContainer" containerID="55b97076ea9178de76095090b9db4024adf581c09b608a8c8a704fa7d5e67af9" Oct 30 13:17:25.002154 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:17:25.002135 5922 scope.go:122] "RemoveContainer" containerID="e8bfc3c2833e78c45b43ea9dd8b219dcab439fc8196e0e3c966c7c5166b42f77" Oct 30 13:17:25.009973 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:17:25.009958 5922 scope.go:122] "RemoveContainer" containerID="74cdbdcd347bc2725a25d46db6364ec9246906ae40e4dc19fd5144ad6109c65a" Oct 30 13:17:25.017327 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:17:25.017311 5922 scope.go:122] "RemoveContainer" containerID="fa7691e1fd67c1f691497b2362b3c83a82437e03ff09c0f9901b99c1e5d74fb6" Oct 30 13:17:25.024581 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:17:25.024565 5922 scope.go:122] "RemoveContainer" containerID="b0e194606ccb4bf0affeef4db95b72c405d4a7c4e285eefdf7001f935b2767e1" Oct 30 13:17:25.031967 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:17:25.031949 5922 scope.go:122] "RemoveContainer" containerID="53f2de8b3865d9ad99c0449d12a59589f74a2eab109176772be9948dea645e1e" Oct 30 13:17:25.039204 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:17:25.039176 5922 scope.go:122] "RemoveContainer" containerID="4e808723219d80c13f4b09952f342e8b7ec69301b4132fd762032e770428bef1" Oct 30 13:17:25.046396 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:17:25.046378 5922 scope.go:122] "RemoveContainer" containerID="eda2bdd0c668e8aeed3acf0f014ad1a65c5d4172ae9f6e745c2334db368cac84" Oct 30 13:17:30.333821 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:17:30.333781 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:17:34.122691 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:17:34.122648 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:34.254763 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:17:34.254732 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:35.978213 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:17:35.978179 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:36.065242 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:17:36.065213 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b817f58-b2kvn"] Oct 30 13:17:36.077949 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:17:36.077923 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9876320b-9tg8r"] Oct 30 13:17:36.079454 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:17:36.079430 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9876320b-9tg8r"] Oct 30 13:17:36.084574 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:17:36.084556 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b817f58-b2kvn"] Oct 30 13:17:36.114890 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:17:36.114865 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:37.643641 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:17:37.643594 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b0d2d00-e66c-410a-8383-04a6c34620fd" path="/var/lib/kubelet/pods/8b0d2d00-e66c-410a-8383-04a6c34620fd/volumes" Oct 30 13:17:37.644013 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:17:37.643998 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2833e9b-a45f-4557-b5c9-075474e38f3f" path="/var/lib/kubelet/pods/a2833e9b-a45f-4557-b5c9-075474e38f3f/volumes" Oct 30 13:17:42.034390 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:17:42.034360 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3391a008-kbn48"] Oct 30 13:17:42.046851 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:17:42.046816 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac6ffc9f-jr8sq"] Oct 30 13:17:42.048162 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:17:42.048143 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3391a008-kbn48"] Oct 30 13:17:42.049342 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:17:42.049314 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac6ffc9f-jr8sq"] Oct 30 13:17:43.643829 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:17:43.643796 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63616383-42a4-4a01-96c0-a7aafaf76c28" path="/var/lib/kubelet/pods/63616383-42a4-4a01-96c0-a7aafaf76c28/volumes" Oct 30 13:17:43.644229 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:17:43.644189 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6eac003e-b8a0-4b88-a296-59e5a9c85a05" path="/var/lib/kubelet/pods/6eac003e-b8a0-4b88-a296-59e5a9c85a05/volumes" Oct 30 13:17:46.203166 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:17:46.203134 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:17:53.552199 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:17:53.552164 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:17:56.051123 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:17:56.051091 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6787f73-9vvt2"] Oct 30 13:17:56.063252 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:17:56.063220 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6787f73-9vvt2"] Oct 30 13:17:57.105600 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:17:57.105563 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:17:57.645683 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:17:57.645635 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b126f0c-889e-4485-a4af-4ee23b88ab57" path="/var/lib/kubelet/pods/8b126f0c-889e-4485-a4af-4ee23b88ab57/volumes" Oct 30 13:18:02.537579 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:02.537539 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:18:16.823130 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:16.823052 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rl5vp"] Oct 30 13:18:16.823572 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:16.823274 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rl5vp" podUID="ef89e987-6307-4b0b-9cdc-8059bf210075" containerName="discovery" containerID="cri-o://aeb303dee201887666d547e9aa369ce09d96c7e8d96512aa49d33765c36bcb8b" gracePeriod=30 Oct 30 13:18:17.072603 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:17.072576 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rl5vp" Oct 30 13:18:17.231910 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:17.231883 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef89e987-6307-4b0b-9cdc-8059bf210075-cacerts\" (UniqueName: \"kubernetes.io/secret/ef89e987-6307-4b0b-9cdc-8059bf210075-cacerts\") pod \"ef89e987-6307-4b0b-9cdc-8059bf210075\" (UID: \"ef89e987-6307-4b0b-9cdc-8059bf210075\") " Oct 30 13:18:17.232066 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:17.231920 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef89e987-6307-4b0b-9cdc-8059bf210075-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ef89e987-6307-4b0b-9cdc-8059bf210075-istio-csr-ca-configmap\") pod \"ef89e987-6307-4b0b-9cdc-8059bf210075\" (UID: \"ef89e987-6307-4b0b-9cdc-8059bf210075\") " Oct 30 13:18:17.232066 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:17.231969 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef89e987-6307-4b0b-9cdc-8059bf210075-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef89e987-6307-4b0b-9cdc-8059bf210075-local-certs\") pod \"ef89e987-6307-4b0b-9cdc-8059bf210075\" (UID: \"ef89e987-6307-4b0b-9cdc-8059bf210075\") " Oct 30 13:18:17.232066 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:17.232003 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef89e987-6307-4b0b-9cdc-8059bf210075-istio-token\" (UniqueName: \"kubernetes.io/projected/ef89e987-6307-4b0b-9cdc-8059bf210075-istio-token\") pod \"ef89e987-6307-4b0b-9cdc-8059bf210075\" (UID: \"ef89e987-6307-4b0b-9cdc-8059bf210075\") " Oct 30 13:18:17.232066 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:17.232054 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef89e987-6307-4b0b-9cdc-8059bf210075-kube-api-access-6ghf7\" (UniqueName: \"kubernetes.io/projected/ef89e987-6307-4b0b-9cdc-8059bf210075-kube-api-access-6ghf7\") pod \"ef89e987-6307-4b0b-9cdc-8059bf210075\" (UID: \"ef89e987-6307-4b0b-9cdc-8059bf210075\") " Oct 30 13:18:17.232281 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:17.232110 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef89e987-6307-4b0b-9cdc-8059bf210075-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ef89e987-6307-4b0b-9cdc-8059bf210075-istio-csr-dns-cert\") pod \"ef89e987-6307-4b0b-9cdc-8059bf210075\" (UID: \"ef89e987-6307-4b0b-9cdc-8059bf210075\") " Oct 30 13:18:17.232281 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:17.232144 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef89e987-6307-4b0b-9cdc-8059bf210075-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ef89e987-6307-4b0b-9cdc-8059bf210075-istio-kubeconfig\") pod \"ef89e987-6307-4b0b-9cdc-8059bf210075\" (UID: \"ef89e987-6307-4b0b-9cdc-8059bf210075\") " Oct 30 13:18:17.232721 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:17.232398 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef89e987-6307-4b0b-9cdc-8059bf210075-istio-csr-ca-configmap" pod "ef89e987-6307-4b0b-9cdc-8059bf210075" (UID: "ef89e987-6307-4b0b-9cdc-8059bf210075"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:18:17.232721 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:17.232582 5922 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ef89e987-6307-4b0b-9cdc-8059bf210075-istio-csr-ca-configmap\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:18:17.234341 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:17.234317 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef89e987-6307-4b0b-9cdc-8059bf210075-cacerts" pod "ef89e987-6307-4b0b-9cdc-8059bf210075" (UID: "ef89e987-6307-4b0b-9cdc-8059bf210075"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:18:17.234481 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:17.234425 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef89e987-6307-4b0b-9cdc-8059bf210075-istio-csr-dns-cert" pod "ef89e987-6307-4b0b-9cdc-8059bf210075" (UID: "ef89e987-6307-4b0b-9cdc-8059bf210075"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:18:17.234692 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:17.234670 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef89e987-6307-4b0b-9cdc-8059bf210075-local-certs" pod "ef89e987-6307-4b0b-9cdc-8059bf210075" (UID: "ef89e987-6307-4b0b-9cdc-8059bf210075"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:17.234795 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:17.234715 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef89e987-6307-4b0b-9cdc-8059bf210075-kube-api-access-6ghf7" pod "ef89e987-6307-4b0b-9cdc-8059bf210075" (UID: "ef89e987-6307-4b0b-9cdc-8059bf210075"). InnerVolumeSpecName "kube-api-access-6ghf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:18:17.234795 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:17.234725 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef89e987-6307-4b0b-9cdc-8059bf210075-istio-token" pod "ef89e987-6307-4b0b-9cdc-8059bf210075" (UID: "ef89e987-6307-4b0b-9cdc-8059bf210075"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:18:17.234908 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:17.234856 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef89e987-6307-4b0b-9cdc-8059bf210075-istio-kubeconfig" pod "ef89e987-6307-4b0b-9cdc-8059bf210075" (UID: "ef89e987-6307-4b0b-9cdc-8059bf210075"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:18:17.252301 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:17.252278 5922 generic.go:356] "Generic (PLEG): container finished" podID="ef89e987-6307-4b0b-9cdc-8059bf210075" containerID="aeb303dee201887666d547e9aa369ce09d96c7e8d96512aa49d33765c36bcb8b" exitCode=0 Oct 30 13:18:17.252400 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:17.252320 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rl5vp" Oct 30 13:18:17.252400 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:17.252368 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rl5vp" event={"ID":"ef89e987-6307-4b0b-9cdc-8059bf210075","Type":"ContainerDied","Data":"aeb303dee201887666d547e9aa369ce09d96c7e8d96512aa49d33765c36bcb8b"} Oct 30 13:18:17.252477 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:17.252414 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rl5vp" event={"ID":"ef89e987-6307-4b0b-9cdc-8059bf210075","Type":"ContainerDied","Data":"006acdfc2532f669cf3a106b6abaedb7749f95c36a789cf4c036f98692ad824c"} Oct 30 13:18:17.252477 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:17.252438 5922 scope.go:122] "RemoveContainer" containerID="aeb303dee201887666d547e9aa369ce09d96c7e8d96512aa49d33765c36bcb8b" Oct 30 13:18:17.261677 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:17.261657 5922 scope.go:122] "RemoveContainer" containerID="aeb303dee201887666d547e9aa369ce09d96c7e8d96512aa49d33765c36bcb8b" Oct 30 13:18:17.262027 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:18:17.262005 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeb303dee201887666d547e9aa369ce09d96c7e8d96512aa49d33765c36bcb8b\": container with ID starting with aeb303dee201887666d547e9aa369ce09d96c7e8d96512aa49d33765c36bcb8b not found: ID does not exist" containerID="aeb303dee201887666d547e9aa369ce09d96c7e8d96512aa49d33765c36bcb8b" Oct 30 13:18:17.262103 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:17.262034 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeb303dee201887666d547e9aa369ce09d96c7e8d96512aa49d33765c36bcb8b"} err="failed to get container status \"aeb303dee201887666d547e9aa369ce09d96c7e8d96512aa49d33765c36bcb8b\": rpc error: code = NotFound desc = could not find container \"aeb303dee201887666d547e9aa369ce09d96c7e8d96512aa49d33765c36bcb8b\": container with ID starting with aeb303dee201887666d547e9aa369ce09d96c7e8d96512aa49d33765c36bcb8b not found: ID does not exist" Oct 30 13:18:17.306384 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:17.306363 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rl5vp"] Oct 30 13:18:17.312040 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:17.312019 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rl5vp"] Oct 30 13:18:17.333662 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:17.333643 5922 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef89e987-6307-4b0b-9cdc-8059bf210075-local-certs\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:18:17.333662 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:17.333662 5922 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef89e987-6307-4b0b-9cdc-8059bf210075-istio-token\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:18:17.333785 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:17.333672 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ghf7\" (UniqueName: \"kubernetes.io/projected/ef89e987-6307-4b0b-9cdc-8059bf210075-kube-api-access-6ghf7\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:18:17.333785 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:17.333681 5922 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ef89e987-6307-4b0b-9cdc-8059bf210075-istio-csr-dns-cert\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:18:17.333785 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:17.333690 5922 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ef89e987-6307-4b0b-9cdc-8059bf210075-istio-kubeconfig\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:18:17.333785 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:17.333698 5922 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ef89e987-6307-4b0b-9cdc-8059bf210075-cacerts\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:18:17.643811 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:17.643716 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef89e987-6307-4b0b-9cdc-8059bf210075" path="/var/lib/kubelet/pods/ef89e987-6307-4b0b-9cdc-8059bf210075/volumes" Oct 30 13:18:19.123805 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:19.123772 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:18:23.551538 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:18:23.551497 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:18:24.256133 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:24.256100 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:18:25.033559 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:25.033519 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26235bd2-9qhbp"] Oct 30 13:18:25.045959 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:25.045930 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26235bd2-9qhbp"] Oct 30 13:18:25.105216 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:25.105187 5922 scope.go:122] "RemoveContainer" containerID="d1d2f917c7a798cbdc510cae1348a6abff58464f94b6a230aa571610620596d2" Oct 30 13:18:25.120103 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:25.120080 5922 scope.go:122] "RemoveContainer" containerID="5c97d255df6aee6196e2feb25d70e4bedc1d130440fb12c1963533329d3e42ff" Oct 30 13:18:25.127788 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:25.127766 5922 scope.go:122] "RemoveContainer" containerID="dcfd34e36b17b2632487f20346286b00179b67724ea3aa31110167dbae92b2a6" Oct 30 13:18:25.135924 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:25.135895 5922 scope.go:122] "RemoveContainer" containerID="3be15fdc0e9aa99da628b6b7c888f0d6cf92969398e5e9204a3066e9f04b0842" Oct 30 13:18:25.143097 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:25.143075 5922 scope.go:122] "RemoveContainer" containerID="fb539cc171ca84c72ad57fd9b50f01fb7b6e3ec2710a9dcb7ccff06728420efe" Oct 30 13:18:25.150308 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:25.150287 5922 scope.go:122] "RemoveContainer" containerID="8db64bf786211700edec801e519a453da01f0e61bb1751ccf51bea7fcf411c36" Oct 30 13:18:25.158076 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:25.158061 5922 scope.go:122] "RemoveContainer" containerID="73b134027f26a17722b5ee94dcaeb7d4c8becfd5d136644bdd15c373f28d035f" Oct 30 13:18:25.643197 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:25.643166 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f661cb7b-0cb2-42c2-a0a9-96dc71421c3e" path="/var/lib/kubelet/pods/f661cb7b-0cb2-42c2-a0a9-96dc71421c3e/volumes" Oct 30 13:18:30.819196 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:30.819165 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hst9t"] Oct 30 13:18:30.819568 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:30.819527 5922 cpu_manager.go:414] "containerMap: removing container" podUID="a68a48f7-d090-4eb7-af40-df304a84b319" containerName="istio-proxy" Oct 30 13:18:30.819568 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:30.819538 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a68a48f7-d090-4eb7-af40-df304a84b319" containerName="istio-proxy" Oct 30 13:18:30.819664 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:30.819577 5922 cpu_manager.go:414] "containerMap: removing container" podUID="b511576b-152c-41a8-ad55-e5341f42de26" containerName="revoke-keys" Oct 30 13:18:30.819664 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:30.819583 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b511576b-152c-41a8-ad55-e5341f42de26" containerName="revoke-keys" Oct 30 13:18:30.819664 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:30.819592 5922 cpu_manager.go:414] "containerMap: removing container" podUID="ef89e987-6307-4b0b-9cdc-8059bf210075" containerName="discovery" Oct 30 13:18:30.819664 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:30.819597 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef89e987-6307-4b0b-9cdc-8059bf210075" containerName="discovery" Oct 30 13:18:30.819664 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:30.819605 5922 cpu_manager.go:414] "containerMap: removing container" podUID="aced22f0-8827-4cad-b04b-85ddb56490af" containerName="istio-proxy" Oct 30 13:18:30.819664 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:30.819622 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aced22f0-8827-4cad-b04b-85ddb56490af" containerName="istio-proxy" Oct 30 13:18:30.819851 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:30.819705 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="b511576b-152c-41a8-ad55-e5341f42de26" containerName="revoke-keys" Oct 30 13:18:30.819851 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:30.819713 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef89e987-6307-4b0b-9cdc-8059bf210075" containerName="discovery" Oct 30 13:18:30.819851 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:30.819719 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="a68a48f7-d090-4eb7-af40-df304a84b319" containerName="istio-proxy" Oct 30 13:18:30.825033 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:30.825016 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hst9t" Oct 30 13:18:30.842445 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:30.842419 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hst9t"] Oct 30 13:18:30.852381 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:30.852350 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:18:30.852638 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:30.852600 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:18:30.852638 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:30.852603 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:18:30.853050 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:30.853033 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:18:30.951124 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:30.951099 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc7b3877-b42c-4918-8957-24fd4e00ef57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-hst9t\" (UID: \"cc7b3877-b42c-4918-8957-24fd4e00ef57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hst9t" Oct 30 13:18:30.951228 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:30.951132 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsvdn\" (UniqueName: \"kubernetes.io/projected/cc7b3877-b42c-4918-8957-24fd4e00ef57-kube-api-access-dsvdn\") pod \"maas-api-revoke-sub-3d4156f1-hst9t\" (UID: \"cc7b3877-b42c-4918-8957-24fd4e00ef57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hst9t" Oct 30 13:18:30.951275 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:30.951250 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc7b3877-b42c-4918-8957-24fd4e00ef57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-hst9t\" (UID: \"cc7b3877-b42c-4918-8957-24fd4e00ef57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hst9t" Oct 30 13:18:31.052319 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:31.052284 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc7b3877-b42c-4918-8957-24fd4e00ef57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-hst9t\" (UID: \"cc7b3877-b42c-4918-8957-24fd4e00ef57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hst9t" Oct 30 13:18:31.052463 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:31.052325 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsvdn\" (UniqueName: \"kubernetes.io/projected/cc7b3877-b42c-4918-8957-24fd4e00ef57-kube-api-access-dsvdn\") pod \"maas-api-revoke-sub-3d4156f1-hst9t\" (UID: \"cc7b3877-b42c-4918-8957-24fd4e00ef57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hst9t" Oct 30 13:18:31.052463 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:31.052367 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc7b3877-b42c-4918-8957-24fd4e00ef57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-hst9t\" (UID: \"cc7b3877-b42c-4918-8957-24fd4e00ef57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hst9t" Oct 30 13:18:31.053042 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:31.053014 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc7b3877-b42c-4918-8957-24fd4e00ef57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-hst9t\" (UID: \"cc7b3877-b42c-4918-8957-24fd4e00ef57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hst9t" Oct 30 13:18:31.054721 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:31.054690 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc7b3877-b42c-4918-8957-24fd4e00ef57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-hst9t\" (UID: \"cc7b3877-b42c-4918-8957-24fd4e00ef57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hst9t" Oct 30 13:18:31.077078 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:31.077021 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsvdn\" (UniqueName: \"kubernetes.io/projected/cc7b3877-b42c-4918-8957-24fd4e00ef57-kube-api-access-dsvdn\") pod \"maas-api-revoke-sub-3d4156f1-hst9t\" (UID: \"cc7b3877-b42c-4918-8957-24fd4e00ef57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hst9t" Oct 30 13:18:31.138024 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:31.138001 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hst9t" Oct 30 13:18:31.434548 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:31.434524 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hst9t"] Oct 30 13:18:31.437429 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:18:31.437401 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc7b3877_b42c_4918_8957_24fd4e00ef57.slice/crio-2e721757f8a3e2e80e68ac378cfe5833c737d6e29f9278b1f3a4381ef197ff92 WatchSource:0}: Error finding container 2e721757f8a3e2e80e68ac378cfe5833c737d6e29f9278b1f3a4381ef197ff92: Status 404 returned error can't find the container with id 2e721757f8a3e2e80e68ac378cfe5833c737d6e29f9278b1f3a4381ef197ff92 Oct 30 13:18:32.336910 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:32.336879 5922 generic.go:356] "Generic (PLEG): container finished" podID="cc7b3877-b42c-4918-8957-24fd4e00ef57" containerID="56cbc23a49ff5ac334ac474abed887d72c78db539f62009c2161b562bfbba7a6" exitCode=0 Oct 30 13:18:32.337248 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:32.336968 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hst9t" event={"ID":"cc7b3877-b42c-4918-8957-24fd4e00ef57","Type":"ContainerDied","Data":"56cbc23a49ff5ac334ac474abed887d72c78db539f62009c2161b562bfbba7a6"} Oct 30 13:18:32.337248 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:32.337022 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hst9t" event={"ID":"cc7b3877-b42c-4918-8957-24fd4e00ef57","Type":"ContainerStarted","Data":"2e721757f8a3e2e80e68ac378cfe5833c737d6e29f9278b1f3a4381ef197ff92"} Oct 30 13:18:33.476574 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:33.476547 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hst9t" Oct 30 13:18:33.570698 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:33.570669 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc7b3877-b42c-4918-8957-24fd4e00ef57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc7b3877-b42c-4918-8957-24fd4e00ef57-maas-api-openshift-ca\") pod \"cc7b3877-b42c-4918-8957-24fd4e00ef57\" (UID: \"cc7b3877-b42c-4918-8957-24fd4e00ef57\") " Oct 30 13:18:33.570825 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:33.570719 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc7b3877-b42c-4918-8957-24fd4e00ef57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc7b3877-b42c-4918-8957-24fd4e00ef57-maas-api-xks-ca\") pod \"cc7b3877-b42c-4918-8957-24fd4e00ef57\" (UID: \"cc7b3877-b42c-4918-8957-24fd4e00ef57\") " Oct 30 13:18:33.570825 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:33.570758 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc7b3877-b42c-4918-8957-24fd4e00ef57-kube-api-access-dsvdn\" (UniqueName: \"kubernetes.io/projected/cc7b3877-b42c-4918-8957-24fd4e00ef57-kube-api-access-dsvdn\") pod \"cc7b3877-b42c-4918-8957-24fd4e00ef57\" (UID: \"cc7b3877-b42c-4918-8957-24fd4e00ef57\") " Oct 30 13:18:33.570998 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:33.570976 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc7b3877-b42c-4918-8957-24fd4e00ef57-maas-api-openshift-ca" pod "cc7b3877-b42c-4918-8957-24fd4e00ef57" (UID: "cc7b3877-b42c-4918-8957-24fd4e00ef57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:18:33.572842 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:33.572825 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc7b3877-b42c-4918-8957-24fd4e00ef57-maas-api-xks-ca" pod "cc7b3877-b42c-4918-8957-24fd4e00ef57" (UID: "cc7b3877-b42c-4918-8957-24fd4e00ef57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:18:33.572976 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:33.572957 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc7b3877-b42c-4918-8957-24fd4e00ef57-kube-api-access-dsvdn" pod "cc7b3877-b42c-4918-8957-24fd4e00ef57" (UID: "cc7b3877-b42c-4918-8957-24fd4e00ef57"). InnerVolumeSpecName "kube-api-access-dsvdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:18:33.671632 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:33.671587 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc7b3877-b42c-4918-8957-24fd4e00ef57-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:18:33.671733 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:33.671633 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsvdn\" (UniqueName: \"kubernetes.io/projected/cc7b3877-b42c-4918-8957-24fd4e00ef57-kube-api-access-dsvdn\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:18:33.671733 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:33.671648 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc7b3877-b42c-4918-8957-24fd4e00ef57-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:18:34.235652 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:34.235601 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:34.347718 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:34.347691 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hst9t" Oct 30 13:18:34.347718 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:34.347698 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hst9t" event={"ID":"cc7b3877-b42c-4918-8957-24fd4e00ef57","Type":"ContainerDied","Data":"2e721757f8a3e2e80e68ac378cfe5833c737d6e29f9278b1f3a4381ef197ff92"} Oct 30 13:18:34.347898 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:34.347748 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e721757f8a3e2e80e68ac378cfe5833c737d6e29f9278b1f3a4381ef197ff92" Oct 30 13:18:42.712396 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:42.712362 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:18:47.531429 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:47.531397 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:18:52.278559 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:52.278520 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:52.748104 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:52.748073 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd"] Oct 30 13:18:52.748520 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:52.748503 5922 cpu_manager.go:414] "containerMap: removing container" podUID="cc7b3877-b42c-4918-8957-24fd4e00ef57" containerName="revoke-keys" Oct 30 13:18:52.748520 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:52.748520 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc7b3877-b42c-4918-8957-24fd4e00ef57" containerName="revoke-keys" Oct 30 13:18:52.748770 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:52.748703 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc7b3877-b42c-4918-8957-24fd4e00ef57" containerName="revoke-keys" Oct 30 13:18:52.752052 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:52.752032 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" Oct 30 13:18:52.768054 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:52.768035 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-cadb6b-openshift-default-dockercfg-w9snt\"" Oct 30 13:18:52.781937 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:52.781893 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd"] Oct 30 13:18:52.810405 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:52.810373 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:52.825433 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:52.825410 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9xq6\" (UniqueName: \"kubernetes.io/projected/3489fce9-ff92-4d68-b15a-f82be6d6ca78-kube-api-access-p9xq6\") pod \"e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd\" (UID: \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\") " pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" Oct 30 13:18:52.825550 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:52.825440 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3489fce9-ff92-4d68-b15a-f82be6d6ca78-credential-socket\") pod \"e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd\" (UID: \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\") " pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" Oct 30 13:18:52.825550 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:52.825462 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3489fce9-ff92-4d68-b15a-f82be6d6ca78-workload-certs\") pod \"e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd\" (UID: \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\") " pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" Oct 30 13:18:52.825644 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:52.825580 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3489fce9-ff92-4d68-b15a-f82be6d6ca78-workload-socket\") pod \"e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd\" (UID: \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\") " pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" Oct 30 13:18:52.825714 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:52.825699 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3489fce9-ff92-4d68-b15a-f82be6d6ca78-istio-podinfo\") pod \"e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd\" (UID: \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\") " pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" Oct 30 13:18:52.825751 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:52.825732 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3489fce9-ff92-4d68-b15a-f82be6d6ca78-istiod-ca-cert\") pod \"e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd\" (UID: \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\") " pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" Oct 30 13:18:52.825789 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:52.825763 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3489fce9-ff92-4d68-b15a-f82be6d6ca78-istio-token\") pod \"e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd\" (UID: \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\") " pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" Oct 30 13:18:52.825882 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:52.825865 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3489fce9-ff92-4d68-b15a-f82be6d6ca78-istio-envoy\") pod \"e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd\" (UID: \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\") " pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" Oct 30 13:18:52.825921 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:52.825899 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3489fce9-ff92-4d68-b15a-f82be6d6ca78-istio-data\") pod \"e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd\" (UID: \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\") " pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" Oct 30 13:18:52.927162 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:52.927135 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9xq6\" (UniqueName: \"kubernetes.io/projected/3489fce9-ff92-4d68-b15a-f82be6d6ca78-kube-api-access-p9xq6\") pod \"e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd\" (UID: \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\") " pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" Oct 30 13:18:52.927304 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:52.927171 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3489fce9-ff92-4d68-b15a-f82be6d6ca78-credential-socket\") pod \"e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd\" (UID: \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\") " pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" Oct 30 13:18:52.927304 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:52.927204 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3489fce9-ff92-4d68-b15a-f82be6d6ca78-workload-certs\") pod \"e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd\" (UID: \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\") " pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" Oct 30 13:18:52.927304 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:52.927244 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3489fce9-ff92-4d68-b15a-f82be6d6ca78-workload-socket\") pod \"e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd\" (UID: \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\") " pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" Oct 30 13:18:52.927500 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:52.927306 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3489fce9-ff92-4d68-b15a-f82be6d6ca78-istio-podinfo\") pod \"e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd\" (UID: \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\") " pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" Oct 30 13:18:52.927500 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:52.927347 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3489fce9-ff92-4d68-b15a-f82be6d6ca78-istiod-ca-cert\") pod \"e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd\" (UID: \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\") " pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" Oct 30 13:18:52.927500 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:52.927393 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3489fce9-ff92-4d68-b15a-f82be6d6ca78-istio-token\") pod \"e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd\" (UID: \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\") " pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" Oct 30 13:18:52.927500 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:52.927442 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3489fce9-ff92-4d68-b15a-f82be6d6ca78-istio-envoy\") pod \"e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd\" (UID: \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\") " pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" Oct 30 13:18:52.927500 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:52.927479 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3489fce9-ff92-4d68-b15a-f82be6d6ca78-istio-data\") pod \"e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd\" (UID: \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\") " pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" Oct 30 13:18:52.927785 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:52.927596 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3489fce9-ff92-4d68-b15a-f82be6d6ca78-credential-socket\") pod \"e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd\" (UID: \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\") " pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" Oct 30 13:18:52.927785 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:52.927599 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3489fce9-ff92-4d68-b15a-f82be6d6ca78-workload-certs\") pod \"e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd\" (UID: \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\") " pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" Oct 30 13:18:52.927901 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:52.927783 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3489fce9-ff92-4d68-b15a-f82be6d6ca78-workload-socket\") pod \"e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd\" (UID: \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\") " pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" Oct 30 13:18:52.927901 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:52.927853 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3489fce9-ff92-4d68-b15a-f82be6d6ca78-istio-data\") pod \"e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd\" (UID: \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\") " pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" Oct 30 13:18:52.928082 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:52.928060 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3489fce9-ff92-4d68-b15a-f82be6d6ca78-istiod-ca-cert\") pod \"e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd\" (UID: \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\") " pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" Oct 30 13:18:52.929697 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:52.929676 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3489fce9-ff92-4d68-b15a-f82be6d6ca78-istio-envoy\") pod \"e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd\" (UID: \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\") " pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" Oct 30 13:18:52.929950 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:52.929927 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3489fce9-ff92-4d68-b15a-f82be6d6ca78-istio-podinfo\") pod \"e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd\" (UID: \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\") " pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" Oct 30 13:18:52.938567 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:52.938541 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:52.951695 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:52.951669 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3489fce9-ff92-4d68-b15a-f82be6d6ca78-istio-token\") pod \"e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd\" (UID: \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\") " pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" Oct 30 13:18:52.952286 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:52.952262 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9xq6\" (UniqueName: \"kubernetes.io/projected/3489fce9-ff92-4d68-b15a-f82be6d6ca78-kube-api-access-p9xq6\") pod \"e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd\" (UID: \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\") " pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" Oct 30 13:18:53.072139 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:53.072063 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" Oct 30 13:18:53.416346 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:53.416278 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd"] Oct 30 13:18:53.419894 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:18:53.419866 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3489fce9_ff92_4d68_b15a_f82be6d6ca78.slice/crio-afdaabaa00c5b79795fb6faef452aa55d49b35e376da1d6a155efed70bcbe26a WatchSource:0}: Error finding container afdaabaa00c5b79795fb6faef452aa55d49b35e376da1d6a155efed70bcbe26a: Status 404 returned error can't find the container with id afdaabaa00c5b79795fb6faef452aa55d49b35e376da1d6a155efed70bcbe26a Oct 30 13:18:53.421920 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:53.421885 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:18:53.421997 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:53.421952 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:18:53.421997 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:53.421980 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:18:53.437151 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:53.437120 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" event={"ID":"3489fce9-ff92-4d68-b15a-f82be6d6ca78","Type":"ContainerStarted","Data":"afdaabaa00c5b79795fb6faef452aa55d49b35e376da1d6a155efed70bcbe26a"} Oct 30 13:18:53.553562 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:18:53.553528 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:18:54.443393 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:54.443358 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" event={"ID":"3489fce9-ff92-4d68-b15a-f82be6d6ca78","Type":"ContainerStarted","Data":"0441bc0510c58ff10f4a23dc4b36a96d1d4cec993c47dc7e6b26d54fe78ca6d2"} Oct 30 13:18:54.513022 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:54.512964 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" podStartSLOduration=2.5129478240000003 podStartE2EDuration="2.512947824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:54.500969668 +0000 UTC m=+1590.970385432" watchObservedRunningTime="2026-10-30 13:18:54.512947824 +0000 UTC m=+1590.982363573" Oct 30 13:18:55.073099 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:55.073069 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" Oct 30 13:18:55.077750 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:55.077725 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" Oct 30 13:18:55.448229 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:55.448197 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" Oct 30 13:18:55.449102 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:55.449071 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" Oct 30 13:18:56.916605 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:56.916564 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:57.120829 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:57.120794 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:58.662934 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:58.662903 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:58.801929 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:58.801895 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:58.863442 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:58.863411 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-cadb6b-59cc7dwmjr"] Oct 30 13:18:58.866686 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:58.866666 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cadb6b-59cc7dwmjr" Oct 30 13:18:58.886351 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:58.886330 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-cadb6b\"" Oct 30 13:18:58.894561 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:58.894540 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-bcf9abfb-dockercfg-67fs6\"" Oct 30 13:18:58.896146 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:58.896122 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-cadb6b-59cc7dwmjr"] Oct 30 13:18:58.924719 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:58.924664 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:58.964838 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:58.964815 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-cadb6b-c4789c8445dz8f"] Oct 30 13:18:58.969119 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:58.969100 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-cadb6b-c4789c8445dz8f" Oct 30 13:18:58.982427 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:58.982400 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/259cc7fa-4775-4c96-8e5b-bbe01c388572-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-cadb6b-59cc7dwmjr\" (UID: \"259cc7fa-4775-4c96-8e5b-bbe01c388572\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cadb6b-59cc7dwmjr" Oct 30 13:18:58.982550 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:58.982466 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb5xh\" (UniqueName: \"kubernetes.io/projected/259cc7fa-4775-4c96-8e5b-bbe01c388572-kube-api-access-vb5xh\") pod \"payload-pre-processing-e2e-worker-single-main-cadb6b-59cc7dwmjr\" (UID: \"259cc7fa-4775-4c96-8e5b-bbe01c388572\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cadb6b-59cc7dwmjr" Oct 30 13:18:58.994274 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:58.994185 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-cadb6b-c4789c8445dz8f"] Oct 30 13:18:59.083420 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:59.083391 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/259cc7fa-4775-4c96-8e5b-bbe01c388572-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-cadb6b-59cc7dwmjr\" (UID: \"259cc7fa-4775-4c96-8e5b-bbe01c388572\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cadb6b-59cc7dwmjr" Oct 30 13:18:59.083574 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:59.083442 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/047f63d3-99a4-4ca8-9d37-1ea4137e0623-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-cadb6b-c4789c8445dz8f\" (UID: \"047f63d3-99a4-4ca8-9d37-1ea4137e0623\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-cadb6b-c4789c8445dz8f" Oct 30 13:18:59.083574 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:59.083560 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb5xh\" (UniqueName: \"kubernetes.io/projected/259cc7fa-4775-4c96-8e5b-bbe01c388572-kube-api-access-vb5xh\") pod \"payload-pre-processing-e2e-worker-single-main-cadb6b-59cc7dwmjr\" (UID: \"259cc7fa-4775-4c96-8e5b-bbe01c388572\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cadb6b-59cc7dwmjr" Oct 30 13:18:59.083740 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:59.083720 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmt5r\" (UniqueName: \"kubernetes.io/projected/047f63d3-99a4-4ca8-9d37-1ea4137e0623-kube-api-access-tmt5r\") pod \"payload-processing-e2e-worker-single-main-cadb6b-c4789c8445dz8f\" (UID: \"047f63d3-99a4-4ca8-9d37-1ea4137e0623\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-cadb6b-c4789c8445dz8f" Oct 30 13:18:59.084239 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:59.084220 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/259cc7fa-4775-4c96-8e5b-bbe01c388572-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-cadb6b-59cc7dwmjr\" (UID: \"259cc7fa-4775-4c96-8e5b-bbe01c388572\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cadb6b-59cc7dwmjr" Oct 30 13:18:59.107920 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:59.107898 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb5xh\" (UniqueName: \"kubernetes.io/projected/259cc7fa-4775-4c96-8e5b-bbe01c388572-kube-api-access-vb5xh\") pod \"payload-pre-processing-e2e-worker-single-main-cadb6b-59cc7dwmjr\" (UID: \"259cc7fa-4775-4c96-8e5b-bbe01c388572\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cadb6b-59cc7dwmjr" Oct 30 13:18:59.179664 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:59.179586 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cadb6b-59cc7dwmjr" Oct 30 13:18:59.185133 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:59.185113 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmt5r\" (UniqueName: \"kubernetes.io/projected/047f63d3-99a4-4ca8-9d37-1ea4137e0623-kube-api-access-tmt5r\") pod \"payload-processing-e2e-worker-single-main-cadb6b-c4789c8445dz8f\" (UID: \"047f63d3-99a4-4ca8-9d37-1ea4137e0623\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-cadb6b-c4789c8445dz8f" Oct 30 13:18:59.185205 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:59.185181 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/047f63d3-99a4-4ca8-9d37-1ea4137e0623-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-cadb6b-c4789c8445dz8f\" (UID: \"047f63d3-99a4-4ca8-9d37-1ea4137e0623\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-cadb6b-c4789c8445dz8f" Oct 30 13:18:59.185853 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:59.185836 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/047f63d3-99a4-4ca8-9d37-1ea4137e0623-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-cadb6b-c4789c8445dz8f\" (UID: \"047f63d3-99a4-4ca8-9d37-1ea4137e0623\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-cadb6b-c4789c8445dz8f" Oct 30 13:18:59.209383 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:59.209363 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmt5r\" (UniqueName: \"kubernetes.io/projected/047f63d3-99a4-4ca8-9d37-1ea4137e0623-kube-api-access-tmt5r\") pod \"payload-processing-e2e-worker-single-main-cadb6b-c4789c8445dz8f\" (UID: \"047f63d3-99a4-4ca8-9d37-1ea4137e0623\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-cadb6b-c4789c8445dz8f" Oct 30 13:18:59.283954 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:59.283921 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-cadb6b-c4789c8445dz8f" Oct 30 13:18:59.484861 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:59.484831 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-cadb6b-59cc7dwmjr"] Oct 30 13:18:59.489062 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:18:59.488735 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod259cc7fa_4775_4c96_8e5b_bbe01c388572.slice/crio-b60b2be66d62b5e16b84e4cb23c153af91d5c3eeb6e2ca555f75602cb8c812ed WatchSource:0}: Error finding container b60b2be66d62b5e16b84e4cb23c153af91d5c3eeb6e2ca555f75602cb8c812ed: Status 404 returned error can't find the container with id b60b2be66d62b5e16b84e4cb23c153af91d5c3eeb6e2ca555f75602cb8c812ed Oct 30 13:18:59.585880 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:18:59.585853 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-cadb6b-c4789c8445dz8f"] Oct 30 13:18:59.589151 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:18:59.589128 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod047f63d3_99a4_4ca8_9d37_1ea4137e0623.slice/crio-5800f956bedda72cb0a63b72ed8cc0d66a002ffa24412ecb49e5e9fe67afe54b WatchSource:0}: Error finding container 5800f956bedda72cb0a63b72ed8cc0d66a002ffa24412ecb49e5e9fe67afe54b: Status 404 returned error can't find the container with id 5800f956bedda72cb0a63b72ed8cc0d66a002ffa24412ecb49e5e9fe67afe54b Oct 30 13:19:00.474476 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:00.474435 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-cadb6b-c4789c8445dz8f" event={"ID":"047f63d3-99a4-4ca8-9d37-1ea4137e0623","Type":"ContainerStarted","Data":"34e39fa5369876f449fbccff8e422ef6a5dafaea9b268771bfbc908ab2417715"} Oct 30 13:19:00.474952 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:00.474496 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-cadb6b-c4789c8445dz8f" event={"ID":"047f63d3-99a4-4ca8-9d37-1ea4137e0623","Type":"ContainerStarted","Data":"5800f956bedda72cb0a63b72ed8cc0d66a002ffa24412ecb49e5e9fe67afe54b"} Oct 30 13:19:00.474952 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:00.474634 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-cadb6b-c4789c8445dz8f" Oct 30 13:19:00.476193 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:00.476171 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cadb6b-59cc7dwmjr" event={"ID":"259cc7fa-4775-4c96-8e5b-bbe01c388572","Type":"ContainerStarted","Data":"75bd66eb23f3ad13ef8820533ca57372290908303a68d177dbde7e090c2601a8"} Oct 30 13:19:00.476329 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:00.476206 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cadb6b-59cc7dwmjr" event={"ID":"259cc7fa-4775-4c96-8e5b-bbe01c388572","Type":"ContainerStarted","Data":"b60b2be66d62b5e16b84e4cb23c153af91d5c3eeb6e2ca555f75602cb8c812ed"} Oct 30 13:19:00.476329 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:00.476271 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cadb6b-59cc7dwmjr" Oct 30 13:19:00.517234 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:00.517111 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-cadb6b-c4789c8445dz8f" podStartSLOduration=2.517096659 podStartE2EDuration="2.517096659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:00.516165639 +0000 UTC m=+1596.985581423" watchObservedRunningTime="2026-10-30 13:19:00.517096659 +0000 UTC m=+1596.986512403" Oct 30 13:19:00.562236 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:00.562173 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cadb6b-59cc7dwmjr" podStartSLOduration=2.562154425 podStartE2EDuration="2.562154425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:00.561393693 +0000 UTC m=+1597.030809458" watchObservedRunningTime="2026-10-30 13:19:00.562154425 +0000 UTC m=+1597.031570168" Oct 30 13:19:00.802333 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:00.802251 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:02.532174 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:02.532130 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:19:02.742063 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:02.742020 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:04.704868 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:04.704823 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:06.510986 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:06.510946 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:08.465635 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:08.462909 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:10.423953 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:10.423917 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:11.458408 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:11.458225 5922 patch_prober.go:29] interesting pod/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:19:11.458408 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:11.458381 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" podUID="3489fce9-ff92-4d68-b15a-f82be6d6ca78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:19:11.484929 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:11.484896 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cadb6b-59cc7dwmjr" Oct 30 13:19:11.485162 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:11.485140 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-cadb6b-c4789c8445dz8f" Oct 30 13:19:11.533068 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:11.533046 5922 generic.go:356] "Generic (PLEG): container finished" podID="3489fce9-ff92-4d68-b15a-f82be6d6ca78" containerID="0441bc0510c58ff10f4a23dc4b36a96d1d4cec993c47dc7e6b26d54fe78ca6d2" exitCode=0 Oct 30 13:19:11.533195 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:11.533126 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" event={"ID":"3489fce9-ff92-4d68-b15a-f82be6d6ca78","Type":"ContainerDied","Data":"0441bc0510c58ff10f4a23dc4b36a96d1d4cec993c47dc7e6b26d54fe78ca6d2"} Oct 30 13:19:11.533494 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:11.533478 5922 scope.go:122] "RemoveContainer" containerID="0441bc0510c58ff10f4a23dc4b36a96d1d4cec993c47dc7e6b26d54fe78ca6d2" Oct 30 13:19:11.534566 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:11.534532 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:19:11.534643 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:11.534595 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:19:11.534690 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:11.534646 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:19:11.824944 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:11.824845 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:12.309975 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:12.309933 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:12.440305 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:12.440276 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:12.539939 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:12.539904 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" event={"ID":"3489fce9-ff92-4d68-b15a-f82be6d6ca78","Type":"ContainerStarted","Data":"b520cdf803093e2a75fb3dccbb716a792893a723345791191c0f34b99e07ba4d"} Oct 30 13:19:12.587799 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:12.587199 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:12.704526 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:12.704484 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:13.078639 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:13.078265 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" Oct 30 13:19:13.078639 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:13.078388 5922 patch_prober.go:29] interesting pod/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 13:19:13.078639 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:13.078441 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" podUID="3489fce9-ff92-4d68-b15a-f82be6d6ca78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 13:19:14.086756 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:14.086716 5922 patch_prober.go:29] interesting pod/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 13:19:14.087198 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:14.086773 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" podUID="3489fce9-ff92-4d68-b15a-f82be6d6ca78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 13:19:14.291410 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:14.291365 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-cadb6b/simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8"] Oct 30 13:19:14.294400 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:14.294378 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-cadb6b/simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8" Oct 30 13:19:14.307460 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:14.307431 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-cadb6b/simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8"] Oct 30 13:19:14.310273 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:14.310242 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:14.319138 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:14.319111 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-cadb6b\"/\"openshift-service-ca.crt\"" Oct 30 13:19:14.319282 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:14.319168 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-cadb6b\"/\"kube-root-ca.crt\"" Oct 30 13:19:14.385683 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:14.384923 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-cadb6b\"/\"simulator-single-main-cadb6b-kserve-self-signed-certs\"" Oct 30 13:19:14.385683 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:14.385064 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-cadb6b\"/\"default-dockercfg-tdjvq\"" Oct 30 13:19:14.432348 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:14.432319 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:14.457515 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:14.457490 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8382fbd3-40e1-4978-be98-37ec97883337-model-cache\") pod \"simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8\" (UID: \"8382fbd3-40e1-4978-be98-37ec97883337\") " pod="e2e-models-e2e-worker-single-main-cadb6b/simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8" Oct 30 13:19:14.457672 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:14.457531 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8382fbd3-40e1-4978-be98-37ec97883337-tmp-dir\") pod \"simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8\" (UID: \"8382fbd3-40e1-4978-be98-37ec97883337\") " pod="e2e-models-e2e-worker-single-main-cadb6b/simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8" Oct 30 13:19:14.457672 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:14.457560 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8382fbd3-40e1-4978-be98-37ec97883337-dshm\") pod \"simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8\" (UID: \"8382fbd3-40e1-4978-be98-37ec97883337\") " pod="e2e-models-e2e-worker-single-main-cadb6b/simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8" Oct 30 13:19:14.457763 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:14.457704 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rfz4\" (UniqueName: \"kubernetes.io/projected/8382fbd3-40e1-4978-be98-37ec97883337-kube-api-access-2rfz4\") pod \"simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8\" (UID: \"8382fbd3-40e1-4978-be98-37ec97883337\") " pod="e2e-models-e2e-worker-single-main-cadb6b/simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8" Oct 30 13:19:14.457763 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:14.457730 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8382fbd3-40e1-4978-be98-37ec97883337-home\") pod \"simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8\" (UID: \"8382fbd3-40e1-4978-be98-37ec97883337\") " pod="e2e-models-e2e-worker-single-main-cadb6b/simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8" Oct 30 13:19:14.457763 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:14.457752 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8382fbd3-40e1-4978-be98-37ec97883337-tls-certs\") pod \"simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8\" (UID: \"8382fbd3-40e1-4978-be98-37ec97883337\") " pod="e2e-models-e2e-worker-single-main-cadb6b/simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8" Oct 30 13:19:14.559331 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:14.559300 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8382fbd3-40e1-4978-be98-37ec97883337-model-cache\") pod \"simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8\" (UID: \"8382fbd3-40e1-4978-be98-37ec97883337\") " pod="e2e-models-e2e-worker-single-main-cadb6b/simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8" Oct 30 13:19:14.559541 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:14.559365 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8382fbd3-40e1-4978-be98-37ec97883337-tmp-dir\") pod \"simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8\" (UID: \"8382fbd3-40e1-4978-be98-37ec97883337\") " pod="e2e-models-e2e-worker-single-main-cadb6b/simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8" Oct 30 13:19:14.559541 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:14.559402 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8382fbd3-40e1-4978-be98-37ec97883337-dshm\") pod \"simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8\" (UID: \"8382fbd3-40e1-4978-be98-37ec97883337\") " pod="e2e-models-e2e-worker-single-main-cadb6b/simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8" Oct 30 13:19:14.559541 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:14.559437 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rfz4\" (UniqueName: \"kubernetes.io/projected/8382fbd3-40e1-4978-be98-37ec97883337-kube-api-access-2rfz4\") pod \"simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8\" (UID: \"8382fbd3-40e1-4978-be98-37ec97883337\") " pod="e2e-models-e2e-worker-single-main-cadb6b/simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8" Oct 30 13:19:14.559541 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:14.559453 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8382fbd3-40e1-4978-be98-37ec97883337-home\") pod \"simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8\" (UID: \"8382fbd3-40e1-4978-be98-37ec97883337\") " pod="e2e-models-e2e-worker-single-main-cadb6b/simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8" Oct 30 13:19:14.559842 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:14.559812 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8382fbd3-40e1-4978-be98-37ec97883337-model-cache\") pod \"simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8\" (UID: \"8382fbd3-40e1-4978-be98-37ec97883337\") " pod="e2e-models-e2e-worker-single-main-cadb6b/simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8" Oct 30 13:19:14.559893 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:14.559856 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8382fbd3-40e1-4978-be98-37ec97883337-tmp-dir\") pod \"simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8\" (UID: \"8382fbd3-40e1-4978-be98-37ec97883337\") " pod="e2e-models-e2e-worker-single-main-cadb6b/simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8" Oct 30 13:19:14.559985 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:14.559560 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8382fbd3-40e1-4978-be98-37ec97883337-tls-certs\") pod \"simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8\" (UID: \"8382fbd3-40e1-4978-be98-37ec97883337\") " pod="e2e-models-e2e-worker-single-main-cadb6b/simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8" Oct 30 13:19:14.560024 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:14.560002 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8382fbd3-40e1-4978-be98-37ec97883337-home\") pod \"simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8\" (UID: \"8382fbd3-40e1-4978-be98-37ec97883337\") " pod="e2e-models-e2e-worker-single-main-cadb6b/simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8" Oct 30 13:19:14.582635 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:14.580175 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8382fbd3-40e1-4978-be98-37ec97883337-tls-certs\") pod \"simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8\" (UID: \"8382fbd3-40e1-4978-be98-37ec97883337\") " pod="e2e-models-e2e-worker-single-main-cadb6b/simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8" Oct 30 13:19:14.582635 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:14.581829 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8382fbd3-40e1-4978-be98-37ec97883337-dshm\") pod \"simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8\" (UID: \"8382fbd3-40e1-4978-be98-37ec97883337\") " pod="e2e-models-e2e-worker-single-main-cadb6b/simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8" Oct 30 13:19:14.589534 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:14.589487 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rfz4\" (UniqueName: \"kubernetes.io/projected/8382fbd3-40e1-4978-be98-37ec97883337-kube-api-access-2rfz4\") pod \"simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8\" (UID: \"8382fbd3-40e1-4978-be98-37ec97883337\") " pod="e2e-models-e2e-worker-single-main-cadb6b/simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8" Oct 30 13:19:14.607356 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:14.607336 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-cadb6b/simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8" Oct 30 13:19:14.921546 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:14.921512 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-cadb6b/simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8"] Oct 30 13:19:14.924331 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:19:14.924302 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8382fbd3_40e1_4978_be98_37ec97883337.slice/crio-eb0e9d75cc5e70e03a7169b5a6b104fe6b0979fad88284b66d7940c486e14b8b WatchSource:0}: Error finding container eb0e9d75cc5e70e03a7169b5a6b104fe6b0979fad88284b66d7940c486e14b8b: Status 404 returned error can't find the container with id eb0e9d75cc5e70e03a7169b5a6b104fe6b0979fad88284b66d7940c486e14b8b Oct 30 13:19:15.086637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:15.083300 5922 patch_prober.go:29] interesting pod/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 13:19:15.086637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:15.083361 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" podUID="3489fce9-ff92-4d68-b15a-f82be6d6ca78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 13:19:15.557493 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:15.557453 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-cadb6b/simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8" event={"ID":"8382fbd3-40e1-4978-be98-37ec97883337","Type":"ContainerStarted","Data":"130db8e05c2ff1143096b079d3befcea64a818f3e63809eeb74fa2cb1c35938a"} Oct 30 13:19:15.557493 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:15.557506 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-cadb6b/simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8" event={"ID":"8382fbd3-40e1-4978-be98-37ec97883337","Type":"ContainerStarted","Data":"eb0e9d75cc5e70e03a7169b5a6b104fe6b0979fad88284b66d7940c486e14b8b"} Oct 30 13:19:15.606036 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:15.605987 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-cadb6b/simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8" podStartSLOduration=1.605971379 podStartE2EDuration="1.605971379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:15.605200034 +0000 UTC m=+1612.074615799" watchObservedRunningTime="2026-10-30 13:19:15.605971379 +0000 UTC m=+1612.075387124" Oct 30 13:19:15.608087 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:15.608043 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-cadb6b/simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8" Oct 30 13:19:16.084639 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:16.080058 5922 patch_prober.go:29] interesting pod/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 13:19:16.084639 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:16.080118 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" podUID="3489fce9-ff92-4d68-b15a-f82be6d6ca78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 13:19:16.154416 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:16.154377 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:16.552633 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:16.551868 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:19:17.082149 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:17.082114 5922 patch_prober.go:29] interesting pod/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 13:19:17.082455 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:17.082411 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" podUID="3489fce9-ff92-4d68-b15a-f82be6d6ca78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 13:19:18.108278 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:18.108243 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" Oct 30 13:19:18.108980 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:18.108682 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" Oct 30 13:19:18.109717 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:18.109680 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" Oct 30 13:19:18.321891 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:18.321850 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:20.102432 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:20.102396 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:22.010120 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:22.010086 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:23.551418 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:19:23.551386 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:19:23.794388 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:23.794353 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:24.607763 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:24.607734 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-cadb6b/simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8" Oct 30 13:19:24.620402 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:24.620378 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-cadb6b/simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8" Oct 30 13:19:24.652683 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:24.652652 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-cadb6b/simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8" Oct 30 13:19:25.204163 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:25.204136 5922 scope.go:122] "RemoveContainer" containerID="7442678e5cddbc80064518ab5cef773d84cf01652e392b873f514c7d1d88ddb3" Oct 30 13:19:25.212674 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:25.212656 5922 scope.go:122] "RemoveContainer" containerID="45a21f64fbecf72930cb4860987efc909d9f8bb5417773bb06baed95c7f54ea1" Oct 30 13:19:25.221223 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:25.221200 5922 scope.go:122] "RemoveContainer" containerID="cec58b9d2860dfe8ec21d2e3b6596249a65d1ba9f5a76b2ad2c4ae542c61fcf0" Oct 30 13:19:25.228585 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:25.228565 5922 scope.go:122] "RemoveContainer" containerID="80b0b7ad6165b05d13b96a3dd18d10c8572298212df926ba9854f3e0221cfaef" Oct 30 13:19:25.237321 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:25.237306 5922 scope.go:122] "RemoveContainer" containerID="a3159e6cdfb74c3491c86fa9e7552ed0afc53029e08cef787d03227e91598f2f" Oct 30 13:19:25.244539 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:25.244520 5922 scope.go:122] "RemoveContainer" containerID="3ddc81c56d742dc0b5ff31f97dcff7f3cddbd1da88d900d17dff91f50db01b32" Oct 30 13:19:25.251704 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:25.251681 5922 scope.go:122] "RemoveContainer" containerID="22aa21c487fb95e41891bcad5aaa0f908a44037b9ff1168dfbfc524df1aa32e7" Oct 30 13:19:25.258714 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:25.258695 5922 scope.go:122] "RemoveContainer" containerID="66bd5a442ae52e7c5cdfa22e310fde3ed76b3bf90a2abc7eb57c59c8f0c363f5" Oct 30 13:19:25.623864 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:25.623792 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:27.619203 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:27.619158 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:29.416036 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:29.415997 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:31.316832 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:31.316793 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:31.561984 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:31.561947 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:19:33.122622 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:33.122585 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:34.326237 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:34.326205 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:34.981318 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:34.981232 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:36.827256 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:36.827221 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:37.210246 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:37.210201 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:38.482207 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:38.482172 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95d7722d-6878x"] Oct 30 13:19:38.487900 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:38.487878 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d7722d-6878x" Oct 30 13:19:38.509237 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:38.508794 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95d7722d-6878x"] Oct 30 13:19:38.517224 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:38.517198 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:19:38.517352 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:38.517283 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:19:38.517434 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:38.517415 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:19:38.517586 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:38.517568 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:19:38.614585 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:38.614546 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:38.627690 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:38.627659 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1583f6bb-57f7-4fec-b7d1-6a190e6070b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95d7722d-6878x\" (UID: \"1583f6bb-57f7-4fec-b7d1-6a190e6070b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d7722d-6878x" Oct 30 13:19:38.627859 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:38.627779 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1583f6bb-57f7-4fec-b7d1-6a190e6070b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95d7722d-6878x\" (UID: \"1583f6bb-57f7-4fec-b7d1-6a190e6070b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d7722d-6878x" Oct 30 13:19:38.627911 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:38.627864 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8flgt\" (UniqueName: \"kubernetes.io/projected/1583f6bb-57f7-4fec-b7d1-6a190e6070b4-kube-api-access-8flgt\") pod \"maas-api-revoke-sub-95d7722d-6878x\" (UID: \"1583f6bb-57f7-4fec-b7d1-6a190e6070b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d7722d-6878x" Oct 30 13:19:38.729181 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:38.729148 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1583f6bb-57f7-4fec-b7d1-6a190e6070b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95d7722d-6878x\" (UID: \"1583f6bb-57f7-4fec-b7d1-6a190e6070b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d7722d-6878x" Oct 30 13:19:38.729345 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:38.729226 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1583f6bb-57f7-4fec-b7d1-6a190e6070b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95d7722d-6878x\" (UID: \"1583f6bb-57f7-4fec-b7d1-6a190e6070b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d7722d-6878x" Oct 30 13:19:38.729345 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:38.729327 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8flgt\" (UniqueName: \"kubernetes.io/projected/1583f6bb-57f7-4fec-b7d1-6a190e6070b4-kube-api-access-8flgt\") pod \"maas-api-revoke-sub-95d7722d-6878x\" (UID: \"1583f6bb-57f7-4fec-b7d1-6a190e6070b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d7722d-6878x" Oct 30 13:19:38.729860 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:38.729838 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1583f6bb-57f7-4fec-b7d1-6a190e6070b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95d7722d-6878x\" (UID: \"1583f6bb-57f7-4fec-b7d1-6a190e6070b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d7722d-6878x" Oct 30 13:19:38.731666 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:38.731607 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1583f6bb-57f7-4fec-b7d1-6a190e6070b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95d7722d-6878x\" (UID: \"1583f6bb-57f7-4fec-b7d1-6a190e6070b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d7722d-6878x" Oct 30 13:19:38.753725 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:38.753665 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8flgt\" (UniqueName: \"kubernetes.io/projected/1583f6bb-57f7-4fec-b7d1-6a190e6070b4-kube-api-access-8flgt\") pod \"maas-api-revoke-sub-95d7722d-6878x\" (UID: \"1583f6bb-57f7-4fec-b7d1-6a190e6070b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d7722d-6878x" Oct 30 13:19:38.800632 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:38.800592 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d7722d-6878x" Oct 30 13:19:39.097005 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:39.096929 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95d7722d-6878x"] Oct 30 13:19:39.100566 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:19:39.100538 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1583f6bb_57f7_4fec_b7d1_6a190e6070b4.slice/crio-7caa038e247db22d91f17b4d2c1d1e61ddf56550653ba82e6d856681bc656c95 WatchSource:0}: Error finding container 7caa038e247db22d91f17b4d2c1d1e61ddf56550653ba82e6d856681bc656c95: Status 404 returned error can't find the container with id 7caa038e247db22d91f17b4d2c1d1e61ddf56550653ba82e6d856681bc656c95 Oct 30 13:19:39.716952 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:39.716907 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d7722d-6878x" event={"ID":"1583f6bb-57f7-4fec-b7d1-6a190e6070b4","Type":"ContainerStarted","Data":"8ac4354d5f023767eadc50d230502a059931aa1266c321d72dedc4afc662597b"} Oct 30 13:19:39.717517 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:39.717489 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d7722d-6878x" event={"ID":"1583f6bb-57f7-4fec-b7d1-6a190e6070b4","Type":"ContainerStarted","Data":"7caa038e247db22d91f17b4d2c1d1e61ddf56550653ba82e6d856681bc656c95"} Oct 30 13:19:39.761956 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:39.761912 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d7722d-6878x" podStartSLOduration=1.761900409 podStartE2EDuration="1.761900409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:39.761400713 +0000 UTC m=+1636.230816461" watchObservedRunningTime="2026-10-30 13:19:39.761900409 +0000 UTC m=+1636.231316132" Oct 30 13:19:40.496495 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:40.496461 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:40.723786 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:40.723751 5922 generic.go:356] "Generic (PLEG): container finished" podID="1583f6bb-57f7-4fec-b7d1-6a190e6070b4" containerID="8ac4354d5f023767eadc50d230502a059931aa1266c321d72dedc4afc662597b" exitCode=0 Oct 30 13:19:40.724160 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:40.723829 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d7722d-6878x" event={"ID":"1583f6bb-57f7-4fec-b7d1-6a190e6070b4","Type":"ContainerDied","Data":"8ac4354d5f023767eadc50d230502a059931aa1266c321d72dedc4afc662597b"} Oct 30 13:19:41.863517 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:41.863489 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d7722d-6878x" Oct 30 13:19:41.958832 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:41.958808 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1583f6bb-57f7-4fec-b7d1-6a190e6070b4-kube-api-access-8flgt\" (UniqueName: \"kubernetes.io/projected/1583f6bb-57f7-4fec-b7d1-6a190e6070b4-kube-api-access-8flgt\") pod \"1583f6bb-57f7-4fec-b7d1-6a190e6070b4\" (UID: \"1583f6bb-57f7-4fec-b7d1-6a190e6070b4\") " Oct 30 13:19:41.958976 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:41.958865 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1583f6bb-57f7-4fec-b7d1-6a190e6070b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1583f6bb-57f7-4fec-b7d1-6a190e6070b4-maas-api-xks-ca\") pod \"1583f6bb-57f7-4fec-b7d1-6a190e6070b4\" (UID: \"1583f6bb-57f7-4fec-b7d1-6a190e6070b4\") " Oct 30 13:19:41.958976 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:41.958930 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1583f6bb-57f7-4fec-b7d1-6a190e6070b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1583f6bb-57f7-4fec-b7d1-6a190e6070b4-maas-api-openshift-ca\") pod \"1583f6bb-57f7-4fec-b7d1-6a190e6070b4\" (UID: \"1583f6bb-57f7-4fec-b7d1-6a190e6070b4\") " Oct 30 13:19:41.959311 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:41.959279 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1583f6bb-57f7-4fec-b7d1-6a190e6070b4-maas-api-openshift-ca" pod "1583f6bb-57f7-4fec-b7d1-6a190e6070b4" (UID: "1583f6bb-57f7-4fec-b7d1-6a190e6070b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:19:41.960971 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:41.960953 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1583f6bb-57f7-4fec-b7d1-6a190e6070b4-kube-api-access-8flgt" pod "1583f6bb-57f7-4fec-b7d1-6a190e6070b4" (UID: "1583f6bb-57f7-4fec-b7d1-6a190e6070b4"). InnerVolumeSpecName "kube-api-access-8flgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:19:41.961123 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:41.961101 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1583f6bb-57f7-4fec-b7d1-6a190e6070b4-maas-api-xks-ca" pod "1583f6bb-57f7-4fec-b7d1-6a190e6070b4" (UID: "1583f6bb-57f7-4fec-b7d1-6a190e6070b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:19:42.060271 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:42.060205 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8flgt\" (UniqueName: \"kubernetes.io/projected/1583f6bb-57f7-4fec-b7d1-6a190e6070b4-kube-api-access-8flgt\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:19:42.060271 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:42.060230 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1583f6bb-57f7-4fec-b7d1-6a190e6070b4-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:19:42.060271 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:42.060241 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1583f6bb-57f7-4fec-b7d1-6a190e6070b4-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:19:42.278360 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:42.278324 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:42.734830 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:42.734805 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d7722d-6878x" Oct 30 13:19:42.735032 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:42.734835 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d7722d-6878x" event={"ID":"1583f6bb-57f7-4fec-b7d1-6a190e6070b4","Type":"ContainerDied","Data":"7caa038e247db22d91f17b4d2c1d1e61ddf56550653ba82e6d856681bc656c95"} Oct 30 13:19:42.735032 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:42.734879 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7caa038e247db22d91f17b4d2c1d1e61ddf56550653ba82e6d856681bc656c95" Oct 30 13:19:44.138489 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:44.138365 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:45.896179 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:45.896137 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:47.919278 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:47.919240 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:49.454129 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:49.454093 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:49.603256 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:49.603226 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cadb6b-gr6sl"] Oct 30 13:19:49.603678 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:49.603665 5922 cpu_manager.go:414] "containerMap: removing container" podUID="1583f6bb-57f7-4fec-b7d1-6a190e6070b4" containerName="revoke-keys" Oct 30 13:19:49.603745 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:49.603680 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1583f6bb-57f7-4fec-b7d1-6a190e6070b4" containerName="revoke-keys" Oct 30 13:19:49.603782 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:49.603773 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="1583f6bb-57f7-4fec-b7d1-6a190e6070b4" containerName="revoke-keys" Oct 30 13:19:49.606953 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:49.606937 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cadb6b-gr6sl" Oct 30 13:19:49.624462 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:49.624441 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:19:49.625663 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:49.625639 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cadb6b-gr6sl"] Oct 30 13:19:49.635763 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:49.635743 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:19:49.635876 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:49.635767 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:19:49.635876 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:49.635792 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:19:49.678028 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:49.677992 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:49.728461 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:49.728360 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k6vm\" (UniqueName: \"kubernetes.io/projected/7172e198-7951-48a7-8429-06f4f11bf722-kube-api-access-7k6vm\") pod \"maas-api-revoke-keys-e2e-worker-single-main-cadb6b-gr6sl\" (UID: \"7172e198-7951-48a7-8429-06f4f11bf722\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cadb6b-gr6sl" Oct 30 13:19:49.728461 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:49.728419 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7172e198-7951-48a7-8429-06f4f11bf722-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-cadb6b-gr6sl\" (UID: \"7172e198-7951-48a7-8429-06f4f11bf722\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cadb6b-gr6sl" Oct 30 13:19:49.728706 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:49.728656 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7172e198-7951-48a7-8429-06f4f11bf722-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-cadb6b-gr6sl\" (UID: \"7172e198-7951-48a7-8429-06f4f11bf722\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cadb6b-gr6sl" Oct 30 13:19:49.830049 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:49.830018 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7172e198-7951-48a7-8429-06f4f11bf722-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-cadb6b-gr6sl\" (UID: \"7172e198-7951-48a7-8429-06f4f11bf722\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cadb6b-gr6sl" Oct 30 13:19:49.830207 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:49.830095 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k6vm\" (UniqueName: \"kubernetes.io/projected/7172e198-7951-48a7-8429-06f4f11bf722-kube-api-access-7k6vm\") pod \"maas-api-revoke-keys-e2e-worker-single-main-cadb6b-gr6sl\" (UID: \"7172e198-7951-48a7-8429-06f4f11bf722\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cadb6b-gr6sl" Oct 30 13:19:49.830207 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:49.830113 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7172e198-7951-48a7-8429-06f4f11bf722-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-cadb6b-gr6sl\" (UID: \"7172e198-7951-48a7-8429-06f4f11bf722\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cadb6b-gr6sl" Oct 30 13:19:49.830647 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:49.830596 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7172e198-7951-48a7-8429-06f4f11bf722-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-cadb6b-gr6sl\" (UID: \"7172e198-7951-48a7-8429-06f4f11bf722\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cadb6b-gr6sl" Oct 30 13:19:49.832403 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:49.832374 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7172e198-7951-48a7-8429-06f4f11bf722-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-cadb6b-gr6sl\" (UID: \"7172e198-7951-48a7-8429-06f4f11bf722\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cadb6b-gr6sl" Oct 30 13:19:49.847029 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:49.847006 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:19:49.854324 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:49.854300 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k6vm\" (UniqueName: \"kubernetes.io/projected/7172e198-7951-48a7-8429-06f4f11bf722-kube-api-access-7k6vm\") pod \"maas-api-revoke-keys-e2e-worker-single-main-cadb6b-gr6sl\" (UID: \"7172e198-7951-48a7-8429-06f4f11bf722\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cadb6b-gr6sl" Oct 30 13:19:49.920975 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:49.920949 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cadb6b-gr6sl" Oct 30 13:19:50.218151 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:50.218030 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cadb6b-gr6sl"] Oct 30 13:19:50.221136 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:19:50.221109 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7172e198_7951_48a7_8429_06f4f11bf722.slice/crio-c4b6bf47bda248d3f17be2ed5c2cdecfac8e4b1d5606c777f44b257f10be1bb2 WatchSource:0}: Error finding container c4b6bf47bda248d3f17be2ed5c2cdecfac8e4b1d5606c777f44b257f10be1bb2: Status 404 returned error can't find the container with id c4b6bf47bda248d3f17be2ed5c2cdecfac8e4b1d5606c777f44b257f10be1bb2 Oct 30 13:19:50.773405 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:50.773313 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cadb6b-gr6sl" event={"ID":"7172e198-7951-48a7-8429-06f4f11bf722","Type":"ContainerStarted","Data":"1cd7e086b87b95c2b1c7e05ae4684e594972633a4fa8ea2e801ed518ed8eb834"} Oct 30 13:19:50.773405 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:50.773363 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cadb6b-gr6sl" event={"ID":"7172e198-7951-48a7-8429-06f4f11bf722","Type":"ContainerStarted","Data":"c4b6bf47bda248d3f17be2ed5c2cdecfac8e4b1d5606c777f44b257f10be1bb2"} Oct 30 13:19:50.815242 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:50.815194 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cadb6b-gr6sl" podStartSLOduration=1.815179576 podStartE2EDuration="1.815179576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:50.814893624 +0000 UTC m=+1647.284309350" watchObservedRunningTime="2026-10-30 13:19:50.815179576 +0000 UTC m=+1647.284595321" Oct 30 13:19:51.668502 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:51.668463 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:51.779904 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:51.779872 5922 generic.go:356] "Generic (PLEG): container finished" podID="7172e198-7951-48a7-8429-06f4f11bf722" containerID="1cd7e086b87b95c2b1c7e05ae4684e594972633a4fa8ea2e801ed518ed8eb834" exitCode=0 Oct 30 13:19:51.780289 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:51.779930 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cadb6b-gr6sl" event={"ID":"7172e198-7951-48a7-8429-06f4f11bf722","Type":"ContainerDied","Data":"1cd7e086b87b95c2b1c7e05ae4684e594972633a4fa8ea2e801ed518ed8eb834"} Oct 30 13:19:52.921064 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:52.921038 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cadb6b-gr6sl" Oct 30 13:19:53.061699 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:53.061605 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7172e198-7951-48a7-8429-06f4f11bf722-kube-api-access-7k6vm\" (UniqueName: \"kubernetes.io/projected/7172e198-7951-48a7-8429-06f4f11bf722-kube-api-access-7k6vm\") pod \"7172e198-7951-48a7-8429-06f4f11bf722\" (UID: \"7172e198-7951-48a7-8429-06f4f11bf722\") " Oct 30 13:19:53.061699 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:53.061676 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7172e198-7951-48a7-8429-06f4f11bf722-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7172e198-7951-48a7-8429-06f4f11bf722-maas-api-xks-ca\") pod \"7172e198-7951-48a7-8429-06f4f11bf722\" (UID: \"7172e198-7951-48a7-8429-06f4f11bf722\") " Oct 30 13:19:53.061927 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:53.061748 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7172e198-7951-48a7-8429-06f4f11bf722-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7172e198-7951-48a7-8429-06f4f11bf722-maas-api-openshift-ca\") pod \"7172e198-7951-48a7-8429-06f4f11bf722\" (UID: \"7172e198-7951-48a7-8429-06f4f11bf722\") " Oct 30 13:19:53.062090 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:53.062063 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7172e198-7951-48a7-8429-06f4f11bf722-maas-api-openshift-ca" pod "7172e198-7951-48a7-8429-06f4f11bf722" (UID: "7172e198-7951-48a7-8429-06f4f11bf722"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:19:53.063669 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:53.063642 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7172e198-7951-48a7-8429-06f4f11bf722-kube-api-access-7k6vm" pod "7172e198-7951-48a7-8429-06f4f11bf722" (UID: "7172e198-7951-48a7-8429-06f4f11bf722"). InnerVolumeSpecName "kube-api-access-7k6vm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:19:53.063752 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:53.063650 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7172e198-7951-48a7-8429-06f4f11bf722-maas-api-xks-ca" pod "7172e198-7951-48a7-8429-06f4f11bf722" (UID: "7172e198-7951-48a7-8429-06f4f11bf722"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:19:53.162512 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:53.162474 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k6vm\" (UniqueName: \"kubernetes.io/projected/7172e198-7951-48a7-8429-06f4f11bf722-kube-api-access-7k6vm\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:19:53.162512 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:53.162511 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7172e198-7951-48a7-8429-06f4f11bf722-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:19:53.162667 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:53.162526 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7172e198-7951-48a7-8429-06f4f11bf722-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:19:53.552708 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:19:53.552674 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:19:53.614454 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:53.614420 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:53.790479 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:53.790442 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cadb6b-gr6sl" event={"ID":"7172e198-7951-48a7-8429-06f4f11bf722","Type":"ContainerDied","Data":"c4b6bf47bda248d3f17be2ed5c2cdecfac8e4b1d5606c777f44b257f10be1bb2"} Oct 30 13:19:53.790668 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:53.790518 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4b6bf47bda248d3f17be2ed5c2cdecfac8e4b1d5606c777f44b257f10be1bb2" Oct 30 13:19:53.790668 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:53.790459 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cadb6b-gr6sl" Oct 30 13:19:54.456439 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:54.456408 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-cadb6b/simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8"] Oct 30 13:19:54.657945 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:54.657910 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:55.535499 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:55.535465 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:57.315268 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:57.315230 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:59.127748 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:19:59.127714 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:00.058466 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:00.058436 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-cadb6b-c4789c8445dz8f"] Oct 30 13:20:00.103756 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:00.103718 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-cadb6b-59cc7dwmjr"] Oct 30 13:20:00.220423 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:00.220130 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:00.339751 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:00.339660 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:01.004320 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:01.004274 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:02.617222 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:02.617183 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:03.263158 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:03.263122 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:03.378125 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:03.378082 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:03.422007 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:03.421975 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cadb6b-gr6sl"] Oct 30 13:20:03.434162 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:03.434132 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cadb6b-gr6sl"] Oct 30 13:20:03.642789 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:03.642724 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7172e198-7951-48a7-8429-06f4f11bf722" path="/var/lib/kubelet/pods/7172e198-7951-48a7-8429-06f4f11bf722/volumes" Oct 30 13:20:03.888839 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:03.888803 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:04.007226 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:04.007193 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:04.847823 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:04.847792 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:20:05.058808 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:05.058757 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-cadb6b-c4789c8445dz8f" podUID="047f63d3-99a4-4ca8-9d37-1ea4137e0623" containerName="payload-processing" containerID="cri-o://34e39fa5369876f449fbccff8e422ef6a5dafaea9b268771bfbc908ab2417715" gracePeriod=25 Oct 30 13:20:05.104191 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:05.104096 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cadb6b-59cc7dwmjr" podUID="259cc7fa-4775-4c96-8e5b-bbe01c388572" containerName="payload-pre-processing" containerID="cri-o://75bd66eb23f3ad13ef8820533ca57372290908303a68d177dbde7e090c2601a8" gracePeriod=25 Oct 30 13:20:06.032313 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:06.032279 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cplvq"] Oct 30 13:20:06.044128 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:06.044099 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-cplvq"] Oct 30 13:20:07.643553 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:07.643517 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b511576b-152c-41a8-ad55-e5341f42de26" path="/var/lib/kubelet/pods/b511576b-152c-41a8-ad55-e5341f42de26/volumes" Oct 30 13:20:09.475696 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:09.475652 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-cadb6b/simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8" podUID="8382fbd3-40e1-4978-be98-37ec97883337" containerName="main" containerID="cri-o://130db8e05c2ff1143096b079d3befcea64a818f3e63809eeb74fa2cb1c35938a" gracePeriod=45 Oct 30 13:20:09.641494 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:09.641455 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-cadb6b/simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8" podUID="8382fbd3-40e1-4978-be98-37ec97883337" containerName="main" probeResult="failure" output="Get \"https://10.134.0.164:8000/ready\": dial tcp 10.134.0.164:8000: connect: connection refused" Oct 30 13:20:09.732252 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:09.732193 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-cadb6b/simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8" Oct 30 13:20:09.811819 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:09.811789 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8382fbd3-40e1-4978-be98-37ec97883337-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8382fbd3-40e1-4978-be98-37ec97883337-model-cache\") pod \"8382fbd3-40e1-4978-be98-37ec97883337\" (UID: \"8382fbd3-40e1-4978-be98-37ec97883337\") " Oct 30 13:20:09.811986 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:09.811851 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8382fbd3-40e1-4978-be98-37ec97883337-tls-certs\" (UniqueName: \"kubernetes.io/secret/8382fbd3-40e1-4978-be98-37ec97883337-tls-certs\") pod \"8382fbd3-40e1-4978-be98-37ec97883337\" (UID: \"8382fbd3-40e1-4978-be98-37ec97883337\") " Oct 30 13:20:09.811986 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:09.811904 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8382fbd3-40e1-4978-be98-37ec97883337-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8382fbd3-40e1-4978-be98-37ec97883337-tmp-dir\") pod \"8382fbd3-40e1-4978-be98-37ec97883337\" (UID: \"8382fbd3-40e1-4978-be98-37ec97883337\") " Oct 30 13:20:09.812102 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:09.811987 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8382fbd3-40e1-4978-be98-37ec97883337-home\" (UniqueName: \"kubernetes.io/empty-dir/8382fbd3-40e1-4978-be98-37ec97883337-home\") pod \"8382fbd3-40e1-4978-be98-37ec97883337\" (UID: \"8382fbd3-40e1-4978-be98-37ec97883337\") " Oct 30 13:20:09.812102 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:09.812031 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8382fbd3-40e1-4978-be98-37ec97883337-kube-api-access-2rfz4\" (UniqueName: \"kubernetes.io/projected/8382fbd3-40e1-4978-be98-37ec97883337-kube-api-access-2rfz4\") pod \"8382fbd3-40e1-4978-be98-37ec97883337\" (UID: \"8382fbd3-40e1-4978-be98-37ec97883337\") " Oct 30 13:20:09.812102 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:09.812065 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8382fbd3-40e1-4978-be98-37ec97883337-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8382fbd3-40e1-4978-be98-37ec97883337-dshm\") pod \"8382fbd3-40e1-4978-be98-37ec97883337\" (UID: \"8382fbd3-40e1-4978-be98-37ec97883337\") " Oct 30 13:20:09.812102 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:09.812068 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8382fbd3-40e1-4978-be98-37ec97883337-model-cache" pod "8382fbd3-40e1-4978-be98-37ec97883337" (UID: "8382fbd3-40e1-4978-be98-37ec97883337"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:09.812335 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:09.812313 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8382fbd3-40e1-4978-be98-37ec97883337-home" pod "8382fbd3-40e1-4978-be98-37ec97883337" (UID: "8382fbd3-40e1-4978-be98-37ec97883337"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:09.812464 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:09.812437 5922 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8382fbd3-40e1-4978-be98-37ec97883337-home\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:20:09.812464 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:09.812457 5922 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8382fbd3-40e1-4978-be98-37ec97883337-model-cache\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:20:09.812728 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:09.812536 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8382fbd3-40e1-4978-be98-37ec97883337-tmp-dir" pod "8382fbd3-40e1-4978-be98-37ec97883337" (UID: "8382fbd3-40e1-4978-be98-37ec97883337"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:09.814364 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:09.814341 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8382fbd3-40e1-4978-be98-37ec97883337-dshm" pod "8382fbd3-40e1-4978-be98-37ec97883337" (UID: "8382fbd3-40e1-4978-be98-37ec97883337"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:09.814478 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:09.814410 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8382fbd3-40e1-4978-be98-37ec97883337-kube-api-access-2rfz4" pod "8382fbd3-40e1-4978-be98-37ec97883337" (UID: "8382fbd3-40e1-4978-be98-37ec97883337"). InnerVolumeSpecName "kube-api-access-2rfz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:20:09.815011 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:09.814989 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8382fbd3-40e1-4978-be98-37ec97883337-tls-certs" pod "8382fbd3-40e1-4978-be98-37ec97883337" (UID: "8382fbd3-40e1-4978-be98-37ec97883337"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:20:09.872523 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:09.872493 5922 generic.go:356] "Generic (PLEG): container finished" podID="8382fbd3-40e1-4978-be98-37ec97883337" containerID="130db8e05c2ff1143096b079d3befcea64a818f3e63809eeb74fa2cb1c35938a" exitCode=0 Oct 30 13:20:09.872666 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:09.872548 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-cadb6b/simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8" Oct 30 13:20:09.872666 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:09.872580 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-cadb6b/simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8" event={"ID":"8382fbd3-40e1-4978-be98-37ec97883337","Type":"ContainerDied","Data":"130db8e05c2ff1143096b079d3befcea64a818f3e63809eeb74fa2cb1c35938a"} Oct 30 13:20:09.872666 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:09.872645 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-cadb6b/simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8" event={"ID":"8382fbd3-40e1-4978-be98-37ec97883337","Type":"ContainerDied","Data":"eb0e9d75cc5e70e03a7169b5a6b104fe6b0979fad88284b66d7940c486e14b8b"} Oct 30 13:20:09.872776 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:09.872668 5922 scope.go:122] "RemoveContainer" containerID="130db8e05c2ff1143096b079d3befcea64a818f3e63809eeb74fa2cb1c35938a" Oct 30 13:20:09.881051 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:09.881033 5922 scope.go:122] "RemoveContainer" containerID="130db8e05c2ff1143096b079d3befcea64a818f3e63809eeb74fa2cb1c35938a" Oct 30 13:20:09.881312 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:20:09.881292 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"130db8e05c2ff1143096b079d3befcea64a818f3e63809eeb74fa2cb1c35938a\": container with ID starting with 130db8e05c2ff1143096b079d3befcea64a818f3e63809eeb74fa2cb1c35938a not found: ID does not exist" containerID="130db8e05c2ff1143096b079d3befcea64a818f3e63809eeb74fa2cb1c35938a" Oct 30 13:20:09.881367 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:09.881319 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"130db8e05c2ff1143096b079d3befcea64a818f3e63809eeb74fa2cb1c35938a"} err="failed to get container status \"130db8e05c2ff1143096b079d3befcea64a818f3e63809eeb74fa2cb1c35938a\": rpc error: code = NotFound desc = could not find container \"130db8e05c2ff1143096b079d3befcea64a818f3e63809eeb74fa2cb1c35938a\": container with ID starting with 130db8e05c2ff1143096b079d3befcea64a818f3e63809eeb74fa2cb1c35938a not found: ID does not exist" Oct 30 13:20:09.913001 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:09.912976 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rfz4\" (UniqueName: \"kubernetes.io/projected/8382fbd3-40e1-4978-be98-37ec97883337-kube-api-access-2rfz4\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:20:09.913001 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:09.913001 5922 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8382fbd3-40e1-4978-be98-37ec97883337-dshm\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:20:09.913126 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:09.913013 5922 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8382fbd3-40e1-4978-be98-37ec97883337-tls-certs\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:20:09.913126 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:09.913022 5922 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8382fbd3-40e1-4978-be98-37ec97883337-tmp-dir\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:20:09.932387 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:09.932363 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-cadb6b/simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8"] Oct 30 13:20:09.945032 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:09.945009 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-cadb6b/simulator-single-main-cadb6b-kserve-7bd49cd76d-2j8l8"] Oct 30 13:20:10.734516 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:10.734481 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:10.851746 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:10.851707 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:11.068920 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:11.068826 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd"] Oct 30 13:20:11.069084 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:11.069042 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" podUID="3489fce9-ff92-4d68-b15a-f82be6d6ca78" containerName="istio-proxy" containerID="cri-o://b520cdf803093e2a75fb3dccbb716a792893a723345791191c0f34b99e07ba4d" gracePeriod=30 Oct 30 13:20:11.096162 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:11.096131 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:11.483166 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:11.483132 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-cadb6b-c4789c8445dz8f" podUID="047f63d3-99a4-4ca8-9d37-1ea4137e0623" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:20:11.483334 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:11.483135 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cadb6b-59cc7dwmjr" podUID="259cc7fa-4775-4c96-8e5b-bbe01c388572" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:20:11.643750 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:11.643720 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8382fbd3-40e1-4978-be98-37ec97883337" path="/var/lib/kubelet/pods/8382fbd3-40e1-4978-be98-37ec97883337/volumes" Oct 30 13:20:12.283772 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:12.283736 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hst9t"] Oct 30 13:20:12.295765 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:12.295734 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hst9t"] Oct 30 13:20:13.642968 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:13.642934 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc7b3877-b42c-4918-8957-24fd4e00ef57" path="/var/lib/kubelet/pods/cc7b3877-b42c-4918-8957-24fd4e00ef57/volumes" Oct 30 13:20:16.322059 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:16.322034 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" Oct 30 13:20:16.472735 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:16.472704 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3489fce9-ff92-4d68-b15a-f82be6d6ca78-kube-api-access-p9xq6\" (UniqueName: \"kubernetes.io/projected/3489fce9-ff92-4d68-b15a-f82be6d6ca78-kube-api-access-p9xq6\") pod \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\" (UID: \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\") " Oct 30 13:20:16.472905 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:16.472762 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3489fce9-ff92-4d68-b15a-f82be6d6ca78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3489fce9-ff92-4d68-b15a-f82be6d6ca78-istiod-ca-cert\") pod \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\" (UID: \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\") " Oct 30 13:20:16.472905 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:16.472794 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3489fce9-ff92-4d68-b15a-f82be6d6ca78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3489fce9-ff92-4d68-b15a-f82be6d6ca78-workload-socket\") pod \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\" (UID: \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\") " Oct 30 13:20:16.473040 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:16.472947 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3489fce9-ff92-4d68-b15a-f82be6d6ca78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3489fce9-ff92-4d68-b15a-f82be6d6ca78-istio-envoy\") pod \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\" (UID: \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\") " Oct 30 13:20:16.473040 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:16.472993 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3489fce9-ff92-4d68-b15a-f82be6d6ca78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3489fce9-ff92-4d68-b15a-f82be6d6ca78-istio-data\") pod \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\" (UID: \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\") " Oct 30 13:20:16.473040 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:16.473013 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3489fce9-ff92-4d68-b15a-f82be6d6ca78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3489fce9-ff92-4d68-b15a-f82be6d6ca78-credential-socket\") pod \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\" (UID: \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\") " Oct 30 13:20:16.473210 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:16.473043 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3489fce9-ff92-4d68-b15a-f82be6d6ca78-istio-token\" (UniqueName: \"kubernetes.io/projected/3489fce9-ff92-4d68-b15a-f82be6d6ca78-istio-token\") pod \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\" (UID: \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\") " Oct 30 13:20:16.473210 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:16.473064 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3489fce9-ff92-4d68-b15a-f82be6d6ca78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3489fce9-ff92-4d68-b15a-f82be6d6ca78-istio-podinfo\") pod \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\" (UID: \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\") " Oct 30 13:20:16.473210 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:16.473100 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3489fce9-ff92-4d68-b15a-f82be6d6ca78-workload-socket" pod "3489fce9-ff92-4d68-b15a-f82be6d6ca78" (UID: "3489fce9-ff92-4d68-b15a-f82be6d6ca78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:16.473210 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:16.473117 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3489fce9-ff92-4d68-b15a-f82be6d6ca78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3489fce9-ff92-4d68-b15a-f82be6d6ca78-workload-certs\") pod \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\" (UID: \"3489fce9-ff92-4d68-b15a-f82be6d6ca78\") " Oct 30 13:20:16.473210 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:16.473188 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3489fce9-ff92-4d68-b15a-f82be6d6ca78-istiod-ca-cert" pod "3489fce9-ff92-4d68-b15a-f82be6d6ca78" (UID: "3489fce9-ff92-4d68-b15a-f82be6d6ca78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:20:16.473490 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:16.473330 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3489fce9-ff92-4d68-b15a-f82be6d6ca78-credential-socket" pod "3489fce9-ff92-4d68-b15a-f82be6d6ca78" (UID: "3489fce9-ff92-4d68-b15a-f82be6d6ca78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:16.473551 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:16.473485 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3489fce9-ff92-4d68-b15a-f82be6d6ca78-istio-data" pod "3489fce9-ff92-4d68-b15a-f82be6d6ca78" (UID: "3489fce9-ff92-4d68-b15a-f82be6d6ca78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:16.473551 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:16.473518 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3489fce9-ff92-4d68-b15a-f82be6d6ca78-workload-certs" pod "3489fce9-ff92-4d68-b15a-f82be6d6ca78" (UID: "3489fce9-ff92-4d68-b15a-f82be6d6ca78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:16.473551 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:16.473542 5922 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3489fce9-ff92-4d68-b15a-f82be6d6ca78-credential-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:20:16.473755 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:16.473561 5922 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3489fce9-ff92-4d68-b15a-f82be6d6ca78-istiod-ca-cert\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:20:16.473755 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:16.473576 5922 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3489fce9-ff92-4d68-b15a-f82be6d6ca78-workload-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:20:16.475022 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:16.474988 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3489fce9-ff92-4d68-b15a-f82be6d6ca78-kube-api-access-p9xq6" pod "3489fce9-ff92-4d68-b15a-f82be6d6ca78" (UID: "3489fce9-ff92-4d68-b15a-f82be6d6ca78"). InnerVolumeSpecName "kube-api-access-p9xq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:20:16.475135 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:16.475041 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3489fce9-ff92-4d68-b15a-f82be6d6ca78-istio-token" pod "3489fce9-ff92-4d68-b15a-f82be6d6ca78" (UID: "3489fce9-ff92-4d68-b15a-f82be6d6ca78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:20:16.475135 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:16.475068 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3489fce9-ff92-4d68-b15a-f82be6d6ca78-istio-envoy" pod "3489fce9-ff92-4d68-b15a-f82be6d6ca78" (UID: "3489fce9-ff92-4d68-b15a-f82be6d6ca78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:16.475135 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:16.475122 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3489fce9-ff92-4d68-b15a-f82be6d6ca78-istio-podinfo" pod "3489fce9-ff92-4d68-b15a-f82be6d6ca78" (UID: "3489fce9-ff92-4d68-b15a-f82be6d6ca78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:20:16.574394 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:16.574371 5922 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3489fce9-ff92-4d68-b15a-f82be6d6ca78-istio-data\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:20:16.574394 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:16.574393 5922 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3489fce9-ff92-4d68-b15a-f82be6d6ca78-istio-token\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:20:16.574527 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:16.574401 5922 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3489fce9-ff92-4d68-b15a-f82be6d6ca78-istio-podinfo\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:20:16.574527 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:16.574410 5922 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3489fce9-ff92-4d68-b15a-f82be6d6ca78-workload-certs\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:20:16.574527 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:16.574418 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9xq6\" (UniqueName: \"kubernetes.io/projected/3489fce9-ff92-4d68-b15a-f82be6d6ca78-kube-api-access-p9xq6\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:20:16.574527 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:16.574427 5922 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3489fce9-ff92-4d68-b15a-f82be6d6ca78-istio-envoy\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:20:16.909713 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:16.909644 5922 generic.go:356] "Generic (PLEG): container finished" podID="3489fce9-ff92-4d68-b15a-f82be6d6ca78" containerID="b520cdf803093e2a75fb3dccbb716a792893a723345791191c0f34b99e07ba4d" exitCode=0 Oct 30 13:20:16.909713 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:16.909690 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" Oct 30 13:20:16.909866 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:16.909726 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" event={"ID":"3489fce9-ff92-4d68-b15a-f82be6d6ca78","Type":"ContainerDied","Data":"b520cdf803093e2a75fb3dccbb716a792893a723345791191c0f34b99e07ba4d"} Oct 30 13:20:16.909866 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:16.909782 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd" event={"ID":"3489fce9-ff92-4d68-b15a-f82be6d6ca78","Type":"ContainerDied","Data":"afdaabaa00c5b79795fb6faef452aa55d49b35e376da1d6a155efed70bcbe26a"} Oct 30 13:20:16.909866 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:16.909806 5922 scope.go:122] "RemoveContainer" containerID="b520cdf803093e2a75fb3dccbb716a792893a723345791191c0f34b99e07ba4d" Oct 30 13:20:16.919005 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:16.918985 5922 scope.go:122] "RemoveContainer" containerID="0441bc0510c58ff10f4a23dc4b36a96d1d4cec993c47dc7e6b26d54fe78ca6d2" Oct 30 13:20:16.926675 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:16.926660 5922 scope.go:122] "RemoveContainer" containerID="b520cdf803093e2a75fb3dccbb716a792893a723345791191c0f34b99e07ba4d" Oct 30 13:20:16.926941 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:20:16.926918 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b520cdf803093e2a75fb3dccbb716a792893a723345791191c0f34b99e07ba4d\": container with ID starting with b520cdf803093e2a75fb3dccbb716a792893a723345791191c0f34b99e07ba4d not found: ID does not exist" containerID="b520cdf803093e2a75fb3dccbb716a792893a723345791191c0f34b99e07ba4d" Oct 30 13:20:16.926993 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:16.926948 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b520cdf803093e2a75fb3dccbb716a792893a723345791191c0f34b99e07ba4d"} err="failed to get container status \"b520cdf803093e2a75fb3dccbb716a792893a723345791191c0f34b99e07ba4d\": rpc error: code = NotFound desc = could not find container \"b520cdf803093e2a75fb3dccbb716a792893a723345791191c0f34b99e07ba4d\": container with ID starting with b520cdf803093e2a75fb3dccbb716a792893a723345791191c0f34b99e07ba4d not found: ID does not exist" Oct 30 13:20:16.926993 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:16.926965 5922 scope.go:122] "RemoveContainer" containerID="0441bc0510c58ff10f4a23dc4b36a96d1d4cec993c47dc7e6b26d54fe78ca6d2" Oct 30 13:20:16.927278 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:20:16.927259 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0441bc0510c58ff10f4a23dc4b36a96d1d4cec993c47dc7e6b26d54fe78ca6d2\": container with ID starting with 0441bc0510c58ff10f4a23dc4b36a96d1d4cec993c47dc7e6b26d54fe78ca6d2 not found: ID does not exist" containerID="0441bc0510c58ff10f4a23dc4b36a96d1d4cec993c47dc7e6b26d54fe78ca6d2" Oct 30 13:20:16.927343 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:16.927284 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0441bc0510c58ff10f4a23dc4b36a96d1d4cec993c47dc7e6b26d54fe78ca6d2"} err="failed to get container status \"0441bc0510c58ff10f4a23dc4b36a96d1d4cec993c47dc7e6b26d54fe78ca6d2\": rpc error: code = NotFound desc = could not find container \"0441bc0510c58ff10f4a23dc4b36a96d1d4cec993c47dc7e6b26d54fe78ca6d2\": container with ID starting with 0441bc0510c58ff10f4a23dc4b36a96d1d4cec993c47dc7e6b26d54fe78ca6d2 not found: ID does not exist" Oct 30 13:20:16.975731 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:16.975698 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd"] Oct 30 13:20:16.988110 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:16.988089 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-cadb6b-openshift-default-6bf77bcc-rdfjd"] Oct 30 13:20:17.643079 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:17.643045 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3489fce9-ff92-4d68-b15a-f82be6d6ca78" path="/var/lib/kubelet/pods/3489fce9-ff92-4d68-b15a-f82be6d6ca78/volumes" Oct 30 13:20:21.483457 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:21.483423 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-cadb6b-c4789c8445dz8f" podUID="047f63d3-99a4-4ca8-9d37-1ea4137e0623" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:20:21.483457 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:21.483451 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cadb6b-59cc7dwmjr" podUID="259cc7fa-4775-4c96-8e5b-bbe01c388572" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:20:23.552676 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:20:23.552633 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:20:24.336526 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:24.336489 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:20:25.656662 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:25.656575 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-cadb6b-c4789c8445dz8f" Oct 30 13:20:25.738329 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:25.738295 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/047f63d3-99a4-4ca8-9d37-1ea4137e0623-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/047f63d3-99a4-4ca8-9d37-1ea4137e0623-plugins-config-volume\") pod \"047f63d3-99a4-4ca8-9d37-1ea4137e0623\" (UID: \"047f63d3-99a4-4ca8-9d37-1ea4137e0623\") " Oct 30 13:20:25.738706 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:25.738680 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/047f63d3-99a4-4ca8-9d37-1ea4137e0623-kube-api-access-tmt5r\" (UniqueName: \"kubernetes.io/projected/047f63d3-99a4-4ca8-9d37-1ea4137e0623-kube-api-access-tmt5r\") pod \"047f63d3-99a4-4ca8-9d37-1ea4137e0623\" (UID: \"047f63d3-99a4-4ca8-9d37-1ea4137e0623\") " Oct 30 13:20:25.742835 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:25.741239 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/047f63d3-99a4-4ca8-9d37-1ea4137e0623-plugins-config-volume" pod "047f63d3-99a4-4ca8-9d37-1ea4137e0623" (UID: "047f63d3-99a4-4ca8-9d37-1ea4137e0623"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:20:25.742835 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:25.742743 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/047f63d3-99a4-4ca8-9d37-1ea4137e0623-kube-api-access-tmt5r" pod "047f63d3-99a4-4ca8-9d37-1ea4137e0623" (UID: "047f63d3-99a4-4ca8-9d37-1ea4137e0623"). InnerVolumeSpecName "kube-api-access-tmt5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:20:25.840635 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:25.839255 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cadb6b-59cc7dwmjr" Oct 30 13:20:25.840635 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:25.840068 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmt5r\" (UniqueName: \"kubernetes.io/projected/047f63d3-99a4-4ca8-9d37-1ea4137e0623-kube-api-access-tmt5r\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:20:25.840635 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:25.840095 5922 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/047f63d3-99a4-4ca8-9d37-1ea4137e0623-plugins-config-volume\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:20:25.948140 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:25.948102 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/259cc7fa-4775-4c96-8e5b-bbe01c388572-kube-api-access-vb5xh\" (UniqueName: \"kubernetes.io/projected/259cc7fa-4775-4c96-8e5b-bbe01c388572-kube-api-access-vb5xh\") pod \"259cc7fa-4775-4c96-8e5b-bbe01c388572\" (UID: \"259cc7fa-4775-4c96-8e5b-bbe01c388572\") " Oct 30 13:20:25.948340 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:25.948166 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/259cc7fa-4775-4c96-8e5b-bbe01c388572-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/259cc7fa-4775-4c96-8e5b-bbe01c388572-plugins-config-volume\") pod \"259cc7fa-4775-4c96-8e5b-bbe01c388572\" (UID: \"259cc7fa-4775-4c96-8e5b-bbe01c388572\") " Oct 30 13:20:25.948922 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:25.948870 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/259cc7fa-4775-4c96-8e5b-bbe01c388572-plugins-config-volume" pod "259cc7fa-4775-4c96-8e5b-bbe01c388572" (UID: "259cc7fa-4775-4c96-8e5b-bbe01c388572"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:20:25.951811 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:25.951786 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/259cc7fa-4775-4c96-8e5b-bbe01c388572-kube-api-access-vb5xh" pod "259cc7fa-4775-4c96-8e5b-bbe01c388572" (UID: "259cc7fa-4775-4c96-8e5b-bbe01c388572"). InnerVolumeSpecName "kube-api-access-vb5xh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:20:25.997631 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:25.961713 5922 generic.go:356] "Generic (PLEG): container finished" podID="259cc7fa-4775-4c96-8e5b-bbe01c388572" containerID="75bd66eb23f3ad13ef8820533ca57372290908303a68d177dbde7e090c2601a8" exitCode=0 Oct 30 13:20:25.997631 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:25.961980 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cadb6b-59cc7dwmjr" Oct 30 13:20:26.033632 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:26.006844 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cadb6b-59cc7dwmjr" event={"ID":"259cc7fa-4775-4c96-8e5b-bbe01c388572","Type":"ContainerDied","Data":"75bd66eb23f3ad13ef8820533ca57372290908303a68d177dbde7e090c2601a8"} Oct 30 13:20:26.033632 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:26.006914 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cadb6b-59cc7dwmjr" event={"ID":"259cc7fa-4775-4c96-8e5b-bbe01c388572","Type":"ContainerDied","Data":"b60b2be66d62b5e16b84e4cb23c153af91d5c3eeb6e2ca555f75602cb8c812ed"} Oct 30 13:20:26.033632 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:26.006951 5922 scope.go:122] "RemoveContainer" containerID="75bd66eb23f3ad13ef8820533ca57372290908303a68d177dbde7e090c2601a8" Oct 30 13:20:26.043894 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:26.041062 5922 generic.go:356] "Generic (PLEG): container finished" podID="047f63d3-99a4-4ca8-9d37-1ea4137e0623" containerID="34e39fa5369876f449fbccff8e422ef6a5dafaea9b268771bfbc908ab2417715" exitCode=0 Oct 30 13:20:26.043894 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:26.041102 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-cadb6b-c4789c8445dz8f" event={"ID":"047f63d3-99a4-4ca8-9d37-1ea4137e0623","Type":"ContainerDied","Data":"34e39fa5369876f449fbccff8e422ef6a5dafaea9b268771bfbc908ab2417715"} Oct 30 13:20:26.043894 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:26.041145 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-cadb6b-c4789c8445dz8f" event={"ID":"047f63d3-99a4-4ca8-9d37-1ea4137e0623","Type":"ContainerDied","Data":"5800f956bedda72cb0a63b72ed8cc0d66a002ffa24412ecb49e5e9fe67afe54b"} Oct 30 13:20:26.043894 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:26.041221 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-cadb6b-c4789c8445dz8f" Oct 30 13:20:26.056422 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:26.050127 5922 scope.go:122] "RemoveContainer" containerID="75bd66eb23f3ad13ef8820533ca57372290908303a68d177dbde7e090c2601a8" Oct 30 13:20:26.056422 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:26.050362 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb5xh\" (UniqueName: \"kubernetes.io/projected/259cc7fa-4775-4c96-8e5b-bbe01c388572-kube-api-access-vb5xh\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:20:26.056422 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:26.050393 5922 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/259cc7fa-4775-4c96-8e5b-bbe01c388572-plugins-config-volume\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:20:26.057167 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:20:26.057136 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75bd66eb23f3ad13ef8820533ca57372290908303a68d177dbde7e090c2601a8\": container with ID starting with 75bd66eb23f3ad13ef8820533ca57372290908303a68d177dbde7e090c2601a8 not found: ID does not exist" containerID="75bd66eb23f3ad13ef8820533ca57372290908303a68d177dbde7e090c2601a8" Oct 30 13:20:26.057252 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:26.057180 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75bd66eb23f3ad13ef8820533ca57372290908303a68d177dbde7e090c2601a8"} err="failed to get container status \"75bd66eb23f3ad13ef8820533ca57372290908303a68d177dbde7e090c2601a8\": rpc error: code = NotFound desc = could not find container \"75bd66eb23f3ad13ef8820533ca57372290908303a68d177dbde7e090c2601a8\": container with ID starting with 75bd66eb23f3ad13ef8820533ca57372290908303a68d177dbde7e090c2601a8 not found: ID does not exist" Oct 30 13:20:26.057252 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:26.057212 5922 scope.go:122] "RemoveContainer" containerID="34e39fa5369876f449fbccff8e422ef6a5dafaea9b268771bfbc908ab2417715" Oct 30 13:20:26.128635 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:26.104886 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-cadb6b-c4789c8445dz8f"] Oct 30 13:20:26.128932 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:26.127747 5922 scope.go:122] "RemoveContainer" containerID="34e39fa5369876f449fbccff8e422ef6a5dafaea9b268771bfbc908ab2417715" Oct 30 13:20:26.134181 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:26.129983 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-cadb6b-c4789c8445dz8f"] Oct 30 13:20:26.134307 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:20:26.134177 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34e39fa5369876f449fbccff8e422ef6a5dafaea9b268771bfbc908ab2417715\": container with ID starting with 34e39fa5369876f449fbccff8e422ef6a5dafaea9b268771bfbc908ab2417715 not found: ID does not exist" containerID="34e39fa5369876f449fbccff8e422ef6a5dafaea9b268771bfbc908ab2417715" Oct 30 13:20:26.134307 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:26.134213 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34e39fa5369876f449fbccff8e422ef6a5dafaea9b268771bfbc908ab2417715"} err="failed to get container status \"34e39fa5369876f449fbccff8e422ef6a5dafaea9b268771bfbc908ab2417715\": rpc error: code = NotFound desc = could not find container \"34e39fa5369876f449fbccff8e422ef6a5dafaea9b268771bfbc908ab2417715\": container with ID starting with 34e39fa5369876f449fbccff8e422ef6a5dafaea9b268771bfbc908ab2417715 not found: ID does not exist" Oct 30 13:20:26.192632 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:26.191656 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-cadb6b-59cc7dwmjr"] Oct 30 13:20:26.193476 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:26.193454 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-cadb6b-59cc7dwmjr"] Oct 30 13:20:27.643229 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:27.643196 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="047f63d3-99a4-4ca8-9d37-1ea4137e0623" path="/var/lib/kubelet/pods/047f63d3-99a4-4ca8-9d37-1ea4137e0623/volumes" Oct 30 13:20:27.643650 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:27.643582 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="259cc7fa-4775-4c96-8e5b-bbe01c388572" path="/var/lib/kubelet/pods/259cc7fa-4775-4c96-8e5b-bbe01c388572/volumes" Oct 30 13:20:29.308535 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:29.308503 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:20:32.186069 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:32.186027 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:32.960686 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:32.960653 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:34.446718 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:34.446673 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:39.053403 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:20:39.053363 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:53.551484 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:20:53.551446 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:21:11.035361 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:21:11.035323 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:13.327729 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:21:13.327693 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:13.456318 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:21:13.456290 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:16.988271 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:21:16.988235 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:17.123791 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:21:17.123751 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:23.551860 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:21:23.551819 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:21:25.374589 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:21:25.374558 5922 scope.go:122] "RemoveContainer" containerID="3918853fc00b71be007238a190583174d846d1bd92a72d8e72fdec3024fbf8d3" Oct 30 13:21:34.577084 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:21:34.577045 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:52.493094 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:21:52.493050 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:53.551856 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:21:53.551821 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:21:53.973000 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:21:53.972967 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:57.932354 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:21:57.932310 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:22:13.218197 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:13.218167 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:22:16.874072 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:16.874026 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wcd82"] Oct 30 13:22:16.874730 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:16.874651 5922 cpu_manager.go:414] "containerMap: removing container" podUID="8382fbd3-40e1-4978-be98-37ec97883337" containerName="main" Oct 30 13:22:16.874730 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:16.874673 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8382fbd3-40e1-4978-be98-37ec97883337" containerName="main" Oct 30 13:22:16.874730 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:16.874702 5922 cpu_manager.go:414] "containerMap: removing container" podUID="3489fce9-ff92-4d68-b15a-f82be6d6ca78" containerName="istio-proxy" Oct 30 13:22:16.874730 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:16.874711 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3489fce9-ff92-4d68-b15a-f82be6d6ca78" containerName="istio-proxy" Oct 30 13:22:16.874986 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:16.874752 5922 cpu_manager.go:414] "containerMap: removing container" podUID="047f63d3-99a4-4ca8-9d37-1ea4137e0623" containerName="payload-processing" Oct 30 13:22:16.874986 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:16.874761 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="047f63d3-99a4-4ca8-9d37-1ea4137e0623" containerName="payload-processing" Oct 30 13:22:16.874986 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:16.874775 5922 cpu_manager.go:414] "containerMap: removing container" podUID="3489fce9-ff92-4d68-b15a-f82be6d6ca78" containerName="istio-proxy" Oct 30 13:22:16.874986 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:16.874782 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3489fce9-ff92-4d68-b15a-f82be6d6ca78" containerName="istio-proxy" Oct 30 13:22:16.874986 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:16.874797 5922 cpu_manager.go:414] "containerMap: removing container" podUID="259cc7fa-4775-4c96-8e5b-bbe01c388572" containerName="payload-pre-processing" Oct 30 13:22:16.874986 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:16.874805 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="259cc7fa-4775-4c96-8e5b-bbe01c388572" containerName="payload-pre-processing" Oct 30 13:22:16.874986 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:16.874823 5922 cpu_manager.go:414] "containerMap: removing container" podUID="7172e198-7951-48a7-8429-06f4f11bf722" containerName="revoke-keys" Oct 30 13:22:16.874986 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:16.874832 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7172e198-7951-48a7-8429-06f4f11bf722" containerName="revoke-keys" Oct 30 13:22:16.874986 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:16.874951 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="047f63d3-99a4-4ca8-9d37-1ea4137e0623" containerName="payload-processing" Oct 30 13:22:16.874986 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:16.874967 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="3489fce9-ff92-4d68-b15a-f82be6d6ca78" containerName="istio-proxy" Oct 30 13:22:16.874986 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:16.874977 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="259cc7fa-4775-4c96-8e5b-bbe01c388572" containerName="payload-pre-processing" Oct 30 13:22:16.875575 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:16.874986 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="7172e198-7951-48a7-8429-06f4f11bf722" containerName="revoke-keys" Oct 30 13:22:16.875575 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:16.875049 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="8382fbd3-40e1-4978-be98-37ec97883337" containerName="main" Oct 30 13:22:16.875575 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:16.875062 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="3489fce9-ff92-4d68-b15a-f82be6d6ca78" containerName="istio-proxy" Oct 30 13:22:16.894711 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:16.893850 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wcd82" Oct 30 13:22:16.934634 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:16.912126 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wcd82"] Oct 30 13:22:16.943265 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:16.940379 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:22:16.943265 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:16.940455 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:22:16.943265 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:16.940551 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:22:16.943265 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:16.940635 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:22:16.986633 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:16.962143 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aece694-bc22-4290-a02d-8dd36aaf4b0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-wcd82\" (UID: \"0aece694-bc22-4290-a02d-8dd36aaf4b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wcd82" Oct 30 13:22:16.986633 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:16.962242 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjvsk\" (UniqueName: \"kubernetes.io/projected/0aece694-bc22-4290-a02d-8dd36aaf4b0e-kube-api-access-sjvsk\") pod \"maas-api-revoke-sub-d2bc93bc-wcd82\" (UID: \"0aece694-bc22-4290-a02d-8dd36aaf4b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wcd82" Oct 30 13:22:16.986633 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:16.962307 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aece694-bc22-4290-a02d-8dd36aaf4b0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-wcd82\" (UID: \"0aece694-bc22-4290-a02d-8dd36aaf4b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wcd82" Oct 30 13:22:17.070656 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:17.070377 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aece694-bc22-4290-a02d-8dd36aaf4b0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-wcd82\" (UID: \"0aece694-bc22-4290-a02d-8dd36aaf4b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wcd82" Oct 30 13:22:17.070656 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:17.070479 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjvsk\" (UniqueName: \"kubernetes.io/projected/0aece694-bc22-4290-a02d-8dd36aaf4b0e-kube-api-access-sjvsk\") pod \"maas-api-revoke-sub-d2bc93bc-wcd82\" (UID: \"0aece694-bc22-4290-a02d-8dd36aaf4b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wcd82" Oct 30 13:22:17.070656 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:17.070542 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aece694-bc22-4290-a02d-8dd36aaf4b0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-wcd82\" (UID: \"0aece694-bc22-4290-a02d-8dd36aaf4b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wcd82" Oct 30 13:22:17.079638 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:17.071677 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aece694-bc22-4290-a02d-8dd36aaf4b0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-wcd82\" (UID: \"0aece694-bc22-4290-a02d-8dd36aaf4b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wcd82" Oct 30 13:22:17.121486 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:17.121458 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjvsk\" (UniqueName: \"kubernetes.io/projected/0aece694-bc22-4290-a02d-8dd36aaf4b0e-kube-api-access-sjvsk\") pod \"maas-api-revoke-sub-d2bc93bc-wcd82\" (UID: \"0aece694-bc22-4290-a02d-8dd36aaf4b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wcd82" Oct 30 13:22:17.133684 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:17.130010 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aece694-bc22-4290-a02d-8dd36aaf4b0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-wcd82\" (UID: \"0aece694-bc22-4290-a02d-8dd36aaf4b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wcd82" Oct 30 13:22:17.272813 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:17.272763 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wcd82" Oct 30 13:22:17.913080 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:17.909086 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wcd82"] Oct 30 13:22:18.569680 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:18.562020 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wcd82" event={"ID":"0aece694-bc22-4290-a02d-8dd36aaf4b0e","Type":"ContainerStarted","Data":"614b7bc07bcdc7704aefd89d6f4a812593afb57877404b91c94b61541d3d32e3"} Oct 30 13:22:18.569680 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:18.562088 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wcd82" event={"ID":"0aece694-bc22-4290-a02d-8dd36aaf4b0e","Type":"ContainerStarted","Data":"bb17d84b0d71972a3d166b6ea6cd30f4684cdd905b954eb456f314bd675a62d9"} Oct 30 13:22:18.605154 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:18.605104 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wcd82" podStartSLOduration=2.605087925 podStartE2EDuration="2.605087925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:22:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:22:18.60412722 +0000 UTC m=+1795.073542964" watchObservedRunningTime="2026-10-30 13:22:18.605087925 +0000 UTC m=+1795.074503672" Oct 30 13:22:19.567959 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:19.567922 5922 generic.go:356] "Generic (PLEG): container finished" podID="0aece694-bc22-4290-a02d-8dd36aaf4b0e" containerID="614b7bc07bcdc7704aefd89d6f4a812593afb57877404b91c94b61541d3d32e3" exitCode=0 Oct 30 13:22:19.568344 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:19.567960 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wcd82" event={"ID":"0aece694-bc22-4290-a02d-8dd36aaf4b0e","Type":"ContainerDied","Data":"614b7bc07bcdc7704aefd89d6f4a812593afb57877404b91c94b61541d3d32e3"} Oct 30 13:22:19.644064 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:19.644030 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:20.709895 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:20.709871 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wcd82" Oct 30 13:22:20.867668 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:20.867577 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aece694-bc22-4290-a02d-8dd36aaf4b0e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aece694-bc22-4290-a02d-8dd36aaf4b0e-maas-api-openshift-ca\") pod \"0aece694-bc22-4290-a02d-8dd36aaf4b0e\" (UID: \"0aece694-bc22-4290-a02d-8dd36aaf4b0e\") " Oct 30 13:22:20.867816 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:20.867698 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aece694-bc22-4290-a02d-8dd36aaf4b0e-kube-api-access-sjvsk\" (UniqueName: \"kubernetes.io/projected/0aece694-bc22-4290-a02d-8dd36aaf4b0e-kube-api-access-sjvsk\") pod \"0aece694-bc22-4290-a02d-8dd36aaf4b0e\" (UID: \"0aece694-bc22-4290-a02d-8dd36aaf4b0e\") " Oct 30 13:22:20.867816 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:20.867744 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0aece694-bc22-4290-a02d-8dd36aaf4b0e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aece694-bc22-4290-a02d-8dd36aaf4b0e-maas-api-xks-ca\") pod \"0aece694-bc22-4290-a02d-8dd36aaf4b0e\" (UID: \"0aece694-bc22-4290-a02d-8dd36aaf4b0e\") " Oct 30 13:22:20.867940 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:20.867905 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aece694-bc22-4290-a02d-8dd36aaf4b0e-maas-api-openshift-ca" pod "0aece694-bc22-4290-a02d-8dd36aaf4b0e" (UID: "0aece694-bc22-4290-a02d-8dd36aaf4b0e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:22:20.868004 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:20.867988 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aece694-bc22-4290-a02d-8dd36aaf4b0e-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:22:20.869830 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:20.869805 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aece694-bc22-4290-a02d-8dd36aaf4b0e-kube-api-access-sjvsk" pod "0aece694-bc22-4290-a02d-8dd36aaf4b0e" (UID: "0aece694-bc22-4290-a02d-8dd36aaf4b0e"). InnerVolumeSpecName "kube-api-access-sjvsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:22:20.869902 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:20.869846 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0aece694-bc22-4290-a02d-8dd36aaf4b0e-maas-api-xks-ca" pod "0aece694-bc22-4290-a02d-8dd36aaf4b0e" (UID: "0aece694-bc22-4290-a02d-8dd36aaf4b0e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:22:20.969349 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:20.969319 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjvsk\" (UniqueName: \"kubernetes.io/projected/0aece694-bc22-4290-a02d-8dd36aaf4b0e-kube-api-access-sjvsk\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:22:20.969349 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:20.969347 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aece694-bc22-4290-a02d-8dd36aaf4b0e-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:22:21.577542 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:21.577509 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wcd82" event={"ID":"0aece694-bc22-4290-a02d-8dd36aaf4b0e","Type":"ContainerDied","Data":"bb17d84b0d71972a3d166b6ea6cd30f4684cdd905b954eb456f314bd675a62d9"} Oct 30 13:22:21.577739 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:21.577580 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb17d84b0d71972a3d166b6ea6cd30f4684cdd905b954eb456f314bd675a62d9" Oct 30 13:22:21.577739 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:21.577519 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wcd82" Oct 30 13:22:23.556455 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:22:23.556420 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:22:28.623332 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:28.623296 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:28.980953 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:28.980909 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw"] Oct 30 13:22:28.981381 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:28.981363 5922 cpu_manager.go:414] "containerMap: removing container" podUID="0aece694-bc22-4290-a02d-8dd36aaf4b0e" containerName="revoke-keys" Oct 30 13:22:28.981477 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:28.981383 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aece694-bc22-4290-a02d-8dd36aaf4b0e" containerName="revoke-keys" Oct 30 13:22:28.981548 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:28.981534 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aece694-bc22-4290-a02d-8dd36aaf4b0e" containerName="revoke-keys" Oct 30 13:22:28.984794 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:28.984774 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" Oct 30 13:22:29.012639 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:29.012600 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw"] Oct 30 13:22:29.019296 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:29.019273 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-2cc21d-openshift-default-dockercfg-4g4x9\"" Oct 30 13:22:29.037330 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:29.037306 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx4q6\" (UniqueName: \"kubernetes.io/projected/ff585ad9-f06c-453b-b23f-f0519a479d9a-kube-api-access-xx4q6\") pod \"e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw\" (UID: \"ff585ad9-f06c-453b-b23f-f0519a479d9a\") " pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" Oct 30 13:22:29.037429 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:29.037406 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff585ad9-f06c-453b-b23f-f0519a479d9a-credential-socket\") pod \"e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw\" (UID: \"ff585ad9-f06c-453b-b23f-f0519a479d9a\") " pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" Oct 30 13:22:29.037496 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:29.037462 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff585ad9-f06c-453b-b23f-f0519a479d9a-istio-envoy\") pod \"e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw\" (UID: \"ff585ad9-f06c-453b-b23f-f0519a479d9a\") " pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" Oct 30 13:22:29.037564 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:29.037546 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff585ad9-f06c-453b-b23f-f0519a479d9a-workload-socket\") pod \"e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw\" (UID: \"ff585ad9-f06c-453b-b23f-f0519a479d9a\") " pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" Oct 30 13:22:29.037606 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:29.037594 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff585ad9-f06c-453b-b23f-f0519a479d9a-istio-token\") pod \"e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw\" (UID: \"ff585ad9-f06c-453b-b23f-f0519a479d9a\") " pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" Oct 30 13:22:29.037680 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:29.037632 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff585ad9-f06c-453b-b23f-f0519a479d9a-istiod-ca-cert\") pod \"e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw\" (UID: \"ff585ad9-f06c-453b-b23f-f0519a479d9a\") " pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" Oct 30 13:22:29.037811 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:29.037784 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff585ad9-f06c-453b-b23f-f0519a479d9a-workload-certs\") pod \"e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw\" (UID: \"ff585ad9-f06c-453b-b23f-f0519a479d9a\") " pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" Oct 30 13:22:29.037928 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:29.037843 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff585ad9-f06c-453b-b23f-f0519a479d9a-istio-podinfo\") pod \"e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw\" (UID: \"ff585ad9-f06c-453b-b23f-f0519a479d9a\") " pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" Oct 30 13:22:29.038006 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:29.037916 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff585ad9-f06c-453b-b23f-f0519a479d9a-istio-data\") pod \"e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw\" (UID: \"ff585ad9-f06c-453b-b23f-f0519a479d9a\") " pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" Oct 30 13:22:29.050606 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:29.050569 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:29.139350 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:29.139285 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff585ad9-f06c-453b-b23f-f0519a479d9a-workload-certs\") pod \"e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw\" (UID: \"ff585ad9-f06c-453b-b23f-f0519a479d9a\") " pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" Oct 30 13:22:29.139350 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:29.139326 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff585ad9-f06c-453b-b23f-f0519a479d9a-istio-podinfo\") pod \"e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw\" (UID: \"ff585ad9-f06c-453b-b23f-f0519a479d9a\") " pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" Oct 30 13:22:29.139350 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:29.139344 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff585ad9-f06c-453b-b23f-f0519a479d9a-istio-data\") pod \"e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw\" (UID: \"ff585ad9-f06c-453b-b23f-f0519a479d9a\") " pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" Oct 30 13:22:29.139350 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:29.139385 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx4q6\" (UniqueName: \"kubernetes.io/projected/ff585ad9-f06c-453b-b23f-f0519a479d9a-kube-api-access-xx4q6\") pod \"e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw\" (UID: \"ff585ad9-f06c-453b-b23f-f0519a479d9a\") " pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" Oct 30 13:22:29.139815 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:29.139435 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff585ad9-f06c-453b-b23f-f0519a479d9a-credential-socket\") pod \"e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw\" (UID: \"ff585ad9-f06c-453b-b23f-f0519a479d9a\") " pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" Oct 30 13:22:29.139815 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:29.139749 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff585ad9-f06c-453b-b23f-f0519a479d9a-istio-envoy\") pod \"e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw\" (UID: \"ff585ad9-f06c-453b-b23f-f0519a479d9a\") " pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" Oct 30 13:22:29.139815 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:29.139774 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff585ad9-f06c-453b-b23f-f0519a479d9a-workload-certs\") pod \"e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw\" (UID: \"ff585ad9-f06c-453b-b23f-f0519a479d9a\") " pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" Oct 30 13:22:29.139815 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:29.139806 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff585ad9-f06c-453b-b23f-f0519a479d9a-credential-socket\") pod \"e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw\" (UID: \"ff585ad9-f06c-453b-b23f-f0519a479d9a\") " pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" Oct 30 13:22:29.140018 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:29.139809 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff585ad9-f06c-453b-b23f-f0519a479d9a-workload-socket\") pod \"e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw\" (UID: \"ff585ad9-f06c-453b-b23f-f0519a479d9a\") " pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" Oct 30 13:22:29.140018 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:29.139875 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff585ad9-f06c-453b-b23f-f0519a479d9a-istio-data\") pod \"e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw\" (UID: \"ff585ad9-f06c-453b-b23f-f0519a479d9a\") " pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" Oct 30 13:22:29.140018 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:29.139884 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff585ad9-f06c-453b-b23f-f0519a479d9a-istio-token\") pod \"e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw\" (UID: \"ff585ad9-f06c-453b-b23f-f0519a479d9a\") " pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" Oct 30 13:22:29.140018 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:29.139939 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff585ad9-f06c-453b-b23f-f0519a479d9a-istiod-ca-cert\") pod \"e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw\" (UID: \"ff585ad9-f06c-453b-b23f-f0519a479d9a\") " pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" Oct 30 13:22:29.140180 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:29.140032 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff585ad9-f06c-453b-b23f-f0519a479d9a-workload-socket\") pod \"e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw\" (UID: \"ff585ad9-f06c-453b-b23f-f0519a479d9a\") " pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" Oct 30 13:22:29.142406 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:29.142379 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff585ad9-f06c-453b-b23f-f0519a479d9a-istio-envoy\") pod \"e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw\" (UID: \"ff585ad9-f06c-453b-b23f-f0519a479d9a\") " pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" Oct 30 13:22:29.142893 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:29.142850 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff585ad9-f06c-453b-b23f-f0519a479d9a-istio-podinfo\") pod \"e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw\" (UID: \"ff585ad9-f06c-453b-b23f-f0519a479d9a\") " pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" Oct 30 13:22:29.142982 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:29.142917 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff585ad9-f06c-453b-b23f-f0519a479d9a-istiod-ca-cert\") pod \"e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw\" (UID: \"ff585ad9-f06c-453b-b23f-f0519a479d9a\") " pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" Oct 30 13:22:29.164510 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:29.164468 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff585ad9-f06c-453b-b23f-f0519a479d9a-istio-token\") pod \"e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw\" (UID: \"ff585ad9-f06c-453b-b23f-f0519a479d9a\") " pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" Oct 30 13:22:29.164749 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:29.164726 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx4q6\" (UniqueName: \"kubernetes.io/projected/ff585ad9-f06c-453b-b23f-f0519a479d9a-kube-api-access-xx4q6\") pod \"e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw\" (UID: \"ff585ad9-f06c-453b-b23f-f0519a479d9a\") " pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" Oct 30 13:22:29.169104 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:29.169065 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:29.297807 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:29.297734 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" Oct 30 13:22:29.611675 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:29.611520 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw"] Oct 30 13:22:29.614792 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:22:29.614762 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff585ad9_f06c_453b_b23f_f0519a479d9a.slice/crio-5650596105b0f78dd7820051dbbb449210aab6c8c8aee3be96f74f351e598ee6 WatchSource:0}: Error finding container 5650596105b0f78dd7820051dbbb449210aab6c8c8aee3be96f74f351e598ee6: Status 404 returned error can't find the container with id 5650596105b0f78dd7820051dbbb449210aab6c8c8aee3be96f74f351e598ee6 Oct 30 13:22:29.616855 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:29.616824 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:22:29.616942 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:29.616887 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:22:29.616942 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:29.616918 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:22:30.619606 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:30.619569 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" event={"ID":"ff585ad9-f06c-453b-b23f-f0519a479d9a","Type":"ContainerStarted","Data":"bdfbc1cb5d79eca526371ea3f3a307c24a6399f3dea7ae9c5e0cfaca88efd078"} Oct 30 13:22:30.620046 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:30.619638 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" event={"ID":"ff585ad9-f06c-453b-b23f-f0519a479d9a","Type":"ContainerStarted","Data":"5650596105b0f78dd7820051dbbb449210aab6c8c8aee3be96f74f351e598ee6"} Oct 30 13:22:30.666753 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:30.666710 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" podStartSLOduration=2.666696619 podStartE2EDuration="2.666696619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:22:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:22:30.664956676 +0000 UTC m=+1807.134372438" watchObservedRunningTime="2026-10-30 13:22:30.666696619 +0000 UTC m=+1807.136112364" Oct 30 13:22:31.298061 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:31.298027 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" Oct 30 13:22:31.302950 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:31.302925 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" Oct 30 13:22:31.624081 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:31.624011 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" Oct 30 13:22:31.624935 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:31.624908 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" Oct 30 13:22:33.043476 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:33.043439 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:33.190819 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:33.190784 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:34.713388 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:34.713349 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:34.919760 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:34.919521 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:34.942278 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:34.942248 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cc21d-69ff6rvfqz"] Oct 30 13:22:34.945874 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:34.945858 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cc21d-69ff6rvfqz" Oct 30 13:22:34.966995 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:34.966929 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cc21d-69ff6rvfqz"] Oct 30 13:22:34.974390 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:34.974366 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-2cc21d\"" Oct 30 13:22:34.974517 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:34.974452 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-a75fe846-dockercfg-6bvct\"" Oct 30 13:22:34.993585 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:34.993550 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/187b4311-8685-4e4b-b7ab-5affa9de6565-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-2cc21d-69ff6rvfqz\" (UID: \"187b4311-8685-4e4b-b7ab-5affa9de6565\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cc21d-69ff6rvfqz" Oct 30 13:22:34.993773 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:34.993756 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csgcg\" (UniqueName: \"kubernetes.io/projected/187b4311-8685-4e4b-b7ab-5affa9de6565-kube-api-access-csgcg\") pod \"payload-pre-processing-e2e-worker-single-main-2cc21d-69ff6rvfqz\" (UID: \"187b4311-8685-4e4b-b7ab-5affa9de6565\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cc21d-69ff6rvfqz" Oct 30 13:22:35.039452 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:35.039414 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2cc21d-84455c4f45pp8q"] Oct 30 13:22:35.044888 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:35.044858 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2cc21d-84455c4f45pp8q" Oct 30 13:22:35.046873 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:35.046845 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:35.062658 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:35.062632 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2cc21d-84455c4f45pp8q"] Oct 30 13:22:35.094963 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:35.094927 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a949341-b8f4-4e96-81d9-bfe93e05b6e0-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-2cc21d-84455c4f45pp8q\" (UID: \"6a949341-b8f4-4e96-81d9-bfe93e05b6e0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2cc21d-84455c4f45pp8q" Oct 30 13:22:35.095105 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:35.094977 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chv27\" (UniqueName: \"kubernetes.io/projected/6a949341-b8f4-4e96-81d9-bfe93e05b6e0-kube-api-access-chv27\") pod \"payload-processing-e2e-worker-single-main-2cc21d-84455c4f45pp8q\" (UID: \"6a949341-b8f4-4e96-81d9-bfe93e05b6e0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2cc21d-84455c4f45pp8q" Oct 30 13:22:35.095161 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:35.095137 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csgcg\" (UniqueName: \"kubernetes.io/projected/187b4311-8685-4e4b-b7ab-5affa9de6565-kube-api-access-csgcg\") pod \"payload-pre-processing-e2e-worker-single-main-2cc21d-69ff6rvfqz\" (UID: \"187b4311-8685-4e4b-b7ab-5affa9de6565\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cc21d-69ff6rvfqz" Oct 30 13:22:35.095217 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:35.095200 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/187b4311-8685-4e4b-b7ab-5affa9de6565-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-2cc21d-69ff6rvfqz\" (UID: \"187b4311-8685-4e4b-b7ab-5affa9de6565\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cc21d-69ff6rvfqz" Oct 30 13:22:35.096002 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:35.095978 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/187b4311-8685-4e4b-b7ab-5affa9de6565-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-2cc21d-69ff6rvfqz\" (UID: \"187b4311-8685-4e4b-b7ab-5affa9de6565\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cc21d-69ff6rvfqz" Oct 30 13:22:35.119328 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:35.119303 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csgcg\" (UniqueName: \"kubernetes.io/projected/187b4311-8685-4e4b-b7ab-5affa9de6565-kube-api-access-csgcg\") pod \"payload-pre-processing-e2e-worker-single-main-2cc21d-69ff6rvfqz\" (UID: \"187b4311-8685-4e4b-b7ab-5affa9de6565\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cc21d-69ff6rvfqz" Oct 30 13:22:35.196105 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:35.196072 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a949341-b8f4-4e96-81d9-bfe93e05b6e0-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-2cc21d-84455c4f45pp8q\" (UID: \"6a949341-b8f4-4e96-81d9-bfe93e05b6e0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2cc21d-84455c4f45pp8q" Oct 30 13:22:35.196219 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:35.196193 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chv27\" (UniqueName: \"kubernetes.io/projected/6a949341-b8f4-4e96-81d9-bfe93e05b6e0-kube-api-access-chv27\") pod \"payload-processing-e2e-worker-single-main-2cc21d-84455c4f45pp8q\" (UID: \"6a949341-b8f4-4e96-81d9-bfe93e05b6e0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2cc21d-84455c4f45pp8q" Oct 30 13:22:35.196734 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:35.196712 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a949341-b8f4-4e96-81d9-bfe93e05b6e0-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-2cc21d-84455c4f45pp8q\" (UID: \"6a949341-b8f4-4e96-81d9-bfe93e05b6e0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2cc21d-84455c4f45pp8q" Oct 30 13:22:35.220301 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:35.220251 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chv27\" (UniqueName: \"kubernetes.io/projected/6a949341-b8f4-4e96-81d9-bfe93e05b6e0-kube-api-access-chv27\") pod \"payload-processing-e2e-worker-single-main-2cc21d-84455c4f45pp8q\" (UID: \"6a949341-b8f4-4e96-81d9-bfe93e05b6e0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2cc21d-84455c4f45pp8q" Oct 30 13:22:35.259295 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:35.259278 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cc21d-69ff6rvfqz" Oct 30 13:22:35.361591 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:35.361561 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2cc21d-84455c4f45pp8q" Oct 30 13:22:35.558400 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:35.558374 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cc21d-69ff6rvfqz"] Oct 30 13:22:35.560830 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:22:35.560803 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod187b4311_8685_4e4b_b7ab_5affa9de6565.slice/crio-0ebc631047b5a926d4ad83dd36386102b056c3a1b9a6b7926d86555319a4ac0a WatchSource:0}: Error finding container 0ebc631047b5a926d4ad83dd36386102b056c3a1b9a6b7926d86555319a4ac0a: Status 404 returned error can't find the container with id 0ebc631047b5a926d4ad83dd36386102b056c3a1b9a6b7926d86555319a4ac0a Oct 30 13:22:35.644451 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:35.644415 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cc21d-69ff6rvfqz" event={"ID":"187b4311-8685-4e4b-b7ab-5affa9de6565","Type":"ContainerStarted","Data":"306ef86256481ab9d600b69582380db42ac68cf326762d14cb09513c56d40c06"} Oct 30 13:22:35.644555 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:35.644465 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cc21d-69ff6rvfqz" event={"ID":"187b4311-8685-4e4b-b7ab-5affa9de6565","Type":"ContainerStarted","Data":"0ebc631047b5a926d4ad83dd36386102b056c3a1b9a6b7926d86555319a4ac0a"} Oct 30 13:22:35.644602 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:35.644568 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cc21d-69ff6rvfqz" Oct 30 13:22:35.663159 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:35.663132 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2cc21d-84455c4f45pp8q"] Oct 30 13:22:35.672951 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:22:35.672918 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a949341_b8f4_4e96_81d9_bfe93e05b6e0.slice/crio-88c6ffd60654f9bea9035e976661f70785dc3d66f0d9827132b6a83b49d52c89 WatchSource:0}: Error finding container 88c6ffd60654f9bea9035e976661f70785dc3d66f0d9827132b6a83b49d52c89: Status 404 returned error can't find the container with id 88c6ffd60654f9bea9035e976661f70785dc3d66f0d9827132b6a83b49d52c89 Oct 30 13:22:35.687006 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:35.686889 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cc21d-69ff6rvfqz" podStartSLOduration=1.686878614 podStartE2EDuration="1.686878614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:22:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:22:35.686359653 +0000 UTC m=+1812.155775416" watchObservedRunningTime="2026-10-30 13:22:35.686878614 +0000 UTC m=+1812.156294359" Oct 30 13:22:36.651267 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:36.651224 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2cc21d-84455c4f45pp8q" event={"ID":"6a949341-b8f4-4e96-81d9-bfe93e05b6e0","Type":"ContainerStarted","Data":"a555581d54766813e10e72d3ab33c3624efc4db03f5fac7382fa27acf8950c76"} Oct 30 13:22:36.651764 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:36.651284 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2cc21d-84455c4f45pp8q" Oct 30 13:22:36.651764 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:36.651303 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2cc21d-84455c4f45pp8q" event={"ID":"6a949341-b8f4-4e96-81d9-bfe93e05b6e0","Type":"ContainerStarted","Data":"88c6ffd60654f9bea9035e976661f70785dc3d66f0d9827132b6a83b49d52c89"} Oct 30 13:22:36.694989 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:36.694944 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2cc21d-84455c4f45pp8q" podStartSLOduration=1.694931991 podStartE2EDuration="1.694931991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:22:36.693203551 +0000 UTC m=+1813.162619289" watchObservedRunningTime="2026-10-30 13:22:36.694931991 +0000 UTC m=+1813.164347736" Oct 30 13:22:36.696972 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:36.696943 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:38.518453 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:38.518409 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:22:38.523888 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:38.523857 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:40.321334 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:40.321293 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:42.205802 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:42.205769 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:44.092684 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:44.092647 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:45.941720 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:45.941687 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:46.654309 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:46.654281 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cc21d-69ff6rvfqz" Oct 30 13:22:47.658800 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:47.658775 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2cc21d-84455c4f45pp8q" Oct 30 13:22:47.671891 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:47.665707 5922 patch_prober.go:29] interesting pod/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:22:47.671891 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:47.665757 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" podUID="ff585ad9-f06c-453b-b23f-f0519a479d9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:22:47.728476 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:47.728414 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:47.973225 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:47.973196 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:48.572529 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:48.572495 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:48.694443 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:48.694405 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:48.740507 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:48.740481 5922 generic.go:356] "Generic (PLEG): container finished" podID="ff585ad9-f06c-453b-b23f-f0519a479d9a" containerID="bdfbc1cb5d79eca526371ea3f3a307c24a6399f3dea7ae9c5e0cfaca88efd078" exitCode=0 Oct 30 13:22:48.740705 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:48.740555 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" event={"ID":"ff585ad9-f06c-453b-b23f-f0519a479d9a","Type":"ContainerDied","Data":"bdfbc1cb5d79eca526371ea3f3a307c24a6399f3dea7ae9c5e0cfaca88efd078"} Oct 30 13:22:48.740949 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:48.740932 5922 scope.go:122] "RemoveContainer" containerID="bdfbc1cb5d79eca526371ea3f3a307c24a6399f3dea7ae9c5e0cfaca88efd078" Oct 30 13:22:48.742176 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:48.742148 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:22:48.742233 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:48.742219 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:22:48.742270 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:48.742254 5922 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:22:48.983363 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:48.983335 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:49.622161 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:49.622121 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:49.749088 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:49.748004 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" event={"ID":"ff585ad9-f06c-453b-b23f-f0519a479d9a","Type":"ContainerStarted","Data":"c75875f019cce449c7b702fa44c14533bda62857886a6a0cc7234e29d68222c7"} Oct 30 13:22:50.298150 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:50.298117 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" Oct 30 13:22:50.298327 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:50.298277 5922 patch_prober.go:29] interesting pod/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 13:22:50.298424 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:50.298318 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" podUID="ff585ad9-f06c-453b-b23f-f0519a479d9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 13:22:51.198087 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:51.198053 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-2cc21d/simulator-single-main-2cc21d-kserve-664597f55c-b4wt2"] Oct 30 13:22:51.200854 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:51.200828 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2cc21d/simulator-single-main-2cc21d-kserve-664597f55c-b4wt2" Oct 30 13:22:51.247925 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:51.247902 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-2cc21d\"/\"openshift-service-ca.crt\"" Oct 30 13:22:51.258715 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:51.258691 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-2cc21d\"/\"kube-root-ca.crt\"" Oct 30 13:22:51.258855 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:51.258741 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-2cc21d\"/\"simulator-single-main-2cc21d-kserve-self-signed-certs\"" Oct 30 13:22:51.260027 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:51.260013 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-2cc21d\"/\"default-dockercfg-xlk2x\"" Oct 30 13:22:51.260322 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:51.260301 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-2cc21d/simulator-single-main-2cc21d-kserve-664597f55c-b4wt2"] Oct 30 13:22:51.299203 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:51.299171 5922 patch_prober.go:29] interesting pod/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 13:22:51.299354 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:51.299253 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" podUID="ff585ad9-f06c-453b-b23f-f0519a479d9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 13:22:51.302793 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:51.302764 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/780b3ef7-5e06-48be-9ed0-d9223b273061-model-cache\") pod \"simulator-single-main-2cc21d-kserve-664597f55c-b4wt2\" (UID: \"780b3ef7-5e06-48be-9ed0-d9223b273061\") " pod="e2e-models-e2e-worker-single-main-2cc21d/simulator-single-main-2cc21d-kserve-664597f55c-b4wt2" Oct 30 13:22:51.302872 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:51.302826 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/780b3ef7-5e06-48be-9ed0-d9223b273061-dshm\") pod \"simulator-single-main-2cc21d-kserve-664597f55c-b4wt2\" (UID: \"780b3ef7-5e06-48be-9ed0-d9223b273061\") " pod="e2e-models-e2e-worker-single-main-2cc21d/simulator-single-main-2cc21d-kserve-664597f55c-b4wt2" Oct 30 13:22:51.302910 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:51.302894 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/780b3ef7-5e06-48be-9ed0-d9223b273061-home\") pod \"simulator-single-main-2cc21d-kserve-664597f55c-b4wt2\" (UID: \"780b3ef7-5e06-48be-9ed0-d9223b273061\") " pod="e2e-models-e2e-worker-single-main-2cc21d/simulator-single-main-2cc21d-kserve-664597f55c-b4wt2" Oct 30 13:22:51.302983 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:51.302954 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8lzv\" (UniqueName: \"kubernetes.io/projected/780b3ef7-5e06-48be-9ed0-d9223b273061-kube-api-access-p8lzv\") pod \"simulator-single-main-2cc21d-kserve-664597f55c-b4wt2\" (UID: \"780b3ef7-5e06-48be-9ed0-d9223b273061\") " pod="e2e-models-e2e-worker-single-main-2cc21d/simulator-single-main-2cc21d-kserve-664597f55c-b4wt2" Oct 30 13:22:51.303104 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:51.303031 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/780b3ef7-5e06-48be-9ed0-d9223b273061-tmp-dir\") pod \"simulator-single-main-2cc21d-kserve-664597f55c-b4wt2\" (UID: \"780b3ef7-5e06-48be-9ed0-d9223b273061\") " pod="e2e-models-e2e-worker-single-main-2cc21d/simulator-single-main-2cc21d-kserve-664597f55c-b4wt2" Oct 30 13:22:51.303148 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:51.303124 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/780b3ef7-5e06-48be-9ed0-d9223b273061-tls-certs\") pod \"simulator-single-main-2cc21d-kserve-664597f55c-b4wt2\" (UID: \"780b3ef7-5e06-48be-9ed0-d9223b273061\") " pod="e2e-models-e2e-worker-single-main-2cc21d/simulator-single-main-2cc21d-kserve-664597f55c-b4wt2" Oct 30 13:22:51.331939 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:51.331915 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:51.403824 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:51.403785 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/780b3ef7-5e06-48be-9ed0-d9223b273061-tls-certs\") pod \"simulator-single-main-2cc21d-kserve-664597f55c-b4wt2\" (UID: \"780b3ef7-5e06-48be-9ed0-d9223b273061\") " pod="e2e-models-e2e-worker-single-main-2cc21d/simulator-single-main-2cc21d-kserve-664597f55c-b4wt2" Oct 30 13:22:51.404004 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:51.403844 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/780b3ef7-5e06-48be-9ed0-d9223b273061-model-cache\") pod \"simulator-single-main-2cc21d-kserve-664597f55c-b4wt2\" (UID: \"780b3ef7-5e06-48be-9ed0-d9223b273061\") " pod="e2e-models-e2e-worker-single-main-2cc21d/simulator-single-main-2cc21d-kserve-664597f55c-b4wt2" Oct 30 13:22:51.404004 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:51.403887 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/780b3ef7-5e06-48be-9ed0-d9223b273061-dshm\") pod \"simulator-single-main-2cc21d-kserve-664597f55c-b4wt2\" (UID: \"780b3ef7-5e06-48be-9ed0-d9223b273061\") " pod="e2e-models-e2e-worker-single-main-2cc21d/simulator-single-main-2cc21d-kserve-664597f55c-b4wt2" Oct 30 13:22:51.404004 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:51.403922 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/780b3ef7-5e06-48be-9ed0-d9223b273061-home\") pod \"simulator-single-main-2cc21d-kserve-664597f55c-b4wt2\" (UID: \"780b3ef7-5e06-48be-9ed0-d9223b273061\") " pod="e2e-models-e2e-worker-single-main-2cc21d/simulator-single-main-2cc21d-kserve-664597f55c-b4wt2" Oct 30 13:22:51.404004 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:51.403984 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8lzv\" (UniqueName: \"kubernetes.io/projected/780b3ef7-5e06-48be-9ed0-d9223b273061-kube-api-access-p8lzv\") pod \"simulator-single-main-2cc21d-kserve-664597f55c-b4wt2\" (UID: \"780b3ef7-5e06-48be-9ed0-d9223b273061\") " pod="e2e-models-e2e-worker-single-main-2cc21d/simulator-single-main-2cc21d-kserve-664597f55c-b4wt2" Oct 30 13:22:51.404210 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:51.404017 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/780b3ef7-5e06-48be-9ed0-d9223b273061-tmp-dir\") pod \"simulator-single-main-2cc21d-kserve-664597f55c-b4wt2\" (UID: \"780b3ef7-5e06-48be-9ed0-d9223b273061\") " pod="e2e-models-e2e-worker-single-main-2cc21d/simulator-single-main-2cc21d-kserve-664597f55c-b4wt2" Oct 30 13:22:51.404400 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:51.404362 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/780b3ef7-5e06-48be-9ed0-d9223b273061-model-cache\") pod \"simulator-single-main-2cc21d-kserve-664597f55c-b4wt2\" (UID: \"780b3ef7-5e06-48be-9ed0-d9223b273061\") " pod="e2e-models-e2e-worker-single-main-2cc21d/simulator-single-main-2cc21d-kserve-664597f55c-b4wt2" Oct 30 13:22:51.404530 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:51.404509 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/780b3ef7-5e06-48be-9ed0-d9223b273061-home\") pod \"simulator-single-main-2cc21d-kserve-664597f55c-b4wt2\" (UID: \"780b3ef7-5e06-48be-9ed0-d9223b273061\") " pod="e2e-models-e2e-worker-single-main-2cc21d/simulator-single-main-2cc21d-kserve-664597f55c-b4wt2" Oct 30 13:22:51.404601 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:51.404548 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/780b3ef7-5e06-48be-9ed0-d9223b273061-tmp-dir\") pod \"simulator-single-main-2cc21d-kserve-664597f55c-b4wt2\" (UID: \"780b3ef7-5e06-48be-9ed0-d9223b273061\") " pod="e2e-models-e2e-worker-single-main-2cc21d/simulator-single-main-2cc21d-kserve-664597f55c-b4wt2" Oct 30 13:22:51.406342 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:51.406321 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/780b3ef7-5e06-48be-9ed0-d9223b273061-dshm\") pod \"simulator-single-main-2cc21d-kserve-664597f55c-b4wt2\" (UID: \"780b3ef7-5e06-48be-9ed0-d9223b273061\") " pod="e2e-models-e2e-worker-single-main-2cc21d/simulator-single-main-2cc21d-kserve-664597f55c-b4wt2" Oct 30 13:22:51.406873 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:51.406849 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/780b3ef7-5e06-48be-9ed0-d9223b273061-tls-certs\") pod \"simulator-single-main-2cc21d-kserve-664597f55c-b4wt2\" (UID: \"780b3ef7-5e06-48be-9ed0-d9223b273061\") " pod="e2e-models-e2e-worker-single-main-2cc21d/simulator-single-main-2cc21d-kserve-664597f55c-b4wt2" Oct 30 13:22:51.427775 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:51.427751 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8lzv\" (UniqueName: \"kubernetes.io/projected/780b3ef7-5e06-48be-9ed0-d9223b273061-kube-api-access-p8lzv\") pod \"simulator-single-main-2cc21d-kserve-664597f55c-b4wt2\" (UID: \"780b3ef7-5e06-48be-9ed0-d9223b273061\") " pod="e2e-models-e2e-worker-single-main-2cc21d/simulator-single-main-2cc21d-kserve-664597f55c-b4wt2" Oct 30 13:22:51.512170 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:51.512083 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2cc21d/simulator-single-main-2cc21d-kserve-664597f55c-b4wt2" Oct 30 13:22:51.543110 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:51.543072 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:51.828285 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:51.828101 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-2cc21d/simulator-single-main-2cc21d-kserve-664597f55c-b4wt2"] Oct 30 13:22:51.831005 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:22:51.830976 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod780b3ef7_5e06_48be_9ed0_d9223b273061.slice/crio-4cbe3feef4541d6f2ada7191ccc04e1fc7307cf22511811a48ed9e2f087867d9 WatchSource:0}: Error finding container 4cbe3feef4541d6f2ada7191ccc04e1fc7307cf22511811a48ed9e2f087867d9: Status 404 returned error can't find the container with id 4cbe3feef4541d6f2ada7191ccc04e1fc7307cf22511811a48ed9e2f087867d9 Oct 30 13:22:52.298246 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:52.298207 5922 patch_prober.go:29] interesting pod/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 13:22:52.298664 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:52.298264 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" podUID="ff585ad9-f06c-453b-b23f-f0519a479d9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 13:22:52.786636 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:52.782904 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2cc21d/simulator-single-main-2cc21d-kserve-664597f55c-b4wt2" event={"ID":"780b3ef7-5e06-48be-9ed0-d9223b273061","Type":"ContainerStarted","Data":"3365448cf21e323f1eed4782d1b15abc27a754c427d4926951a27d44121ce4e2"} Oct 30 13:22:52.786636 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:52.782967 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2cc21d/simulator-single-main-2cc21d-kserve-664597f55c-b4wt2" event={"ID":"780b3ef7-5e06-48be-9ed0-d9223b273061","Type":"ContainerStarted","Data":"4cbe3feef4541d6f2ada7191ccc04e1fc7307cf22511811a48ed9e2f087867d9"} Oct 30 13:22:52.826941 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:52.826896 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-2cc21d/simulator-single-main-2cc21d-kserve-664597f55c-b4wt2" podStartSLOduration=1.826883385 podStartE2EDuration="1.826883385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:22:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:22:52.825281782 +0000 UTC m=+1829.294697545" watchObservedRunningTime="2026-10-30 13:22:52.826883385 +0000 UTC m=+1829.296299130" Oct 30 13:22:53.298244 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:53.298210 5922 patch_prober.go:29] interesting pod/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 13:22:53.298423 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:53.298262 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" podUID="ff585ad9-f06c-453b-b23f-f0519a479d9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 13:22:53.387806 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:53.387770 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:53.452851 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:53.452818 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:22:53.513255 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:53.513220 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-2cc21d/simulator-single-main-2cc21d-kserve-664597f55c-b4wt2" Oct 30 13:22:53.557992 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:22:53.557911 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:22:54.299074 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:54.299034 5922 patch_prober.go:29] interesting pod/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 13:22:54.299574 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:54.299085 5922 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" podUID="ff585ad9-f06c-453b-b23f-f0519a479d9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 13:22:55.180738 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:55.180695 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:55.302459 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:55.302420 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" Oct 30 13:22:55.302850 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:55.302772 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" Oct 30 13:22:55.303374 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:55.303356 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" Oct 30 13:22:57.223728 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:57.223692 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:59.008767 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:22:59.008731 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:00.837939 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:00.837905 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:01.513161 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:01.513134 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-2cc21d/simulator-single-main-2cc21d-kserve-664597f55c-b4wt2" Oct 30 13:23:01.525314 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:01.525290 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-2cc21d/simulator-single-main-2cc21d-kserve-664597f55c-b4wt2" Oct 30 13:23:01.840250 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:01.840178 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-2cc21d/simulator-single-main-2cc21d-kserve-664597f55c-b4wt2" Oct 30 13:23:02.829830 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:02.829774 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:03.234341 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:03.234302 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:03.348268 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:03.348235 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:04.615081 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:04.615047 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:06.397538 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:06.397494 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:07.725476 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:07.725444 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:23:08.239695 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:08.239658 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:09.609040 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:09.609006 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c27731c-wnfxv"] Oct 30 13:23:09.612296 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:09.612280 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c27731c-wnfxv" Oct 30 13:23:09.631278 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:09.631248 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c27731c-wnfxv"] Oct 30 13:23:09.644659 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:09.641565 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:23:09.644659 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:09.641584 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:23:09.644659 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:09.641732 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:23:09.644659 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:09.641840 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:23:09.698026 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:09.698003 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf64f284-eceb-44e1-a9da-9209a50ae78b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c27731c-wnfxv\" (UID: \"bf64f284-eceb-44e1-a9da-9209a50ae78b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c27731c-wnfxv" Oct 30 13:23:09.698154 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:09.698032 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf64f284-eceb-44e1-a9da-9209a50ae78b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c27731c-wnfxv\" (UID: \"bf64f284-eceb-44e1-a9da-9209a50ae78b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c27731c-wnfxv" Oct 30 13:23:09.698154 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:09.698117 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9k8s\" (UniqueName: \"kubernetes.io/projected/bf64f284-eceb-44e1-a9da-9209a50ae78b-kube-api-access-b9k8s\") pod \"maas-api-revoke-sub-9c27731c-wnfxv\" (UID: \"bf64f284-eceb-44e1-a9da-9209a50ae78b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c27731c-wnfxv" Oct 30 13:23:09.799228 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:09.799194 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf64f284-eceb-44e1-a9da-9209a50ae78b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c27731c-wnfxv\" (UID: \"bf64f284-eceb-44e1-a9da-9209a50ae78b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c27731c-wnfxv" Oct 30 13:23:09.799379 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:09.799245 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf64f284-eceb-44e1-a9da-9209a50ae78b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c27731c-wnfxv\" (UID: \"bf64f284-eceb-44e1-a9da-9209a50ae78b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c27731c-wnfxv" Oct 30 13:23:09.799379 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:09.799308 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9k8s\" (UniqueName: \"kubernetes.io/projected/bf64f284-eceb-44e1-a9da-9209a50ae78b-kube-api-access-b9k8s\") pod \"maas-api-revoke-sub-9c27731c-wnfxv\" (UID: \"bf64f284-eceb-44e1-a9da-9209a50ae78b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c27731c-wnfxv" Oct 30 13:23:09.799846 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:09.799826 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf64f284-eceb-44e1-a9da-9209a50ae78b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c27731c-wnfxv\" (UID: \"bf64f284-eceb-44e1-a9da-9209a50ae78b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c27731c-wnfxv" Oct 30 13:23:09.801576 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:09.801557 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf64f284-eceb-44e1-a9da-9209a50ae78b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c27731c-wnfxv\" (UID: \"bf64f284-eceb-44e1-a9da-9209a50ae78b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c27731c-wnfxv" Oct 30 13:23:09.823289 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:09.823265 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9k8s\" (UniqueName: \"kubernetes.io/projected/bf64f284-eceb-44e1-a9da-9209a50ae78b-kube-api-access-b9k8s\") pod \"maas-api-revoke-sub-9c27731c-wnfxv\" (UID: \"bf64f284-eceb-44e1-a9da-9209a50ae78b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c27731c-wnfxv" Oct 30 13:23:09.924163 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:09.924109 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c27731c-wnfxv" Oct 30 13:23:10.043879 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:10.043842 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:10.222602 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:10.222577 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c27731c-wnfxv"] Oct 30 13:23:10.224363 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:23:10.224334 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf64f284_eceb_44e1_a9da_9209a50ae78b.slice/crio-f7a2137582354271bdd07afd3a3f78becc7278aac64a0d5c76ca7cafd674c76f WatchSource:0}: Error finding container f7a2137582354271bdd07afd3a3f78becc7278aac64a0d5c76ca7cafd674c76f: Status 404 returned error can't find the container with id f7a2137582354271bdd07afd3a3f78becc7278aac64a0d5c76ca7cafd674c76f Oct 30 13:23:10.875408 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:10.875378 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c27731c-wnfxv" event={"ID":"bf64f284-eceb-44e1-a9da-9209a50ae78b","Type":"ContainerStarted","Data":"80c27ec4186179ce4d1e1d1c504dab9cfa27ee439a3fea9d48737aba9b4ef8b3"} Oct 30 13:23:10.875793 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:10.875426 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c27731c-wnfxv" event={"ID":"bf64f284-eceb-44e1-a9da-9209a50ae78b","Type":"ContainerStarted","Data":"f7a2137582354271bdd07afd3a3f78becc7278aac64a0d5c76ca7cafd674c76f"} Oct 30 13:23:10.917776 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:10.917734 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c27731c-wnfxv" podStartSLOduration=1.917722723 podStartE2EDuration="1.917722723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:23:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:23:10.916334998 +0000 UTC m=+1847.385750743" watchObservedRunningTime="2026-10-30 13:23:10.917722723 +0000 UTC m=+1847.387138502" Oct 30 13:23:11.849630 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:11.849588 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:11.881838 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:11.881805 5922 generic.go:356] "Generic (PLEG): container finished" podID="bf64f284-eceb-44e1-a9da-9209a50ae78b" containerID="80c27ec4186179ce4d1e1d1c504dab9cfa27ee439a3fea9d48737aba9b4ef8b3" exitCode=0 Oct 30 13:23:11.882228 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:11.881870 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c27731c-wnfxv" event={"ID":"bf64f284-eceb-44e1-a9da-9209a50ae78b","Type":"ContainerDied","Data":"80c27ec4186179ce4d1e1d1c504dab9cfa27ee439a3fea9d48737aba9b4ef8b3"} Oct 30 13:23:11.968386 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:11.968352 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:12.085920 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:12.085876 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:13.024140 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:13.024116 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c27731c-wnfxv" Oct 30 13:23:13.129355 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:13.129322 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf64f284-eceb-44e1-a9da-9209a50ae78b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf64f284-eceb-44e1-a9da-9209a50ae78b-maas-api-xks-ca\") pod \"bf64f284-eceb-44e1-a9da-9209a50ae78b\" (UID: \"bf64f284-eceb-44e1-a9da-9209a50ae78b\") " Oct 30 13:23:13.129504 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:13.129402 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf64f284-eceb-44e1-a9da-9209a50ae78b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf64f284-eceb-44e1-a9da-9209a50ae78b-maas-api-openshift-ca\") pod \"bf64f284-eceb-44e1-a9da-9209a50ae78b\" (UID: \"bf64f284-eceb-44e1-a9da-9209a50ae78b\") " Oct 30 13:23:13.129504 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:13.129466 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf64f284-eceb-44e1-a9da-9209a50ae78b-kube-api-access-b9k8s\" (UniqueName: \"kubernetes.io/projected/bf64f284-eceb-44e1-a9da-9209a50ae78b-kube-api-access-b9k8s\") pod \"bf64f284-eceb-44e1-a9da-9209a50ae78b\" (UID: \"bf64f284-eceb-44e1-a9da-9209a50ae78b\") " Oct 30 13:23:13.129771 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:13.129749 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf64f284-eceb-44e1-a9da-9209a50ae78b-maas-api-openshift-ca" pod "bf64f284-eceb-44e1-a9da-9209a50ae78b" (UID: "bf64f284-eceb-44e1-a9da-9209a50ae78b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:23:13.129921 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:13.129902 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf64f284-eceb-44e1-a9da-9209a50ae78b-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:23:13.131346 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:13.131326 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf64f284-eceb-44e1-a9da-9209a50ae78b-maas-api-xks-ca" pod "bf64f284-eceb-44e1-a9da-9209a50ae78b" (UID: "bf64f284-eceb-44e1-a9da-9209a50ae78b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:23:13.131406 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:13.131342 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf64f284-eceb-44e1-a9da-9209a50ae78b-kube-api-access-b9k8s" pod "bf64f284-eceb-44e1-a9da-9209a50ae78b" (UID: "bf64f284-eceb-44e1-a9da-9209a50ae78b"). InnerVolumeSpecName "kube-api-access-b9k8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:23:13.230945 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:13.230919 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9k8s\" (UniqueName: \"kubernetes.io/projected/bf64f284-eceb-44e1-a9da-9209a50ae78b-kube-api-access-b9k8s\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:23:13.230945 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:13.230941 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf64f284-eceb-44e1-a9da-9209a50ae78b-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:23:13.822625 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:13.822572 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:13.897444 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:13.897401 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c27731c-wnfxv" Oct 30 13:23:13.897642 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:13.897403 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c27731c-wnfxv" event={"ID":"bf64f284-eceb-44e1-a9da-9209a50ae78b","Type":"ContainerDied","Data":"f7a2137582354271bdd07afd3a3f78becc7278aac64a0d5c76ca7cafd674c76f"} Oct 30 13:23:13.897642 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:13.897534 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7a2137582354271bdd07afd3a3f78becc7278aac64a0d5c76ca7cafd674c76f" Oct 30 13:23:15.618440 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:15.618406 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:17.432177 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:17.432143 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:19.194893 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:19.194850 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:20.499204 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:20.499169 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:20.657854 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:20.657823 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cc21d-6g8v8"] Oct 30 13:23:20.658229 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:20.658217 5922 cpu_manager.go:414] "containerMap: removing container" podUID="bf64f284-eceb-44e1-a9da-9209a50ae78b" containerName="revoke-keys" Oct 30 13:23:20.658279 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:20.658231 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf64f284-eceb-44e1-a9da-9209a50ae78b" containerName="revoke-keys" Oct 30 13:23:20.658329 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:20.658320 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf64f284-eceb-44e1-a9da-9209a50ae78b" containerName="revoke-keys" Oct 30 13:23:20.660339 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:20.660323 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cc21d-6g8v8" Oct 30 13:23:20.676214 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:20.676193 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:23:20.676316 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:20.676227 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:23:20.687437 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:20.687414 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:23:20.687541 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:20.687491 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:23:20.689541 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:20.689507 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cc21d-6g8v8"] Oct 30 13:23:20.801075 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:20.800993 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmfs8\" (UniqueName: \"kubernetes.io/projected/880d3e6b-4501-4619-96f3-8fde98d2b520-kube-api-access-vmfs8\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2cc21d-6g8v8\" (UID: \"880d3e6b-4501-4619-96f3-8fde98d2b520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cc21d-6g8v8" Oct 30 13:23:20.801216 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:20.801195 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/880d3e6b-4501-4619-96f3-8fde98d2b520-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2cc21d-6g8v8\" (UID: \"880d3e6b-4501-4619-96f3-8fde98d2b520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cc21d-6g8v8" Oct 30 13:23:20.801265 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:20.801223 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/880d3e6b-4501-4619-96f3-8fde98d2b520-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2cc21d-6g8v8\" (UID: \"880d3e6b-4501-4619-96f3-8fde98d2b520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cc21d-6g8v8" Oct 30 13:23:20.901735 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:20.901700 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmfs8\" (UniqueName: \"kubernetes.io/projected/880d3e6b-4501-4619-96f3-8fde98d2b520-kube-api-access-vmfs8\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2cc21d-6g8v8\" (UID: \"880d3e6b-4501-4619-96f3-8fde98d2b520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cc21d-6g8v8" Oct 30 13:23:20.901912 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:20.901822 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/880d3e6b-4501-4619-96f3-8fde98d2b520-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2cc21d-6g8v8\" (UID: \"880d3e6b-4501-4619-96f3-8fde98d2b520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cc21d-6g8v8" Oct 30 13:23:20.901912 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:20.901854 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/880d3e6b-4501-4619-96f3-8fde98d2b520-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2cc21d-6g8v8\" (UID: \"880d3e6b-4501-4619-96f3-8fde98d2b520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cc21d-6g8v8" Oct 30 13:23:20.902499 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:20.902474 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/880d3e6b-4501-4619-96f3-8fde98d2b520-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2cc21d-6g8v8\" (UID: \"880d3e6b-4501-4619-96f3-8fde98d2b520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cc21d-6g8v8" Oct 30 13:23:20.904111 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:20.904091 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/880d3e6b-4501-4619-96f3-8fde98d2b520-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2cc21d-6g8v8\" (UID: \"880d3e6b-4501-4619-96f3-8fde98d2b520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cc21d-6g8v8" Oct 30 13:23:20.925862 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:20.925837 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmfs8\" (UniqueName: \"kubernetes.io/projected/880d3e6b-4501-4619-96f3-8fde98d2b520-kube-api-access-vmfs8\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2cc21d-6g8v8\" (UID: \"880d3e6b-4501-4619-96f3-8fde98d2b520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cc21d-6g8v8" Oct 30 13:23:20.971559 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:20.971535 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cc21d-6g8v8" Oct 30 13:23:21.153723 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:21.153644 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:21.239763 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:21.239730 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:23:21.277182 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:21.277158 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cc21d-6g8v8"] Oct 30 13:23:21.280130 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:23:21.280102 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod880d3e6b_4501_4619_96f3_8fde98d2b520.slice/crio-401d068c13cde09cf8d1eae88f6d498ec3736ef5ff1fded3fb9598875de0ec10 WatchSource:0}: Error finding container 401d068c13cde09cf8d1eae88f6d498ec3736ef5ff1fded3fb9598875de0ec10: Status 404 returned error can't find the container with id 401d068c13cde09cf8d1eae88f6d498ec3736ef5ff1fded3fb9598875de0ec10 Oct 30 13:23:21.938195 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:21.938121 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cc21d-6g8v8" event={"ID":"880d3e6b-4501-4619-96f3-8fde98d2b520","Type":"ContainerStarted","Data":"e5e10a9684fb2c105a9294111bd3c6cc61199b1c1c488927d4a89c0da5fe8560"} Oct 30 13:23:21.938195 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:21.938172 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cc21d-6g8v8" event={"ID":"880d3e6b-4501-4619-96f3-8fde98d2b520","Type":"ContainerStarted","Data":"401d068c13cde09cf8d1eae88f6d498ec3736ef5ff1fded3fb9598875de0ec10"} Oct 30 13:23:21.979707 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:21.979654 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cc21d-6g8v8" podStartSLOduration=1.979638244 podStartE2EDuration="1.979638244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:23:21.978946872 +0000 UTC m=+1858.448362616" watchObservedRunningTime="2026-10-30 13:23:21.979638244 +0000 UTC m=+1858.449053987" Oct 30 13:23:22.944421 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:22.944389 5922 generic.go:356] "Generic (PLEG): container finished" podID="880d3e6b-4501-4619-96f3-8fde98d2b520" containerID="e5e10a9684fb2c105a9294111bd3c6cc61199b1c1c488927d4a89c0da5fe8560" exitCode=0 Oct 30 13:23:22.944854 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:22.944472 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cc21d-6g8v8" event={"ID":"880d3e6b-4501-4619-96f3-8fde98d2b520","Type":"ContainerDied","Data":"e5e10a9684fb2c105a9294111bd3c6cc61199b1c1c488927d4a89c0da5fe8560"} Oct 30 13:23:23.127521 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:23.127490 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:23.551986 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:23:23.551954 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:23:24.084254 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:24.084227 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cc21d-6g8v8" Oct 30 13:23:24.130819 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:24.130792 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/880d3e6b-4501-4619-96f3-8fde98d2b520-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/880d3e6b-4501-4619-96f3-8fde98d2b520-maas-api-openshift-ca\") pod \"880d3e6b-4501-4619-96f3-8fde98d2b520\" (UID: \"880d3e6b-4501-4619-96f3-8fde98d2b520\") " Oct 30 13:23:24.130946 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:24.130840 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/880d3e6b-4501-4619-96f3-8fde98d2b520-kube-api-access-vmfs8\" (UniqueName: \"kubernetes.io/projected/880d3e6b-4501-4619-96f3-8fde98d2b520-kube-api-access-vmfs8\") pod \"880d3e6b-4501-4619-96f3-8fde98d2b520\" (UID: \"880d3e6b-4501-4619-96f3-8fde98d2b520\") " Oct 30 13:23:24.130946 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:24.130920 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/880d3e6b-4501-4619-96f3-8fde98d2b520-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/880d3e6b-4501-4619-96f3-8fde98d2b520-maas-api-xks-ca\") pod \"880d3e6b-4501-4619-96f3-8fde98d2b520\" (UID: \"880d3e6b-4501-4619-96f3-8fde98d2b520\") " Oct 30 13:23:24.131216 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:24.131191 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/880d3e6b-4501-4619-96f3-8fde98d2b520-maas-api-openshift-ca" pod "880d3e6b-4501-4619-96f3-8fde98d2b520" (UID: "880d3e6b-4501-4619-96f3-8fde98d2b520"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:23:24.133512 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:24.133483 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/880d3e6b-4501-4619-96f3-8fde98d2b520-maas-api-xks-ca" pod "880d3e6b-4501-4619-96f3-8fde98d2b520" (UID: "880d3e6b-4501-4619-96f3-8fde98d2b520"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:23:24.133637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:24.133586 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/880d3e6b-4501-4619-96f3-8fde98d2b520-kube-api-access-vmfs8" pod "880d3e6b-4501-4619-96f3-8fde98d2b520" (UID: "880d3e6b-4501-4619-96f3-8fde98d2b520"). InnerVolumeSpecName "kube-api-access-vmfs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:23:24.232271 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:24.232208 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/880d3e6b-4501-4619-96f3-8fde98d2b520-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:23:24.232271 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:24.232232 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmfs8\" (UniqueName: \"kubernetes.io/projected/880d3e6b-4501-4619-96f3-8fde98d2b520-kube-api-access-vmfs8\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:23:24.232271 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:24.232243 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/880d3e6b-4501-4619-96f3-8fde98d2b520-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:23:24.917049 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:24.917018 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:24.958731 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:24.958696 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cc21d-6g8v8" event={"ID":"880d3e6b-4501-4619-96f3-8fde98d2b520","Type":"ContainerDied","Data":"401d068c13cde09cf8d1eae88f6d498ec3736ef5ff1fded3fb9598875de0ec10"} Oct 30 13:23:24.958933 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:24.958735 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cc21d-6g8v8" Oct 30 13:23:24.958933 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:24.958754 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="401d068c13cde09cf8d1eae88f6d498ec3736ef5ff1fded3fb9598875de0ec10" Oct 30 13:23:25.443032 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:25.442999 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-2cc21d/simulator-single-main-2cc21d-kserve-664597f55c-b4wt2"] Oct 30 13:23:25.538671 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:25.538641 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:26.752547 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:26.752512 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:28.507863 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:28.507823 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:30.315587 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:30.315550 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:31.115311 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:31.115277 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2cc21d-84455c4f45pp8q"] Oct 30 13:23:31.150399 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:31.150367 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cc21d-69ff6rvfqz"] Oct 30 13:23:31.272558 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:31.272516 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:31.397710 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:31.397587 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:32.216095 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:32.216060 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:33.863105 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:33.863070 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:34.400050 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:34.400013 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:34.530145 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:34.530106 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cc21d-6g8v8"] Oct 30 13:23:34.531781 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:34.531738 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:34.541800 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:34.541774 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2cc21d-6g8v8"] Oct 30 13:23:34.802332 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:34.802301 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:35.036526 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:35.036487 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:35.643119 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:35.643084 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="880d3e6b-4501-4619-96f3-8fde98d2b520" path="/var/lib/kubelet/pods/880d3e6b-4501-4619-96f3-8fde98d2b520/volumes" Oct 30 13:23:36.116587 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:36.116551 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2cc21d-84455c4f45pp8q" podUID="6a949341-b8f4-4e96-81d9-bfe93e05b6e0" containerName="payload-processing" containerID="cri-o://a555581d54766813e10e72d3ab33c3624efc4db03f5fac7382fa27acf8950c76" gracePeriod=25 Oct 30 13:23:36.150887 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:36.150850 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cc21d-69ff6rvfqz" podUID="187b4311-8685-4e4b-b7ab-5affa9de6565" containerName="payload-pre-processing" containerID="cri-o://306ef86256481ab9d600b69582380db42ac68cf326762d14cb09513c56d40c06" gracePeriod=25 Oct 30 13:23:36.220845 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:36.220809 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:23:36.653240 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:36.653206 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cc21d-69ff6rvfqz" podUID="187b4311-8685-4e4b-b7ab-5affa9de6565" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:23:37.657651 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:37.657593 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2cc21d-84455c4f45pp8q" podUID="6a949341-b8f4-4e96-81d9-bfe93e05b6e0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:23:40.464997 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:40.464958 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-2cc21d/simulator-single-main-2cc21d-kserve-664597f55c-b4wt2" podUID="780b3ef7-5e06-48be-9ed0-d9223b273061" containerName="main" containerID="cri-o://3365448cf21e323f1eed4782d1b15abc27a754c427d4926951a27d44121ce4e2" gracePeriod=45 Oct 30 13:23:40.712165 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:40.712138 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2cc21d/simulator-single-main-2cc21d-kserve-664597f55c-b4wt2" Oct 30 13:23:40.775303 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:40.775225 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/780b3ef7-5e06-48be-9ed0-d9223b273061-home\" (UniqueName: \"kubernetes.io/empty-dir/780b3ef7-5e06-48be-9ed0-d9223b273061-home\") pod \"780b3ef7-5e06-48be-9ed0-d9223b273061\" (UID: \"780b3ef7-5e06-48be-9ed0-d9223b273061\") " Oct 30 13:23:40.775453 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:40.775385 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/780b3ef7-5e06-48be-9ed0-d9223b273061-kube-api-access-p8lzv\" (UniqueName: \"kubernetes.io/projected/780b3ef7-5e06-48be-9ed0-d9223b273061-kube-api-access-p8lzv\") pod \"780b3ef7-5e06-48be-9ed0-d9223b273061\" (UID: \"780b3ef7-5e06-48be-9ed0-d9223b273061\") " Oct 30 13:23:40.775453 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:40.775421 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/780b3ef7-5e06-48be-9ed0-d9223b273061-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/780b3ef7-5e06-48be-9ed0-d9223b273061-tmp-dir\") pod \"780b3ef7-5e06-48be-9ed0-d9223b273061\" (UID: \"780b3ef7-5e06-48be-9ed0-d9223b273061\") " Oct 30 13:23:40.775573 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:40.775460 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/780b3ef7-5e06-48be-9ed0-d9223b273061-dshm\" (UniqueName: \"kubernetes.io/empty-dir/780b3ef7-5e06-48be-9ed0-d9223b273061-dshm\") pod \"780b3ef7-5e06-48be-9ed0-d9223b273061\" (UID: \"780b3ef7-5e06-48be-9ed0-d9223b273061\") " Oct 30 13:23:40.775573 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:40.775502 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780b3ef7-5e06-48be-9ed0-d9223b273061-home" pod "780b3ef7-5e06-48be-9ed0-d9223b273061" (UID: "780b3ef7-5e06-48be-9ed0-d9223b273061"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:23:40.775713 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:40.775597 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/780b3ef7-5e06-48be-9ed0-d9223b273061-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/780b3ef7-5e06-48be-9ed0-d9223b273061-model-cache\") pod \"780b3ef7-5e06-48be-9ed0-d9223b273061\" (UID: \"780b3ef7-5e06-48be-9ed0-d9223b273061\") " Oct 30 13:23:40.775713 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:40.775688 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/780b3ef7-5e06-48be-9ed0-d9223b273061-tls-certs\" (UniqueName: \"kubernetes.io/secret/780b3ef7-5e06-48be-9ed0-d9223b273061-tls-certs\") pod \"780b3ef7-5e06-48be-9ed0-d9223b273061\" (UID: \"780b3ef7-5e06-48be-9ed0-d9223b273061\") " Oct 30 13:23:40.775826 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:40.775731 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780b3ef7-5e06-48be-9ed0-d9223b273061-tmp-dir" pod "780b3ef7-5e06-48be-9ed0-d9223b273061" (UID: "780b3ef7-5e06-48be-9ed0-d9223b273061"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:23:40.775915 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:40.775896 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780b3ef7-5e06-48be-9ed0-d9223b273061-model-cache" pod "780b3ef7-5e06-48be-9ed0-d9223b273061" (UID: "780b3ef7-5e06-48be-9ed0-d9223b273061"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:23:40.776062 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:40.776045 5922 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/780b3ef7-5e06-48be-9ed0-d9223b273061-model-cache\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:23:40.776182 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:40.776064 5922 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/780b3ef7-5e06-48be-9ed0-d9223b273061-home\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:23:40.776182 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:40.776073 5922 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/780b3ef7-5e06-48be-9ed0-d9223b273061-tmp-dir\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:23:40.778117 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:40.778085 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/780b3ef7-5e06-48be-9ed0-d9223b273061-tls-certs" pod "780b3ef7-5e06-48be-9ed0-d9223b273061" (UID: "780b3ef7-5e06-48be-9ed0-d9223b273061"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:23:40.778374 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:40.778344 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780b3ef7-5e06-48be-9ed0-d9223b273061-dshm" pod "780b3ef7-5e06-48be-9ed0-d9223b273061" (UID: "780b3ef7-5e06-48be-9ed0-d9223b273061"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:23:40.778491 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:40.778399 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/780b3ef7-5e06-48be-9ed0-d9223b273061-kube-api-access-p8lzv" pod "780b3ef7-5e06-48be-9ed0-d9223b273061" (UID: "780b3ef7-5e06-48be-9ed0-d9223b273061"). InnerVolumeSpecName "kube-api-access-p8lzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:23:40.877554 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:40.877522 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8lzv\" (UniqueName: \"kubernetes.io/projected/780b3ef7-5e06-48be-9ed0-d9223b273061-kube-api-access-p8lzv\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:23:40.877554 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:40.877552 5922 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/780b3ef7-5e06-48be-9ed0-d9223b273061-dshm\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:23:40.877760 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:40.877561 5922 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/780b3ef7-5e06-48be-9ed0-d9223b273061-tls-certs\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:23:41.039714 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:41.039640 5922 generic.go:356] "Generic (PLEG): container finished" podID="780b3ef7-5e06-48be-9ed0-d9223b273061" containerID="3365448cf21e323f1eed4782d1b15abc27a754c427d4926951a27d44121ce4e2" exitCode=0 Oct 30 13:23:41.039714 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:41.039678 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2cc21d/simulator-single-main-2cc21d-kserve-664597f55c-b4wt2" event={"ID":"780b3ef7-5e06-48be-9ed0-d9223b273061","Type":"ContainerDied","Data":"3365448cf21e323f1eed4782d1b15abc27a754c427d4926951a27d44121ce4e2"} Oct 30 13:23:41.039714 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:41.039679 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2cc21d/simulator-single-main-2cc21d-kserve-664597f55c-b4wt2" Oct 30 13:23:41.039714 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:41.039713 5922 scope.go:122] "RemoveContainer" containerID="3365448cf21e323f1eed4782d1b15abc27a754c427d4926951a27d44121ce4e2" Oct 30 13:23:41.039973 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:41.039703 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2cc21d/simulator-single-main-2cc21d-kserve-664597f55c-b4wt2" event={"ID":"780b3ef7-5e06-48be-9ed0-d9223b273061","Type":"ContainerDied","Data":"4cbe3feef4541d6f2ada7191ccc04e1fc7307cf22511811a48ed9e2f087867d9"} Oct 30 13:23:41.049570 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:41.049549 5922 scope.go:122] "RemoveContainer" containerID="3365448cf21e323f1eed4782d1b15abc27a754c427d4926951a27d44121ce4e2" Oct 30 13:23:41.049878 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:23:41.049861 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3365448cf21e323f1eed4782d1b15abc27a754c427d4926951a27d44121ce4e2\": container with ID starting with 3365448cf21e323f1eed4782d1b15abc27a754c427d4926951a27d44121ce4e2 not found: ID does not exist" containerID="3365448cf21e323f1eed4782d1b15abc27a754c427d4926951a27d44121ce4e2" Oct 30 13:23:41.049941 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:41.049886 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3365448cf21e323f1eed4782d1b15abc27a754c427d4926951a27d44121ce4e2"} err="failed to get container status \"3365448cf21e323f1eed4782d1b15abc27a754c427d4926951a27d44121ce4e2\": rpc error: code = NotFound desc = could not find container \"3365448cf21e323f1eed4782d1b15abc27a754c427d4926951a27d44121ce4e2\": container with ID starting with 3365448cf21e323f1eed4782d1b15abc27a754c427d4926951a27d44121ce4e2 not found: ID does not exist" Oct 30 13:23:41.104342 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:41.104321 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-2cc21d/simulator-single-main-2cc21d-kserve-664597f55c-b4wt2"] Oct 30 13:23:41.118330 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:41.118303 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-2cc21d/simulator-single-main-2cc21d-kserve-664597f55c-b4wt2"] Oct 30 13:23:41.642989 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:41.642955 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="780b3ef7-5e06-48be-9ed0-d9223b273061" path="/var/lib/kubelet/pods/780b3ef7-5e06-48be-9ed0-d9223b273061/volumes" Oct 30 13:23:41.704006 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:41.703973 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:41.822338 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:41.822305 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:42.047003 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:42.046969 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw"] Oct 30 13:23:42.047223 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:42.047196 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" podUID="ff585ad9-f06c-453b-b23f-f0519a479d9a" containerName="istio-proxy" containerID="cri-o://c75875f019cce449c7b702fa44c14533bda62857886a6a0cc7234e29d68222c7" gracePeriod=30 Oct 30 13:23:42.122132 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:42.122095 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:46.411891 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:46.411857 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4"] Oct 30 13:23:46.412487 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:46.412464 5922 cpu_manager.go:414] "containerMap: removing container" podUID="880d3e6b-4501-4619-96f3-8fde98d2b520" containerName="revoke-keys" Oct 30 13:23:46.412580 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:46.412489 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="880d3e6b-4501-4619-96f3-8fde98d2b520" containerName="revoke-keys" Oct 30 13:23:46.412580 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:46.412531 5922 cpu_manager.go:414] "containerMap: removing container" podUID="780b3ef7-5e06-48be-9ed0-d9223b273061" containerName="main" Oct 30 13:23:46.412580 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:46.412540 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="780b3ef7-5e06-48be-9ed0-d9223b273061" containerName="main" Oct 30 13:23:46.412787 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:46.412682 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="880d3e6b-4501-4619-96f3-8fde98d2b520" containerName="revoke-keys" Oct 30 13:23:46.412787 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:46.412701 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="780b3ef7-5e06-48be-9ed0-d9223b273061" containerName="main" Oct 30 13:23:46.415030 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:46.415008 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4" Oct 30 13:23:46.442027 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:46.442001 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-068f3f8d-kserve-self-signed-certs\"" Oct 30 13:23:46.444565 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:46.444539 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4"] Oct 30 13:23:46.460173 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:46.460142 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:46.533547 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:46.533520 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6effa00-3db7-4a53-90ac-f387e2dc1d5a-home\") pod \"e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4\" (UID: \"f6effa00-3db7-4a53-90ac-f387e2dc1d5a\") " pod="llm/e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4" Oct 30 13:23:46.533672 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:46.533573 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6effa00-3db7-4a53-90ac-f387e2dc1d5a-tmp-dir\") pod \"e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4\" (UID: \"f6effa00-3db7-4a53-90ac-f387e2dc1d5a\") " pod="llm/e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4" Oct 30 13:23:46.533715 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:46.533664 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6effa00-3db7-4a53-90ac-f387e2dc1d5a-dshm\") pod \"e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4\" (UID: \"f6effa00-3db7-4a53-90ac-f387e2dc1d5a\") " pod="llm/e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4" Oct 30 13:23:46.533715 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:46.533698 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6effa00-3db7-4a53-90ac-f387e2dc1d5a-tls-certs\") pod \"e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4\" (UID: \"f6effa00-3db7-4a53-90ac-f387e2dc1d5a\") " pod="llm/e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4" Oct 30 13:23:46.533829 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:46.533809 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjv26\" (UniqueName: \"kubernetes.io/projected/f6effa00-3db7-4a53-90ac-f387e2dc1d5a-kube-api-access-fjv26\") pod \"e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4\" (UID: \"f6effa00-3db7-4a53-90ac-f387e2dc1d5a\") " pod="llm/e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4" Oct 30 13:23:46.533871 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:46.533858 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6effa00-3db7-4a53-90ac-f387e2dc1d5a-model-cache\") pod \"e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4\" (UID: \"f6effa00-3db7-4a53-90ac-f387e2dc1d5a\") " pod="llm/e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4" Oct 30 13:23:46.635091 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:46.635056 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6effa00-3db7-4a53-90ac-f387e2dc1d5a-model-cache\") pod \"e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4\" (UID: \"f6effa00-3db7-4a53-90ac-f387e2dc1d5a\") " pod="llm/e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4" Oct 30 13:23:46.635259 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:46.635107 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6effa00-3db7-4a53-90ac-f387e2dc1d5a-home\") pod \"e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4\" (UID: \"f6effa00-3db7-4a53-90ac-f387e2dc1d5a\") " pod="llm/e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4" Oct 30 13:23:46.635259 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:46.635142 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6effa00-3db7-4a53-90ac-f387e2dc1d5a-tmp-dir\") pod \"e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4\" (UID: \"f6effa00-3db7-4a53-90ac-f387e2dc1d5a\") " pod="llm/e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4" Oct 30 13:23:46.635259 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:46.635165 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6effa00-3db7-4a53-90ac-f387e2dc1d5a-dshm\") pod \"e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4\" (UID: \"f6effa00-3db7-4a53-90ac-f387e2dc1d5a\") " pod="llm/e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4" Oct 30 13:23:46.635259 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:46.635182 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6effa00-3db7-4a53-90ac-f387e2dc1d5a-tls-certs\") pod \"e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4\" (UID: \"f6effa00-3db7-4a53-90ac-f387e2dc1d5a\") " pod="llm/e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4" Oct 30 13:23:46.635259 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:46.635237 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjv26\" (UniqueName: \"kubernetes.io/projected/f6effa00-3db7-4a53-90ac-f387e2dc1d5a-kube-api-access-fjv26\") pod \"e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4\" (UID: \"f6effa00-3db7-4a53-90ac-f387e2dc1d5a\") " pod="llm/e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4" Oct 30 13:23:46.635579 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:46.635540 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6effa00-3db7-4a53-90ac-f387e2dc1d5a-model-cache\") pod \"e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4\" (UID: \"f6effa00-3db7-4a53-90ac-f387e2dc1d5a\") " pod="llm/e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4" Oct 30 13:23:46.635579 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:46.635556 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6effa00-3db7-4a53-90ac-f387e2dc1d5a-tmp-dir\") pod \"e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4\" (UID: \"f6effa00-3db7-4a53-90ac-f387e2dc1d5a\") " pod="llm/e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4" Oct 30 13:23:46.635798 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:46.635608 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6effa00-3db7-4a53-90ac-f387e2dc1d5a-home\") pod \"e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4\" (UID: \"f6effa00-3db7-4a53-90ac-f387e2dc1d5a\") " pod="llm/e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4" Oct 30 13:23:46.637472 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:46.637448 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6effa00-3db7-4a53-90ac-f387e2dc1d5a-dshm\") pod \"e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4\" (UID: \"f6effa00-3db7-4a53-90ac-f387e2dc1d5a\") " pod="llm/e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4" Oct 30 13:23:46.637794 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:46.637774 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6effa00-3db7-4a53-90ac-f387e2dc1d5a-tls-certs\") pod \"e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4\" (UID: \"f6effa00-3db7-4a53-90ac-f387e2dc1d5a\") " pod="llm/e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4" Oct 30 13:23:46.652696 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:46.652670 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cc21d-69ff6rvfqz" podUID="187b4311-8685-4e4b-b7ab-5affa9de6565" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:23:46.659298 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:46.659271 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjv26\" (UniqueName: \"kubernetes.io/projected/f6effa00-3db7-4a53-90ac-f387e2dc1d5a-kube-api-access-fjv26\") pod \"e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4\" (UID: \"f6effa00-3db7-4a53-90ac-f387e2dc1d5a\") " pod="llm/e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4" Oct 30 13:23:46.726316 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:46.726292 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4" Oct 30 13:23:47.022183 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:47.022110 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4"] Oct 30 13:23:47.025094 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:23:47.025068 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6effa00_3db7_4a53_90ac_f387e2dc1d5a.slice/crio-6477f27917cb54d25ee1bf129ab04c215429dec9ad2f8ccbfa39742db86fd62a WatchSource:0}: Error finding container 6477f27917cb54d25ee1bf129ab04c215429dec9ad2f8ccbfa39742db86fd62a: Status 404 returned error can't find the container with id 6477f27917cb54d25ee1bf129ab04c215429dec9ad2f8ccbfa39742db86fd62a Oct 30 13:23:47.077546 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:47.077518 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4" event={"ID":"f6effa00-3db7-4a53-90ac-f387e2dc1d5a","Type":"ContainerStarted","Data":"6477f27917cb54d25ee1bf129ab04c215429dec9ad2f8ccbfa39742db86fd62a"} Oct 30 13:23:47.279189 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:47.279133 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" Oct 30 13:23:47.445205 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:47.445172 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff585ad9-f06c-453b-b23f-f0519a479d9a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff585ad9-f06c-453b-b23f-f0519a479d9a-istio-podinfo\") pod \"ff585ad9-f06c-453b-b23f-f0519a479d9a\" (UID: \"ff585ad9-f06c-453b-b23f-f0519a479d9a\") " Oct 30 13:23:47.445643 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:47.445215 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff585ad9-f06c-453b-b23f-f0519a479d9a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff585ad9-f06c-453b-b23f-f0519a479d9a-istio-envoy\") pod \"ff585ad9-f06c-453b-b23f-f0519a479d9a\" (UID: \"ff585ad9-f06c-453b-b23f-f0519a479d9a\") " Oct 30 13:23:47.445643 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:47.445244 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff585ad9-f06c-453b-b23f-f0519a479d9a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff585ad9-f06c-453b-b23f-f0519a479d9a-workload-socket\") pod \"ff585ad9-f06c-453b-b23f-f0519a479d9a\" (UID: \"ff585ad9-f06c-453b-b23f-f0519a479d9a\") " Oct 30 13:23:47.445643 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:47.445278 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff585ad9-f06c-453b-b23f-f0519a479d9a-istio-token\" (UniqueName: \"kubernetes.io/projected/ff585ad9-f06c-453b-b23f-f0519a479d9a-istio-token\") pod \"ff585ad9-f06c-453b-b23f-f0519a479d9a\" (UID: \"ff585ad9-f06c-453b-b23f-f0519a479d9a\") " Oct 30 13:23:47.445643 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:47.445305 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff585ad9-f06c-453b-b23f-f0519a479d9a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff585ad9-f06c-453b-b23f-f0519a479d9a-istio-data\") pod \"ff585ad9-f06c-453b-b23f-f0519a479d9a\" (UID: \"ff585ad9-f06c-453b-b23f-f0519a479d9a\") " Oct 30 13:23:47.445643 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:47.445330 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff585ad9-f06c-453b-b23f-f0519a479d9a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff585ad9-f06c-453b-b23f-f0519a479d9a-workload-certs\") pod \"ff585ad9-f06c-453b-b23f-f0519a479d9a\" (UID: \"ff585ad9-f06c-453b-b23f-f0519a479d9a\") " Oct 30 13:23:47.445643 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:47.445366 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff585ad9-f06c-453b-b23f-f0519a479d9a-kube-api-access-xx4q6\" (UniqueName: \"kubernetes.io/projected/ff585ad9-f06c-453b-b23f-f0519a479d9a-kube-api-access-xx4q6\") pod \"ff585ad9-f06c-453b-b23f-f0519a479d9a\" (UID: \"ff585ad9-f06c-453b-b23f-f0519a479d9a\") " Oct 30 13:23:47.445643 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:47.445407 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff585ad9-f06c-453b-b23f-f0519a479d9a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff585ad9-f06c-453b-b23f-f0519a479d9a-credential-socket\") pod \"ff585ad9-f06c-453b-b23f-f0519a479d9a\" (UID: \"ff585ad9-f06c-453b-b23f-f0519a479d9a\") " Oct 30 13:23:47.445643 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:47.445445 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff585ad9-f06c-453b-b23f-f0519a479d9a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff585ad9-f06c-453b-b23f-f0519a479d9a-istiod-ca-cert\") pod \"ff585ad9-f06c-453b-b23f-f0519a479d9a\" (UID: \"ff585ad9-f06c-453b-b23f-f0519a479d9a\") " Oct 30 13:23:47.445643 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:47.445608 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff585ad9-f06c-453b-b23f-f0519a479d9a-workload-socket" pod "ff585ad9-f06c-453b-b23f-f0519a479d9a" (UID: "ff585ad9-f06c-453b-b23f-f0519a479d9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:23:47.446189 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:47.445934 5922 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff585ad9-f06c-453b-b23f-f0519a479d9a-workload-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:23:47.446189 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:47.445948 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff585ad9-f06c-453b-b23f-f0519a479d9a-istiod-ca-cert" pod "ff585ad9-f06c-453b-b23f-f0519a479d9a" (UID: "ff585ad9-f06c-453b-b23f-f0519a479d9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:23:47.446189 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:47.445977 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff585ad9-f06c-453b-b23f-f0519a479d9a-credential-socket" pod "ff585ad9-f06c-453b-b23f-f0519a479d9a" (UID: "ff585ad9-f06c-453b-b23f-f0519a479d9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:23:47.446189 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:47.446146 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff585ad9-f06c-453b-b23f-f0519a479d9a-workload-certs" pod "ff585ad9-f06c-453b-b23f-f0519a479d9a" (UID: "ff585ad9-f06c-453b-b23f-f0519a479d9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:23:47.446189 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:47.446175 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff585ad9-f06c-453b-b23f-f0519a479d9a-istio-data" pod "ff585ad9-f06c-453b-b23f-f0519a479d9a" (UID: "ff585ad9-f06c-453b-b23f-f0519a479d9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:23:47.447692 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:47.447669 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff585ad9-f06c-453b-b23f-f0519a479d9a-istio-token" pod "ff585ad9-f06c-453b-b23f-f0519a479d9a" (UID: "ff585ad9-f06c-453b-b23f-f0519a479d9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:23:47.447815 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:47.447794 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff585ad9-f06c-453b-b23f-f0519a479d9a-istio-podinfo" pod "ff585ad9-f06c-453b-b23f-f0519a479d9a" (UID: "ff585ad9-f06c-453b-b23f-f0519a479d9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:23:47.447907 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:47.447886 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff585ad9-f06c-453b-b23f-f0519a479d9a-istio-envoy" pod "ff585ad9-f06c-453b-b23f-f0519a479d9a" (UID: "ff585ad9-f06c-453b-b23f-f0519a479d9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:23:47.448087 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:47.448067 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff585ad9-f06c-453b-b23f-f0519a479d9a-kube-api-access-xx4q6" pod "ff585ad9-f06c-453b-b23f-f0519a479d9a" (UID: "ff585ad9-f06c-453b-b23f-f0519a479d9a"). InnerVolumeSpecName "kube-api-access-xx4q6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:23:47.547434 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:47.547362 5922 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff585ad9-f06c-453b-b23f-f0519a479d9a-istio-podinfo\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:23:47.547434 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:47.547388 5922 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff585ad9-f06c-453b-b23f-f0519a479d9a-istio-envoy\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:23:47.547434 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:47.547397 5922 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff585ad9-f06c-453b-b23f-f0519a479d9a-istio-token\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:23:47.547434 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:47.547404 5922 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff585ad9-f06c-453b-b23f-f0519a479d9a-istio-data\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:23:47.547434 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:47.547412 5922 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff585ad9-f06c-453b-b23f-f0519a479d9a-workload-certs\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:23:47.547434 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:47.547420 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx4q6\" (UniqueName: \"kubernetes.io/projected/ff585ad9-f06c-453b-b23f-f0519a479d9a-kube-api-access-xx4q6\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:23:47.547434 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:47.547431 5922 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff585ad9-f06c-453b-b23f-f0519a479d9a-credential-socket\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:23:47.547434 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:47.547439 5922 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff585ad9-f06c-453b-b23f-f0519a479d9a-istiod-ca-cert\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:23:47.657893 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:47.657861 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2cc21d-84455c4f45pp8q" podUID="6a949341-b8f4-4e96-81d9-bfe93e05b6e0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:23:48.084097 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:48.084067 5922 generic.go:356] "Generic (PLEG): container finished" podID="ff585ad9-f06c-453b-b23f-f0519a479d9a" containerID="c75875f019cce449c7b702fa44c14533bda62857886a6a0cc7234e29d68222c7" exitCode=0 Oct 30 13:23:48.084289 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:48.084114 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" Oct 30 13:23:48.084289 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:48.084153 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" event={"ID":"ff585ad9-f06c-453b-b23f-f0519a479d9a","Type":"ContainerDied","Data":"c75875f019cce449c7b702fa44c14533bda62857886a6a0cc7234e29d68222c7"} Oct 30 13:23:48.084289 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:48.084210 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw" event={"ID":"ff585ad9-f06c-453b-b23f-f0519a479d9a","Type":"ContainerDied","Data":"5650596105b0f78dd7820051dbbb449210aab6c8c8aee3be96f74f351e598ee6"} Oct 30 13:23:48.084289 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:48.084240 5922 scope.go:122] "RemoveContainer" containerID="c75875f019cce449c7b702fa44c14533bda62857886a6a0cc7234e29d68222c7" Oct 30 13:23:48.085979 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:48.085947 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4" event={"ID":"f6effa00-3db7-4a53-90ac-f387e2dc1d5a","Type":"ContainerStarted","Data":"43aa27bf62da445a6f487cdd4f08a213432e2f7fcadb5a7e40b08ea273d52b0b"} Oct 30 13:23:48.093033 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:48.093016 5922 scope.go:122] "RemoveContainer" containerID="bdfbc1cb5d79eca526371ea3f3a307c24a6399f3dea7ae9c5e0cfaca88efd078" Oct 30 13:23:48.100702 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:48.100681 5922 scope.go:122] "RemoveContainer" containerID="c75875f019cce449c7b702fa44c14533bda62857886a6a0cc7234e29d68222c7" Oct 30 13:23:48.100970 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:23:48.100952 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c75875f019cce449c7b702fa44c14533bda62857886a6a0cc7234e29d68222c7\": container with ID starting with c75875f019cce449c7b702fa44c14533bda62857886a6a0cc7234e29d68222c7 not found: ID does not exist" containerID="c75875f019cce449c7b702fa44c14533bda62857886a6a0cc7234e29d68222c7" Oct 30 13:23:48.101053 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:48.100979 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c75875f019cce449c7b702fa44c14533bda62857886a6a0cc7234e29d68222c7"} err="failed to get container status \"c75875f019cce449c7b702fa44c14533bda62857886a6a0cc7234e29d68222c7\": rpc error: code = NotFound desc = could not find container \"c75875f019cce449c7b702fa44c14533bda62857886a6a0cc7234e29d68222c7\": container with ID starting with c75875f019cce449c7b702fa44c14533bda62857886a6a0cc7234e29d68222c7 not found: ID does not exist" Oct 30 13:23:48.101053 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:48.101004 5922 scope.go:122] "RemoveContainer" containerID="bdfbc1cb5d79eca526371ea3f3a307c24a6399f3dea7ae9c5e0cfaca88efd078" Oct 30 13:23:48.101230 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:23:48.101212 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdfbc1cb5d79eca526371ea3f3a307c24a6399f3dea7ae9c5e0cfaca88efd078\": container with ID starting with bdfbc1cb5d79eca526371ea3f3a307c24a6399f3dea7ae9c5e0cfaca88efd078 not found: ID does not exist" containerID="bdfbc1cb5d79eca526371ea3f3a307c24a6399f3dea7ae9c5e0cfaca88efd078" Oct 30 13:23:48.101267 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:48.101235 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdfbc1cb5d79eca526371ea3f3a307c24a6399f3dea7ae9c5e0cfaca88efd078"} err="failed to get container status \"bdfbc1cb5d79eca526371ea3f3a307c24a6399f3dea7ae9c5e0cfaca88efd078\": rpc error: code = NotFound desc = could not find container \"bdfbc1cb5d79eca526371ea3f3a307c24a6399f3dea7ae9c5e0cfaca88efd078\": container with ID starting with bdfbc1cb5d79eca526371ea3f3a307c24a6399f3dea7ae9c5e0cfaca88efd078 not found: ID does not exist" Oct 30 13:23:48.138215 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:48.138194 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw"] Oct 30 13:23:48.150118 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:48.150093 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2cc21d-openshift-default-765cffb457wmjcw"] Oct 30 13:23:48.184627 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:48.184571 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4" podStartSLOduration=2.184560675 podStartE2EDuration="2.184560675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:23:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:23:48.18319929 +0000 UTC m=+1884.652615036" watchObservedRunningTime="2026-10-30 13:23:48.184560675 +0000 UTC m=+1884.653976465" Oct 30 13:23:48.640624 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:48.640570 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:23:48.727130 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:48.727093 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4" Oct 30 13:23:49.652633 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:49.651717 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff585ad9-f06c-453b-b23f-f0519a479d9a" path="/var/lib/kubelet/pods/ff585ad9-f06c-453b-b23f-f0519a479d9a/volumes" Oct 30 13:23:51.290759 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:51.290703 5922 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5x6r5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:23:51.291504 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:51.291453 5922 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5x6r5" podUID="3584258d-ca3c-4574-a39e-a70cc78d0ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:23:53.551823 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:23:53.551784 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:23:56.396667 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:56.396639 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2cc21d-84455c4f45pp8q" Oct 30 13:23:56.425491 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:56.425465 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cc21d-69ff6rvfqz" Oct 30 13:23:56.478922 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:56.478894 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/187b4311-8685-4e4b-b7ab-5affa9de6565-kube-api-access-csgcg\" (UniqueName: \"kubernetes.io/projected/187b4311-8685-4e4b-b7ab-5affa9de6565-kube-api-access-csgcg\") pod \"187b4311-8685-4e4b-b7ab-5affa9de6565\" (UID: \"187b4311-8685-4e4b-b7ab-5affa9de6565\") " Oct 30 13:23:56.479045 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:56.478930 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a949341-b8f4-4e96-81d9-bfe93e05b6e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a949341-b8f4-4e96-81d9-bfe93e05b6e0-plugins-config-volume\") pod \"6a949341-b8f4-4e96-81d9-bfe93e05b6e0\" (UID: \"6a949341-b8f4-4e96-81d9-bfe93e05b6e0\") " Oct 30 13:23:56.479045 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:56.478953 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/187b4311-8685-4e4b-b7ab-5affa9de6565-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/187b4311-8685-4e4b-b7ab-5affa9de6565-plugins-config-volume\") pod \"187b4311-8685-4e4b-b7ab-5affa9de6565\" (UID: \"187b4311-8685-4e4b-b7ab-5affa9de6565\") " Oct 30 13:23:56.479137 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:56.479114 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a949341-b8f4-4e96-81d9-bfe93e05b6e0-kube-api-access-chv27\" (UniqueName: \"kubernetes.io/projected/6a949341-b8f4-4e96-81d9-bfe93e05b6e0-kube-api-access-chv27\") pod \"6a949341-b8f4-4e96-81d9-bfe93e05b6e0\" (UID: \"6a949341-b8f4-4e96-81d9-bfe93e05b6e0\") " Oct 30 13:23:56.479276 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:56.479252 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a949341-b8f4-4e96-81d9-bfe93e05b6e0-plugins-config-volume" pod "6a949341-b8f4-4e96-81d9-bfe93e05b6e0" (UID: "6a949341-b8f4-4e96-81d9-bfe93e05b6e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:23:56.479372 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:56.479351 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/187b4311-8685-4e4b-b7ab-5affa9de6565-plugins-config-volume" pod "187b4311-8685-4e4b-b7ab-5affa9de6565" (UID: "187b4311-8685-4e4b-b7ab-5affa9de6565"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:23:56.479450 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:56.479433 5922 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a949341-b8f4-4e96-81d9-bfe93e05b6e0-plugins-config-volume\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:23:56.479496 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:56.479453 5922 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/187b4311-8685-4e4b-b7ab-5affa9de6565-plugins-config-volume\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:23:56.481361 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:56.481337 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a949341-b8f4-4e96-81d9-bfe93e05b6e0-kube-api-access-chv27" pod "6a949341-b8f4-4e96-81d9-bfe93e05b6e0" (UID: "6a949341-b8f4-4e96-81d9-bfe93e05b6e0"). InnerVolumeSpecName "kube-api-access-chv27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:23:56.481361 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:56.481348 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/187b4311-8685-4e4b-b7ab-5affa9de6565-kube-api-access-csgcg" pod "187b4311-8685-4e4b-b7ab-5affa9de6565" (UID: "187b4311-8685-4e4b-b7ab-5affa9de6565"). InnerVolumeSpecName "kube-api-access-csgcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:23:56.579921 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:56.579897 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chv27\" (UniqueName: \"kubernetes.io/projected/6a949341-b8f4-4e96-81d9-bfe93e05b6e0-kube-api-access-chv27\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:23:56.579921 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:56.579920 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csgcg\" (UniqueName: \"kubernetes.io/projected/187b4311-8685-4e4b-b7ab-5affa9de6565-kube-api-access-csgcg\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:23:56.727069 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:56.727033 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4" Oct 30 13:23:56.739671 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:56.739650 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4" Oct 30 13:23:57.152499 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:57.152467 5922 generic.go:356] "Generic (PLEG): container finished" podID="187b4311-8685-4e4b-b7ab-5affa9de6565" containerID="306ef86256481ab9d600b69582380db42ac68cf326762d14cb09513c56d40c06" exitCode=0 Oct 30 13:23:57.152640 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:57.152510 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cc21d-69ff6rvfqz" Oct 30 13:23:57.152640 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:57.152544 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cc21d-69ff6rvfqz" event={"ID":"187b4311-8685-4e4b-b7ab-5affa9de6565","Type":"ContainerDied","Data":"306ef86256481ab9d600b69582380db42ac68cf326762d14cb09513c56d40c06"} Oct 30 13:23:57.152640 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:57.152597 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cc21d-69ff6rvfqz" event={"ID":"187b4311-8685-4e4b-b7ab-5affa9de6565","Type":"ContainerDied","Data":"0ebc631047b5a926d4ad83dd36386102b056c3a1b9a6b7926d86555319a4ac0a"} Oct 30 13:23:57.152640 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:57.152637 5922 scope.go:122] "RemoveContainer" containerID="306ef86256481ab9d600b69582380db42ac68cf326762d14cb09513c56d40c06" Oct 30 13:23:57.154045 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:57.154025 5922 generic.go:356] "Generic (PLEG): container finished" podID="6a949341-b8f4-4e96-81d9-bfe93e05b6e0" containerID="a555581d54766813e10e72d3ab33c3624efc4db03f5fac7382fa27acf8950c76" exitCode=0 Oct 30 13:23:57.154164 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:57.154089 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2cc21d-84455c4f45pp8q" Oct 30 13:23:57.154164 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:57.154117 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2cc21d-84455c4f45pp8q" event={"ID":"6a949341-b8f4-4e96-81d9-bfe93e05b6e0","Type":"ContainerDied","Data":"a555581d54766813e10e72d3ab33c3624efc4db03f5fac7382fa27acf8950c76"} Oct 30 13:23:57.154296 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:57.154177 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2cc21d-84455c4f45pp8q" event={"ID":"6a949341-b8f4-4e96-81d9-bfe93e05b6e0","Type":"ContainerDied","Data":"88c6ffd60654f9bea9035e976661f70785dc3d66f0d9827132b6a83b49d52c89"} Oct 30 13:23:57.162876 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:57.162688 5922 scope.go:122] "RemoveContainer" containerID="306ef86256481ab9d600b69582380db42ac68cf326762d14cb09513c56d40c06" Oct 30 13:23:57.162986 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:23:57.162952 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"306ef86256481ab9d600b69582380db42ac68cf326762d14cb09513c56d40c06\": container with ID starting with 306ef86256481ab9d600b69582380db42ac68cf326762d14cb09513c56d40c06 not found: ID does not exist" containerID="306ef86256481ab9d600b69582380db42ac68cf326762d14cb09513c56d40c06" Oct 30 13:23:57.163064 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:57.162986 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"306ef86256481ab9d600b69582380db42ac68cf326762d14cb09513c56d40c06"} err="failed to get container status \"306ef86256481ab9d600b69582380db42ac68cf326762d14cb09513c56d40c06\": rpc error: code = NotFound desc = could not find container \"306ef86256481ab9d600b69582380db42ac68cf326762d14cb09513c56d40c06\": container with ID starting with 306ef86256481ab9d600b69582380db42ac68cf326762d14cb09513c56d40c06 not found: ID does not exist" Oct 30 13:23:57.163064 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:57.163009 5922 scope.go:122] "RemoveContainer" containerID="a555581d54766813e10e72d3ab33c3624efc4db03f5fac7382fa27acf8950c76" Oct 30 13:23:57.168173 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:57.168153 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4" Oct 30 13:23:57.171176 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:57.171161 5922 scope.go:122] "RemoveContainer" containerID="a555581d54766813e10e72d3ab33c3624efc4db03f5fac7382fa27acf8950c76" Oct 30 13:23:57.171419 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:23:57.171401 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a555581d54766813e10e72d3ab33c3624efc4db03f5fac7382fa27acf8950c76\": container with ID starting with a555581d54766813e10e72d3ab33c3624efc4db03f5fac7382fa27acf8950c76 not found: ID does not exist" containerID="a555581d54766813e10e72d3ab33c3624efc4db03f5fac7382fa27acf8950c76" Oct 30 13:23:57.171487 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:57.171443 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a555581d54766813e10e72d3ab33c3624efc4db03f5fac7382fa27acf8950c76"} err="failed to get container status \"a555581d54766813e10e72d3ab33c3624efc4db03f5fac7382fa27acf8950c76\": rpc error: code = NotFound desc = could not find container \"a555581d54766813e10e72d3ab33c3624efc4db03f5fac7382fa27acf8950c76\": container with ID starting with a555581d54766813e10e72d3ab33c3624efc4db03f5fac7382fa27acf8950c76 not found: ID does not exist" Oct 30 13:23:57.205596 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:57.205575 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2cc21d-84455c4f45pp8q"] Oct 30 13:23:57.210710 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:57.210692 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2cc21d-84455c4f45pp8q"] Oct 30 13:23:57.246114 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:57.246093 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cc21d-69ff6rvfqz"] Oct 30 13:23:57.251745 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:57.251725 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2cc21d-69ff6rvfqz"] Oct 30 13:23:57.643952 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:57.643915 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="187b4311-8685-4e4b-b7ab-5affa9de6565" path="/var/lib/kubelet/pods/187b4311-8685-4e4b-b7ab-5affa9de6565/volumes" Oct 30 13:23:57.644302 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:23:57.644283 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a949341-b8f4-4e96-81d9-bfe93e05b6e0" path="/var/lib/kubelet/pods/6a949341-b8f4-4e96-81d9-bfe93e05b6e0/volumes" Oct 30 13:24:03.418793 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:03.418758 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b"] Oct 30 13:24:03.419169 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:03.419136 5922 cpu_manager.go:414] "containerMap: removing container" podUID="6a949341-b8f4-4e96-81d9-bfe93e05b6e0" containerName="payload-processing" Oct 30 13:24:03.419169 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:03.419146 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a949341-b8f4-4e96-81d9-bfe93e05b6e0" containerName="payload-processing" Oct 30 13:24:03.419169 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:03.419161 5922 cpu_manager.go:414] "containerMap: removing container" podUID="ff585ad9-f06c-453b-b23f-f0519a479d9a" containerName="istio-proxy" Oct 30 13:24:03.419169 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:03.419166 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff585ad9-f06c-453b-b23f-f0519a479d9a" containerName="istio-proxy" Oct 30 13:24:03.419304 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:03.419183 5922 cpu_manager.go:414] "containerMap: removing container" podUID="ff585ad9-f06c-453b-b23f-f0519a479d9a" containerName="istio-proxy" Oct 30 13:24:03.419304 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:03.419189 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff585ad9-f06c-453b-b23f-f0519a479d9a" containerName="istio-proxy" Oct 30 13:24:03.419304 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:03.419198 5922 cpu_manager.go:414] "containerMap: removing container" podUID="187b4311-8685-4e4b-b7ab-5affa9de6565" containerName="payload-pre-processing" Oct 30 13:24:03.419304 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:03.419203 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="187b4311-8685-4e4b-b7ab-5affa9de6565" containerName="payload-pre-processing" Oct 30 13:24:03.419304 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:03.419265 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff585ad9-f06c-453b-b23f-f0519a479d9a" containerName="istio-proxy" Oct 30 13:24:03.419304 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:03.419273 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="187b4311-8685-4e4b-b7ab-5affa9de6565" containerName="payload-pre-processing" Oct 30 13:24:03.419304 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:03.419284 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a949341-b8f4-4e96-81d9-bfe93e05b6e0" containerName="payload-processing" Oct 30 13:24:03.424059 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:03.424040 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b" Oct 30 13:24:03.430851 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:03.430812 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:03.454650 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:03.454624 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-068f3f8d-kserve-self-signed-certs\"" Oct 30 13:24:03.457083 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:03.457054 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b"] Oct 30 13:24:03.543860 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:03.543830 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abed9e1f-8848-47f4-81e0-929784183401-home\") pod \"e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b\" (UID: \"abed9e1f-8848-47f4-81e0-929784183401\") " pod="llm/e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b" Oct 30 13:24:03.543974 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:03.543879 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8rgm\" (UniqueName: \"kubernetes.io/projected/abed9e1f-8848-47f4-81e0-929784183401-kube-api-access-h8rgm\") pod \"e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b\" (UID: \"abed9e1f-8848-47f4-81e0-929784183401\") " pod="llm/e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b" Oct 30 13:24:03.544021 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:03.543994 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abed9e1f-8848-47f4-81e0-929784183401-model-cache\") pod \"e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b\" (UID: \"abed9e1f-8848-47f4-81e0-929784183401\") " pod="llm/e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b" Oct 30 13:24:03.544057 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:03.544020 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abed9e1f-8848-47f4-81e0-929784183401-tls-certs\") pod \"e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b\" (UID: \"abed9e1f-8848-47f4-81e0-929784183401\") " pod="llm/e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b" Oct 30 13:24:03.544125 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:03.544103 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abed9e1f-8848-47f4-81e0-929784183401-tmp-dir\") pod \"e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b\" (UID: \"abed9e1f-8848-47f4-81e0-929784183401\") " pod="llm/e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b" Oct 30 13:24:03.544169 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:03.544158 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abed9e1f-8848-47f4-81e0-929784183401-dshm\") pod \"e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b\" (UID: \"abed9e1f-8848-47f4-81e0-929784183401\") " pod="llm/e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b" Oct 30 13:24:03.558591 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:03.558566 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:03.646476 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:03.646442 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8rgm\" (UniqueName: \"kubernetes.io/projected/abed9e1f-8848-47f4-81e0-929784183401-kube-api-access-h8rgm\") pod \"e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b\" (UID: \"abed9e1f-8848-47f4-81e0-929784183401\") " pod="llm/e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b" Oct 30 13:24:03.646671 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:03.646532 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abed9e1f-8848-47f4-81e0-929784183401-model-cache\") pod \"e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b\" (UID: \"abed9e1f-8848-47f4-81e0-929784183401\") " pod="llm/e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b" Oct 30 13:24:03.646671 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:03.646552 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abed9e1f-8848-47f4-81e0-929784183401-tls-certs\") pod \"e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b\" (UID: \"abed9e1f-8848-47f4-81e0-929784183401\") " pod="llm/e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b" Oct 30 13:24:03.646821 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:03.646782 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abed9e1f-8848-47f4-81e0-929784183401-tmp-dir\") pod \"e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b\" (UID: \"abed9e1f-8848-47f4-81e0-929784183401\") " pod="llm/e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b" Oct 30 13:24:03.646896 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:03.646829 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abed9e1f-8848-47f4-81e0-929784183401-dshm\") pod \"e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b\" (UID: \"abed9e1f-8848-47f4-81e0-929784183401\") " pod="llm/e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b" Oct 30 13:24:03.646956 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:03.646907 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abed9e1f-8848-47f4-81e0-929784183401-home\") pod \"e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b\" (UID: \"abed9e1f-8848-47f4-81e0-929784183401\") " pod="llm/e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b" Oct 30 13:24:03.647065 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:03.647018 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abed9e1f-8848-47f4-81e0-929784183401-model-cache\") pod \"e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b\" (UID: \"abed9e1f-8848-47f4-81e0-929784183401\") " pod="llm/e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b" Oct 30 13:24:03.647266 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:03.647238 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abed9e1f-8848-47f4-81e0-929784183401-home\") pod \"e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b\" (UID: \"abed9e1f-8848-47f4-81e0-929784183401\") " pod="llm/e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b" Oct 30 13:24:03.647369 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:03.647298 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abed9e1f-8848-47f4-81e0-929784183401-tmp-dir\") pod \"e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b\" (UID: \"abed9e1f-8848-47f4-81e0-929784183401\") " pod="llm/e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b" Oct 30 13:24:03.649080 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:03.649055 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abed9e1f-8848-47f4-81e0-929784183401-tls-certs\") pod \"e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b\" (UID: \"abed9e1f-8848-47f4-81e0-929784183401\") " pod="llm/e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b" Oct 30 13:24:03.649217 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:03.649198 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abed9e1f-8848-47f4-81e0-929784183401-dshm\") pod \"e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b\" (UID: \"abed9e1f-8848-47f4-81e0-929784183401\") " pod="llm/e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b" Oct 30 13:24:03.671035 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:03.670975 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8rgm\" (UniqueName: \"kubernetes.io/projected/abed9e1f-8848-47f4-81e0-929784183401-kube-api-access-h8rgm\") pod \"e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b\" (UID: \"abed9e1f-8848-47f4-81e0-929784183401\") " pod="llm/e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b" Oct 30 13:24:03.675733 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:03.675702 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:03.736727 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:03.736699 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b" Oct 30 13:24:04.036871 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:04.036836 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b"] Oct 30 13:24:04.039718 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:24:04.039690 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabed9e1f_8848_47f4_81e0_929784183401.slice/crio-c4c111370cbbdacfa98993ae84f0f1bd955b10b7663ff1090cd17b7020f6ff30 WatchSource:0}: Error finding container c4c111370cbbdacfa98993ae84f0f1bd955b10b7663ff1090cd17b7020f6ff30: Status 404 returned error can't find the container with id c4c111370cbbdacfa98993ae84f0f1bd955b10b7663ff1090cd17b7020f6ff30 Oct 30 13:24:04.192745 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:04.192710 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b" event={"ID":"abed9e1f-8848-47f4-81e0-929784183401","Type":"ContainerStarted","Data":"3048156983cd077ad363ffb81ee5607a58b70566cb0b626fa26f672b45f4289e"} Oct 30 13:24:04.192745 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:04.192751 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b" event={"ID":"abed9e1f-8848-47f4-81e0-929784183401","Type":"ContainerStarted","Data":"c4c111370cbbdacfa98993ae84f0f1bd955b10b7663ff1090cd17b7020f6ff30"} Oct 30 13:24:04.235943 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:04.235898 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b" podStartSLOduration=1.235886401 podStartE2EDuration="1.235886401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:24:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:24:04.234960252 +0000 UTC m=+1900.704376015" watchObservedRunningTime="2026-10-30 13:24:04.235886401 +0000 UTC m=+1900.705302165" Oct 30 13:24:04.737548 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:04.737514 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b" Oct 30 13:24:05.657599 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:05.657563 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:24:09.752558 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:09.752516 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:13.737668 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:13.737637 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b" Oct 30 13:24:13.750028 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:13.750005 5922 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b" Oct 30 13:24:14.274029 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:14.273995 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b" Oct 30 13:24:20.901767 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:20.901732 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:24:23.558921 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:24:23.558886 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:24:33.154940 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:33.154894 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4"] Oct 30 13:24:33.211300 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:33.211267 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:34.912154 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:34.912113 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:35.133457 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:35.133420 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:24:38.659604 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:38.659570 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b"] Oct 30 13:24:38.702814 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:38.702781 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:41.721189 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:41.721151 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:24:42.037683 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:42.037298 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95d7722d-6878x"] Oct 30 13:24:42.049714 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:42.047004 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95d7722d-6878x"] Oct 30 13:24:43.644768 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:43.644735 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1583f6bb-57f7-4fec-b7d1-6a190e6070b4" path="/var/lib/kubelet/pods/1583f6bb-57f7-4fec-b7d1-6a190e6070b4/volumes" Oct 30 13:24:48.175805 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:48.175749 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4" podUID="f6effa00-3db7-4a53-90ac-f387e2dc1d5a" containerName="main" containerID="cri-o://43aa27bf62da445a6f487cdd4f08a213432e2f7fcadb5a7e40b08ea273d52b0b" gracePeriod=45 Oct 30 13:24:48.422228 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:48.422199 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4" Oct 30 13:24:48.474877 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:48.474843 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6effa00-3db7-4a53-90ac-f387e2dc1d5a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6effa00-3db7-4a53-90ac-f387e2dc1d5a-model-cache\") pod \"f6effa00-3db7-4a53-90ac-f387e2dc1d5a\" (UID: \"f6effa00-3db7-4a53-90ac-f387e2dc1d5a\") " Oct 30 13:24:48.475030 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:48.474943 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6effa00-3db7-4a53-90ac-f387e2dc1d5a-kube-api-access-fjv26\" (UniqueName: \"kubernetes.io/projected/f6effa00-3db7-4a53-90ac-f387e2dc1d5a-kube-api-access-fjv26\") pod \"f6effa00-3db7-4a53-90ac-f387e2dc1d5a\" (UID: \"f6effa00-3db7-4a53-90ac-f387e2dc1d5a\") " Oct 30 13:24:48.475030 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:48.475006 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6effa00-3db7-4a53-90ac-f387e2dc1d5a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6effa00-3db7-4a53-90ac-f387e2dc1d5a-dshm\") pod \"f6effa00-3db7-4a53-90ac-f387e2dc1d5a\" (UID: \"f6effa00-3db7-4a53-90ac-f387e2dc1d5a\") " Oct 30 13:24:48.475030 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:48.475030 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6effa00-3db7-4a53-90ac-f387e2dc1d5a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6effa00-3db7-4a53-90ac-f387e2dc1d5a-tmp-dir\") pod \"f6effa00-3db7-4a53-90ac-f387e2dc1d5a\" (UID: \"f6effa00-3db7-4a53-90ac-f387e2dc1d5a\") " Oct 30 13:24:48.475217 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:48.475055 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6effa00-3db7-4a53-90ac-f387e2dc1d5a-home\" (UniqueName: \"kubernetes.io/empty-dir/f6effa00-3db7-4a53-90ac-f387e2dc1d5a-home\") pod \"f6effa00-3db7-4a53-90ac-f387e2dc1d5a\" (UID: \"f6effa00-3db7-4a53-90ac-f387e2dc1d5a\") " Oct 30 13:24:48.475217 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:48.475078 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6effa00-3db7-4a53-90ac-f387e2dc1d5a-model-cache" pod "f6effa00-3db7-4a53-90ac-f387e2dc1d5a" (UID: "f6effa00-3db7-4a53-90ac-f387e2dc1d5a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:48.475217 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:48.475094 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6effa00-3db7-4a53-90ac-f387e2dc1d5a-tls-certs\" (UniqueName: \"kubernetes.io/secret/f6effa00-3db7-4a53-90ac-f387e2dc1d5a-tls-certs\") pod \"f6effa00-3db7-4a53-90ac-f387e2dc1d5a\" (UID: \"f6effa00-3db7-4a53-90ac-f387e2dc1d5a\") " Oct 30 13:24:48.475380 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:48.475333 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6effa00-3db7-4a53-90ac-f387e2dc1d5a-tmp-dir" pod "f6effa00-3db7-4a53-90ac-f387e2dc1d5a" (UID: "f6effa00-3db7-4a53-90ac-f387e2dc1d5a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:48.475380 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:48.475348 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6effa00-3db7-4a53-90ac-f387e2dc1d5a-home" pod "f6effa00-3db7-4a53-90ac-f387e2dc1d5a" (UID: "f6effa00-3db7-4a53-90ac-f387e2dc1d5a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:48.475380 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:48.475355 5922 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6effa00-3db7-4a53-90ac-f387e2dc1d5a-model-cache\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:24:48.477241 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:48.477214 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6effa00-3db7-4a53-90ac-f387e2dc1d5a-dshm" pod "f6effa00-3db7-4a53-90ac-f387e2dc1d5a" (UID: "f6effa00-3db7-4a53-90ac-f387e2dc1d5a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:48.477345 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:48.477272 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6effa00-3db7-4a53-90ac-f387e2dc1d5a-tls-certs" pod "f6effa00-3db7-4a53-90ac-f387e2dc1d5a" (UID: "f6effa00-3db7-4a53-90ac-f387e2dc1d5a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:24:48.477390 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:48.477369 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6effa00-3db7-4a53-90ac-f387e2dc1d5a-kube-api-access-fjv26" pod "f6effa00-3db7-4a53-90ac-f387e2dc1d5a" (UID: "f6effa00-3db7-4a53-90ac-f387e2dc1d5a"). InnerVolumeSpecName "kube-api-access-fjv26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:24:48.490172 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:48.490146 5922 generic.go:356] "Generic (PLEG): container finished" podID="f6effa00-3db7-4a53-90ac-f387e2dc1d5a" containerID="43aa27bf62da445a6f487cdd4f08a213432e2f7fcadb5a7e40b08ea273d52b0b" exitCode=0 Oct 30 13:24:48.490290 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:48.490196 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4" Oct 30 13:24:48.490290 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:48.490213 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4" event={"ID":"f6effa00-3db7-4a53-90ac-f387e2dc1d5a","Type":"ContainerDied","Data":"43aa27bf62da445a6f487cdd4f08a213432e2f7fcadb5a7e40b08ea273d52b0b"} Oct 30 13:24:48.490290 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:48.490256 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4" event={"ID":"f6effa00-3db7-4a53-90ac-f387e2dc1d5a","Type":"ContainerDied","Data":"6477f27917cb54d25ee1bf129ab04c215429dec9ad2f8ccbfa39742db86fd62a"} Oct 30 13:24:48.490290 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:48.490279 5922 scope.go:122] "RemoveContainer" containerID="43aa27bf62da445a6f487cdd4f08a213432e2f7fcadb5a7e40b08ea273d52b0b" Oct 30 13:24:48.498777 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:48.498760 5922 scope.go:122] "RemoveContainer" containerID="43aa27bf62da445a6f487cdd4f08a213432e2f7fcadb5a7e40b08ea273d52b0b" Oct 30 13:24:48.499020 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:24:48.498997 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43aa27bf62da445a6f487cdd4f08a213432e2f7fcadb5a7e40b08ea273d52b0b\": container with ID starting with 43aa27bf62da445a6f487cdd4f08a213432e2f7fcadb5a7e40b08ea273d52b0b not found: ID does not exist" containerID="43aa27bf62da445a6f487cdd4f08a213432e2f7fcadb5a7e40b08ea273d52b0b" Oct 30 13:24:48.499155 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:48.499024 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43aa27bf62da445a6f487cdd4f08a213432e2f7fcadb5a7e40b08ea273d52b0b"} err="failed to get container status \"43aa27bf62da445a6f487cdd4f08a213432e2f7fcadb5a7e40b08ea273d52b0b\": rpc error: code = NotFound desc = could not find container \"43aa27bf62da445a6f487cdd4f08a213432e2f7fcadb5a7e40b08ea273d52b0b\": container with ID starting with 43aa27bf62da445a6f487cdd4f08a213432e2f7fcadb5a7e40b08ea273d52b0b not found: ID does not exist" Oct 30 13:24:48.542219 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:48.542193 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4"] Oct 30 13:24:48.547561 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:48.547538 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-068f3f8d-kserve-955f7f565-mfdk4"] Oct 30 13:24:48.576650 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:48.576631 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjv26\" (UniqueName: \"kubernetes.io/projected/f6effa00-3db7-4a53-90ac-f387e2dc1d5a-kube-api-access-fjv26\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:24:48.576650 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:48.576649 5922 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6effa00-3db7-4a53-90ac-f387e2dc1d5a-dshm\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:24:48.576756 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:48.576659 5922 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6effa00-3db7-4a53-90ac-f387e2dc1d5a-tmp-dir\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:24:48.576756 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:48.576668 5922 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6effa00-3db7-4a53-90ac-f387e2dc1d5a-home\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:24:48.576756 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:48.576675 5922 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6effa00-3db7-4a53-90ac-f387e2dc1d5a-tls-certs\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:24:49.643459 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:49.643425 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6effa00-3db7-4a53-90ac-f387e2dc1d5a" path="/var/lib/kubelet/pods/f6effa00-3db7-4a53-90ac-f387e2dc1d5a/volumes" Oct 30 13:24:53.552265 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:24:53.552226 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:24:53.617250 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:53.617218 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:24:53.682225 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:53.682195 5922 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b" podUID="abed9e1f-8848-47f4-81e0-929784183401" containerName="main" containerID="cri-o://3048156983cd077ad363ffb81ee5607a58b70566cb0b626fa26f672b45f4289e" gracePeriod=45 Oct 30 13:24:53.933261 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:53.933236 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b" Oct 30 13:24:54.023639 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:54.023590 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abed9e1f-8848-47f4-81e0-929784183401-dshm\" (UniqueName: \"kubernetes.io/empty-dir/abed9e1f-8848-47f4-81e0-929784183401-dshm\") pod \"abed9e1f-8848-47f4-81e0-929784183401\" (UID: \"abed9e1f-8848-47f4-81e0-929784183401\") " Oct 30 13:24:54.023839 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:54.023660 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abed9e1f-8848-47f4-81e0-929784183401-kube-api-access-h8rgm\" (UniqueName: \"kubernetes.io/projected/abed9e1f-8848-47f4-81e0-929784183401-kube-api-access-h8rgm\") pod \"abed9e1f-8848-47f4-81e0-929784183401\" (UID: \"abed9e1f-8848-47f4-81e0-929784183401\") " Oct 30 13:24:54.023839 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:54.023701 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abed9e1f-8848-47f4-81e0-929784183401-home\" (UniqueName: \"kubernetes.io/empty-dir/abed9e1f-8848-47f4-81e0-929784183401-home\") pod \"abed9e1f-8848-47f4-81e0-929784183401\" (UID: \"abed9e1f-8848-47f4-81e0-929784183401\") " Oct 30 13:24:54.023839 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:54.023731 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abed9e1f-8848-47f4-81e0-929784183401-tls-certs\" (UniqueName: \"kubernetes.io/secret/abed9e1f-8848-47f4-81e0-929784183401-tls-certs\") pod \"abed9e1f-8848-47f4-81e0-929784183401\" (UID: \"abed9e1f-8848-47f4-81e0-929784183401\") " Oct 30 13:24:54.023839 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:54.023767 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abed9e1f-8848-47f4-81e0-929784183401-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abed9e1f-8848-47f4-81e0-929784183401-model-cache\") pod \"abed9e1f-8848-47f4-81e0-929784183401\" (UID: \"abed9e1f-8848-47f4-81e0-929784183401\") " Oct 30 13:24:54.023839 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:54.023792 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abed9e1f-8848-47f4-81e0-929784183401-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abed9e1f-8848-47f4-81e0-929784183401-tmp-dir\") pod \"abed9e1f-8848-47f4-81e0-929784183401\" (UID: \"abed9e1f-8848-47f4-81e0-929784183401\") " Oct 30 13:24:54.024293 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:54.024248 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abed9e1f-8848-47f4-81e0-929784183401-home" pod "abed9e1f-8848-47f4-81e0-929784183401" (UID: "abed9e1f-8848-47f4-81e0-929784183401"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:54.024453 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:54.024382 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abed9e1f-8848-47f4-81e0-929784183401-model-cache" pod "abed9e1f-8848-47f4-81e0-929784183401" (UID: "abed9e1f-8848-47f4-81e0-929784183401"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:54.024541 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:54.024520 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abed9e1f-8848-47f4-81e0-929784183401-tmp-dir" pod "abed9e1f-8848-47f4-81e0-929784183401" (UID: "abed9e1f-8848-47f4-81e0-929784183401"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:54.026234 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:54.026209 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abed9e1f-8848-47f4-81e0-929784183401-dshm" pod "abed9e1f-8848-47f4-81e0-929784183401" (UID: "abed9e1f-8848-47f4-81e0-929784183401"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:24:54.026330 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:54.026260 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abed9e1f-8848-47f4-81e0-929784183401-tls-certs" pod "abed9e1f-8848-47f4-81e0-929784183401" (UID: "abed9e1f-8848-47f4-81e0-929784183401"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:24:54.026537 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:54.026517 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abed9e1f-8848-47f4-81e0-929784183401-kube-api-access-h8rgm" pod "abed9e1f-8848-47f4-81e0-929784183401" (UID: "abed9e1f-8848-47f4-81e0-929784183401"). InnerVolumeSpecName "kube-api-access-h8rgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:24:54.125689 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:54.125187 5922 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abed9e1f-8848-47f4-81e0-929784183401-dshm\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:24:54.125689 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:54.125225 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8rgm\" (UniqueName: \"kubernetes.io/projected/abed9e1f-8848-47f4-81e0-929784183401-kube-api-access-h8rgm\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:24:54.125689 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:54.125245 5922 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abed9e1f-8848-47f4-81e0-929784183401-home\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:24:54.125689 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:54.125258 5922 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abed9e1f-8848-47f4-81e0-929784183401-tls-certs\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:24:54.125689 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:54.125273 5922 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abed9e1f-8848-47f4-81e0-929784183401-model-cache\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:24:54.125689 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:54.125287 5922 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abed9e1f-8848-47f4-81e0-929784183401-tmp-dir\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:24:54.536939 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:54.536481 5922 generic.go:356] "Generic (PLEG): container finished" podID="abed9e1f-8848-47f4-81e0-929784183401" containerID="3048156983cd077ad363ffb81ee5607a58b70566cb0b626fa26f672b45f4289e" exitCode=0 Oct 30 13:24:54.536939 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:54.536538 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b" event={"ID":"abed9e1f-8848-47f4-81e0-929784183401","Type":"ContainerDied","Data":"3048156983cd077ad363ffb81ee5607a58b70566cb0b626fa26f672b45f4289e"} Oct 30 13:24:54.536939 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:54.536575 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b" event={"ID":"abed9e1f-8848-47f4-81e0-929784183401","Type":"ContainerDied","Data":"c4c111370cbbdacfa98993ae84f0f1bd955b10b7663ff1090cd17b7020f6ff30"} Oct 30 13:24:54.536939 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:54.536599 5922 scope.go:122] "RemoveContainer" containerID="3048156983cd077ad363ffb81ee5607a58b70566cb0b626fa26f672b45f4289e" Oct 30 13:24:54.536939 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:54.536754 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b" Oct 30 13:24:54.548349 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:54.548327 5922 scope.go:122] "RemoveContainer" containerID="3048156983cd077ad363ffb81ee5607a58b70566cb0b626fa26f672b45f4289e" Oct 30 13:24:54.548715 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:24:54.548587 5922 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3048156983cd077ad363ffb81ee5607a58b70566cb0b626fa26f672b45f4289e\": container with ID starting with 3048156983cd077ad363ffb81ee5607a58b70566cb0b626fa26f672b45f4289e not found: ID does not exist" containerID="3048156983cd077ad363ffb81ee5607a58b70566cb0b626fa26f672b45f4289e" Oct 30 13:24:54.548715 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:54.548633 5922 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3048156983cd077ad363ffb81ee5607a58b70566cb0b626fa26f672b45f4289e"} err="failed to get container status \"3048156983cd077ad363ffb81ee5607a58b70566cb0b626fa26f672b45f4289e\": rpc error: code = NotFound desc = could not find container \"3048156983cd077ad363ffb81ee5607a58b70566cb0b626fa26f672b45f4289e\": container with ID starting with 3048156983cd077ad363ffb81ee5607a58b70566cb0b626fa26f672b45f4289e not found: ID does not exist" Oct 30 13:24:54.611645 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:54.609346 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b"] Oct 30 13:24:54.625054 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:54.624951 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-068f3f8d-kserve-857685887c-zrx5b"] Oct 30 13:24:55.655637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:55.654949 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abed9e1f-8848-47f4-81e0-929784183401" path="/var/lib/kubelet/pods/abed9e1f-8848-47f4-81e0-929784183401/volumes" Oct 30 13:24:57.833717 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:24:57.833684 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:25:04.804467 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:04.804427 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52369a7d-kkvmr"] Oct 30 13:25:04.804900 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:04.804885 5922 cpu_manager.go:414] "containerMap: removing container" podUID="abed9e1f-8848-47f4-81e0-929784183401" containerName="main" Oct 30 13:25:04.804957 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:04.804902 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abed9e1f-8848-47f4-81e0-929784183401" containerName="main" Oct 30 13:25:04.804957 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:04.804923 5922 cpu_manager.go:414] "containerMap: removing container" podUID="f6effa00-3db7-4a53-90ac-f387e2dc1d5a" containerName="main" Oct 30 13:25:04.804957 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:04.804930 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6effa00-3db7-4a53-90ac-f387e2dc1d5a" containerName="main" Oct 30 13:25:04.805056 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:04.805019 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="abed9e1f-8848-47f4-81e0-929784183401" containerName="main" Oct 30 13:25:04.805056 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:04.805030 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6effa00-3db7-4a53-90ac-f387e2dc1d5a" containerName="main" Oct 30 13:25:04.805056 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:04.805042 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff585ad9-f06c-453b-b23f-f0519a479d9a" containerName="istio-proxy" Oct 30 13:25:04.809529 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:04.809510 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52369a7d-kkvmr" Oct 30 13:25:04.827941 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:04.827912 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52369a7d-kkvmr"] Oct 30 13:25:04.837984 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:04.837958 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:25:04.838146 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:04.837961 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:25:04.838208 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:04.837969 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:25:04.838208 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:04.838047 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:25:04.868648 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:04.868586 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52369a7d-kkvmr\" (UID: \"5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52369a7d-kkvmr" Oct 30 13:25:04.868790 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:04.868734 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56vxg\" (UniqueName: \"kubernetes.io/projected/5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb-kube-api-access-56vxg\") pod \"maas-api-revoke-sub-52369a7d-kkvmr\" (UID: \"5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52369a7d-kkvmr" Oct 30 13:25:04.868790 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:04.868775 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52369a7d-kkvmr\" (UID: \"5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52369a7d-kkvmr" Oct 30 13:25:04.970664 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:04.970631 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52369a7d-kkvmr\" (UID: \"5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52369a7d-kkvmr" Oct 30 13:25:04.970851 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:04.970746 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56vxg\" (UniqueName: \"kubernetes.io/projected/5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb-kube-api-access-56vxg\") pod \"maas-api-revoke-sub-52369a7d-kkvmr\" (UID: \"5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52369a7d-kkvmr" Oct 30 13:25:04.970851 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:04.970795 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52369a7d-kkvmr\" (UID: \"5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52369a7d-kkvmr" Oct 30 13:25:04.971461 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:04.971439 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52369a7d-kkvmr\" (UID: \"5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52369a7d-kkvmr" Oct 30 13:25:04.972958 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:04.972937 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52369a7d-kkvmr\" (UID: \"5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52369a7d-kkvmr" Oct 30 13:25:04.997028 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:04.997007 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56vxg\" (UniqueName: \"kubernetes.io/projected/5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb-kube-api-access-56vxg\") pod \"maas-api-revoke-sub-52369a7d-kkvmr\" (UID: \"5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52369a7d-kkvmr" Oct 30 13:25:05.122124 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:05.122061 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52369a7d-kkvmr" Oct 30 13:25:05.420388 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:05.420309 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52369a7d-kkvmr"] Oct 30 13:25:05.423905 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:25:05.423876 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ea58b8c_9fdd_48ae_9bf3_36ce2fadaaeb.slice/crio-ae1e344ddff83d9f812e38ca4baf70eb73ee18428b52be395f88cfb4dc8ec37d WatchSource:0}: Error finding container ae1e344ddff83d9f812e38ca4baf70eb73ee18428b52be395f88cfb4dc8ec37d: Status 404 returned error can't find the container with id ae1e344ddff83d9f812e38ca4baf70eb73ee18428b52be395f88cfb4dc8ec37d Oct 30 13:25:05.626597 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:05.626564 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52369a7d-kkvmr" event={"ID":"5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb","Type":"ContainerStarted","Data":"ae1e344ddff83d9f812e38ca4baf70eb73ee18428b52be395f88cfb4dc8ec37d"} Oct 30 13:25:06.632790 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:06.632757 5922 generic.go:356] "Generic (PLEG): container finished" podID="5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb" containerID="641056c453008ae2c59ffe6ae6b435e90a18b6712e3a55a69c4191154c2d7fce" exitCode=0 Oct 30 13:25:06.633223 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:06.632810 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52369a7d-kkvmr" event={"ID":"5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb","Type":"ContainerDied","Data":"641056c453008ae2c59ffe6ae6b435e90a18b6712e3a55a69c4191154c2d7fce"} Oct 30 13:25:07.779969 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:07.779943 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52369a7d-kkvmr" Oct 30 13:25:07.896320 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:07.896254 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb-maas-api-openshift-ca\") pod \"5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb\" (UID: \"5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb\") " Oct 30 13:25:07.896466 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:07.896341 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb-kube-api-access-56vxg\" (UniqueName: \"kubernetes.io/projected/5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb-kube-api-access-56vxg\") pod \"5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb\" (UID: \"5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb\") " Oct 30 13:25:07.896466 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:07.896393 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb-maas-api-xks-ca\") pod \"5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb\" (UID: \"5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb\") " Oct 30 13:25:07.896628 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:07.896594 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb-maas-api-openshift-ca" pod "5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb" (UID: "5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:25:07.898471 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:07.898443 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb-kube-api-access-56vxg" pod "5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb" (UID: "5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb"). InnerVolumeSpecName "kube-api-access-56vxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:07.898471 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:07.898446 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb-maas-api-xks-ca" pod "5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb" (UID: "5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:25:07.997705 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:07.997681 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:25:07.997705 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:07.997704 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56vxg\" (UniqueName: \"kubernetes.io/projected/5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb-kube-api-access-56vxg\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:25:07.997847 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:07.997714 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:25:08.650167 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:08.650140 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52369a7d-kkvmr" Oct 30 13:25:08.650167 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:08.650148 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52369a7d-kkvmr" event={"ID":"5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb","Type":"ContainerDied","Data":"ae1e344ddff83d9f812e38ca4baf70eb73ee18428b52be395f88cfb4dc8ec37d"} Oct 30 13:25:08.650359 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:08.650186 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae1e344ddff83d9f812e38ca4baf70eb73ee18428b52be395f88cfb4dc8ec37d" Oct 30 13:25:16.257637 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:16.257593 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:25:23.552083 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:25:23.552047 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:25:25.437938 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:25.437907 5922 scope.go:122] "RemoveContainer" containerID="56cbc23a49ff5ac334ac474abed887d72c78db539f62009c2161b562bfbba7a6" Oct 30 13:25:25.981734 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:25.981700 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:25:34.011690 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:34.011658 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:25:35.037331 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:35.037243 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:37.901316 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:37.901282 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:38.026580 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:38.026549 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:40.780597 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:40.780562 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qg8cf"] Oct 30 13:25:40.780999 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:40.780984 5922 cpu_manager.go:414] "containerMap: removing container" podUID="5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb" containerName="revoke-keys" Oct 30 13:25:40.780999 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:40.780998 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb" containerName="revoke-keys" Oct 30 13:25:40.781084 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:40.781050 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb" containerName="revoke-keys" Oct 30 13:25:40.785403 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:40.785386 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qg8cf" Oct 30 13:25:40.805402 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:40.805376 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qg8cf"] Oct 30 13:25:40.815758 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:40.815735 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:25:40.815878 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:40.815736 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:25:40.815878 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:40.815769 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:25:40.815878 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:40.815789 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:25:40.874944 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:40.874916 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ab5a960-ff93-432e-adf6-cf8a1887b965-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-qg8cf\" (UID: \"3ab5a960-ff93-432e-adf6-cf8a1887b965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qg8cf" Oct 30 13:25:40.875063 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:40.874964 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjxdr\" (UniqueName: \"kubernetes.io/projected/3ab5a960-ff93-432e-adf6-cf8a1887b965-kube-api-access-jjxdr\") pod \"maas-api-revoke-sub-f03629db-qg8cf\" (UID: \"3ab5a960-ff93-432e-adf6-cf8a1887b965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qg8cf" Oct 30 13:25:40.875063 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:40.875030 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ab5a960-ff93-432e-adf6-cf8a1887b965-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-qg8cf\" (UID: \"3ab5a960-ff93-432e-adf6-cf8a1887b965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qg8cf" Oct 30 13:25:40.976084 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:40.976057 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ab5a960-ff93-432e-adf6-cf8a1887b965-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-qg8cf\" (UID: \"3ab5a960-ff93-432e-adf6-cf8a1887b965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qg8cf" Oct 30 13:25:40.976196 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:40.976147 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ab5a960-ff93-432e-adf6-cf8a1887b965-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-qg8cf\" (UID: \"3ab5a960-ff93-432e-adf6-cf8a1887b965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qg8cf" Oct 30 13:25:40.976241 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:40.976206 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjxdr\" (UniqueName: \"kubernetes.io/projected/3ab5a960-ff93-432e-adf6-cf8a1887b965-kube-api-access-jjxdr\") pod \"maas-api-revoke-sub-f03629db-qg8cf\" (UID: \"3ab5a960-ff93-432e-adf6-cf8a1887b965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qg8cf" Oct 30 13:25:40.976728 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:40.976712 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ab5a960-ff93-432e-adf6-cf8a1887b965-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-qg8cf\" (UID: \"3ab5a960-ff93-432e-adf6-cf8a1887b965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qg8cf" Oct 30 13:25:40.978464 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:40.978444 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ab5a960-ff93-432e-adf6-cf8a1887b965-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-qg8cf\" (UID: \"3ab5a960-ff93-432e-adf6-cf8a1887b965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qg8cf" Oct 30 13:25:41.002237 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:41.002214 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjxdr\" (UniqueName: \"kubernetes.io/projected/3ab5a960-ff93-432e-adf6-cf8a1887b965-kube-api-access-jjxdr\") pod \"maas-api-revoke-sub-f03629db-qg8cf\" (UID: \"3ab5a960-ff93-432e-adf6-cf8a1887b965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qg8cf" Oct 30 13:25:41.097654 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:41.097590 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qg8cf" Oct 30 13:25:41.391899 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:41.391873 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qg8cf"] Oct 30 13:25:41.394885 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:25:41.394854 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ab5a960_ff93_432e_adf6_cf8a1887b965.slice/crio-cc7cb825df1bcfe538f94661e919805426fd23af8648841c32bedf3540f39471 WatchSource:0}: Error finding container cc7cb825df1bcfe538f94661e919805426fd23af8648841c32bedf3540f39471: Status 404 returned error can't find the container with id cc7cb825df1bcfe538f94661e919805426fd23af8648841c32bedf3540f39471 Oct 30 13:25:41.850382 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:41.850348 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qg8cf" event={"ID":"3ab5a960-ff93-432e-adf6-cf8a1887b965","Type":"ContainerStarted","Data":"ff1a0e26f46f9524e5b1518cdd5a78b6ab01a1a76ccdc0890d8b15379fcd8501"} Oct 30 13:25:41.850776 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:41.850402 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qg8cf" event={"ID":"3ab5a960-ff93-432e-adf6-cf8a1887b965","Type":"ContainerStarted","Data":"cc7cb825df1bcfe538f94661e919805426fd23af8648841c32bedf3540f39471"} Oct 30 13:25:41.891766 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:41.891726 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qg8cf" podStartSLOduration=1.891714285 podStartE2EDuration="1.891714285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:25:41.890729647 +0000 UTC m=+1998.360145393" watchObservedRunningTime="2026-10-30 13:25:41.891714285 +0000 UTC m=+1998.361130030" Oct 30 13:25:42.856477 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:42.856444 5922 generic.go:356] "Generic (PLEG): container finished" podID="3ab5a960-ff93-432e-adf6-cf8a1887b965" containerID="ff1a0e26f46f9524e5b1518cdd5a78b6ab01a1a76ccdc0890d8b15379fcd8501" exitCode=0 Oct 30 13:25:42.856895 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:42.856499 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qg8cf" event={"ID":"3ab5a960-ff93-432e-adf6-cf8a1887b965","Type":"ContainerDied","Data":"ff1a0e26f46f9524e5b1518cdd5a78b6ab01a1a76ccdc0890d8b15379fcd8501"} Oct 30 13:25:43.998877 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:43.998852 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qg8cf" Oct 30 13:25:44.102788 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:44.102758 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ab5a960-ff93-432e-adf6-cf8a1887b965-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ab5a960-ff93-432e-adf6-cf8a1887b965-maas-api-openshift-ca\") pod \"3ab5a960-ff93-432e-adf6-cf8a1887b965\" (UID: \"3ab5a960-ff93-432e-adf6-cf8a1887b965\") " Oct 30 13:25:44.102956 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:44.102809 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ab5a960-ff93-432e-adf6-cf8a1887b965-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ab5a960-ff93-432e-adf6-cf8a1887b965-maas-api-xks-ca\") pod \"3ab5a960-ff93-432e-adf6-cf8a1887b965\" (UID: \"3ab5a960-ff93-432e-adf6-cf8a1887b965\") " Oct 30 13:25:44.103029 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:44.102965 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ab5a960-ff93-432e-adf6-cf8a1887b965-kube-api-access-jjxdr\" (UniqueName: \"kubernetes.io/projected/3ab5a960-ff93-432e-adf6-cf8a1887b965-kube-api-access-jjxdr\") pod \"3ab5a960-ff93-432e-adf6-cf8a1887b965\" (UID: \"3ab5a960-ff93-432e-adf6-cf8a1887b965\") " Oct 30 13:25:44.103167 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:44.103144 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ab5a960-ff93-432e-adf6-cf8a1887b965-maas-api-openshift-ca" pod "3ab5a960-ff93-432e-adf6-cf8a1887b965" (UID: "3ab5a960-ff93-432e-adf6-cf8a1887b965"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:25:44.103246 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:44.103231 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ab5a960-ff93-432e-adf6-cf8a1887b965-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:25:44.104792 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:44.104770 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ab5a960-ff93-432e-adf6-cf8a1887b965-maas-api-xks-ca" pod "3ab5a960-ff93-432e-adf6-cf8a1887b965" (UID: "3ab5a960-ff93-432e-adf6-cf8a1887b965"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:25:44.104888 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:44.104872 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ab5a960-ff93-432e-adf6-cf8a1887b965-kube-api-access-jjxdr" pod "3ab5a960-ff93-432e-adf6-cf8a1887b965" (UID: "3ab5a960-ff93-432e-adf6-cf8a1887b965"). InnerVolumeSpecName "kube-api-access-jjxdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:44.203780 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:44.203759 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ab5a960-ff93-432e-adf6-cf8a1887b965-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:25:44.203780 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:44.203780 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjxdr\" (UniqueName: \"kubernetes.io/projected/3ab5a960-ff93-432e-adf6-cf8a1887b965-kube-api-access-jjxdr\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:25:44.866949 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:44.866920 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qg8cf" Oct 30 13:25:44.867120 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:44.866925 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qg8cf" event={"ID":"3ab5a960-ff93-432e-adf6-cf8a1887b965","Type":"ContainerDied","Data":"cc7cb825df1bcfe538f94661e919805426fd23af8648841c32bedf3540f39471"} Oct 30 13:25:44.867120 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:44.867038 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc7cb825df1bcfe538f94661e919805426fd23af8648841c32bedf3540f39471" Oct 30 13:25:47.847490 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:47.847457 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:25:53.552072 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:25:53.551989 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:25:53.725211 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:53.725176 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-wznwd"] Oct 30 13:25:53.725589 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:53.725576 5922 cpu_manager.go:414] "containerMap: removing container" podUID="3ab5a960-ff93-432e-adf6-cf8a1887b965" containerName="revoke-keys" Oct 30 13:25:53.725589 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:53.725589 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ab5a960-ff93-432e-adf6-cf8a1887b965" containerName="revoke-keys" Oct 30 13:25:53.725704 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:53.725686 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ab5a960-ff93-432e-adf6-cf8a1887b965" containerName="revoke-keys" Oct 30 13:25:53.728955 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:53.728939 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-wznwd" Oct 30 13:25:53.748712 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:53.748689 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-wznwd"] Oct 30 13:25:53.758244 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:53.758215 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:25:53.758244 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:53.758230 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:25:53.758419 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:53.758254 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:25:53.758419 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:53.758230 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:25:53.824541 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:53.824470 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb2ks\" (UniqueName: \"kubernetes.io/projected/ca5a22ea-d27f-415c-b49a-73544a21096d-kube-api-access-jb2ks\") pod \"maas-api-revoke-sub-0165d7d2-wznwd\" (UID: \"ca5a22ea-d27f-415c-b49a-73544a21096d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-wznwd" Oct 30 13:25:53.824696 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:53.824552 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca5a22ea-d27f-415c-b49a-73544a21096d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-wznwd\" (UID: \"ca5a22ea-d27f-415c-b49a-73544a21096d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-wznwd" Oct 30 13:25:53.824696 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:53.824599 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca5a22ea-d27f-415c-b49a-73544a21096d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-wznwd\" (UID: \"ca5a22ea-d27f-415c-b49a-73544a21096d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-wznwd" Oct 30 13:25:53.925461 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:53.925432 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca5a22ea-d27f-415c-b49a-73544a21096d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-wznwd\" (UID: \"ca5a22ea-d27f-415c-b49a-73544a21096d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-wznwd" Oct 30 13:25:53.925461 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:53.925461 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca5a22ea-d27f-415c-b49a-73544a21096d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-wznwd\" (UID: \"ca5a22ea-d27f-415c-b49a-73544a21096d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-wznwd" Oct 30 13:25:53.925714 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:53.925517 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb2ks\" (UniqueName: \"kubernetes.io/projected/ca5a22ea-d27f-415c-b49a-73544a21096d-kube-api-access-jb2ks\") pod \"maas-api-revoke-sub-0165d7d2-wznwd\" (UID: \"ca5a22ea-d27f-415c-b49a-73544a21096d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-wznwd" Oct 30 13:25:53.926123 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:53.926097 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca5a22ea-d27f-415c-b49a-73544a21096d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-wznwd\" (UID: \"ca5a22ea-d27f-415c-b49a-73544a21096d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-wznwd" Oct 30 13:25:53.927821 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:53.927798 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca5a22ea-d27f-415c-b49a-73544a21096d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-wznwd\" (UID: \"ca5a22ea-d27f-415c-b49a-73544a21096d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-wznwd" Oct 30 13:25:53.949535 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:53.949509 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb2ks\" (UniqueName: \"kubernetes.io/projected/ca5a22ea-d27f-415c-b49a-73544a21096d-kube-api-access-jb2ks\") pod \"maas-api-revoke-sub-0165d7d2-wznwd\" (UID: \"ca5a22ea-d27f-415c-b49a-73544a21096d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-wznwd" Oct 30 13:25:54.042079 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:54.042056 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-wznwd" Oct 30 13:25:54.342622 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:54.342577 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-wznwd"] Oct 30 13:25:54.345991 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:25:54.345931 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca5a22ea_d27f_415c_b49a_73544a21096d.slice/crio-d819463db1833a21a8b38ff4b5d0b5d6cead7d5ddd488a9a021bce3975dfb9d6 WatchSource:0}: Error finding container d819463db1833a21a8b38ff4b5d0b5d6cead7d5ddd488a9a021bce3975dfb9d6: Status 404 returned error can't find the container with id d819463db1833a21a8b38ff4b5d0b5d6cead7d5ddd488a9a021bce3975dfb9d6 Oct 30 13:25:54.929037 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:54.929008 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-wznwd" event={"ID":"ca5a22ea-d27f-415c-b49a-73544a21096d","Type":"ContainerStarted","Data":"2ede2af066911ea6108c7de9cfaaab62c781f67ee0eb92aae6ef3f4418f2540a"} Oct 30 13:25:54.929449 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:54.929064 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-wznwd" event={"ID":"ca5a22ea-d27f-415c-b49a-73544a21096d","Type":"ContainerStarted","Data":"d819463db1833a21a8b38ff4b5d0b5d6cead7d5ddd488a9a021bce3975dfb9d6"} Oct 30 13:25:54.971033 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:54.970995 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-wznwd" podStartSLOduration=1.970983671 podStartE2EDuration="1.970983671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:25:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:25:54.970877974 +0000 UTC m=+2011.440293731" watchObservedRunningTime="2026-10-30 13:25:54.970983671 +0000 UTC m=+2011.440399415" Oct 30 13:25:55.935562 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:55.935525 5922 generic.go:356] "Generic (PLEG): container finished" podID="ca5a22ea-d27f-415c-b49a-73544a21096d" containerID="2ede2af066911ea6108c7de9cfaaab62c781f67ee0eb92aae6ef3f4418f2540a" exitCode=0 Oct 30 13:25:55.935988 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:55.935607 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-wznwd" event={"ID":"ca5a22ea-d27f-415c-b49a-73544a21096d","Type":"ContainerDied","Data":"2ede2af066911ea6108c7de9cfaaab62c781f67ee0eb92aae6ef3f4418f2540a"} Oct 30 13:25:57.072634 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:57.072597 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-wznwd" Oct 30 13:25:57.153714 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:57.153684 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca5a22ea-d27f-415c-b49a-73544a21096d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca5a22ea-d27f-415c-b49a-73544a21096d-maas-api-openshift-ca\") pod \"ca5a22ea-d27f-415c-b49a-73544a21096d\" (UID: \"ca5a22ea-d27f-415c-b49a-73544a21096d\") " Oct 30 13:25:57.153874 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:57.153735 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca5a22ea-d27f-415c-b49a-73544a21096d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca5a22ea-d27f-415c-b49a-73544a21096d-maas-api-xks-ca\") pod \"ca5a22ea-d27f-415c-b49a-73544a21096d\" (UID: \"ca5a22ea-d27f-415c-b49a-73544a21096d\") " Oct 30 13:25:57.153874 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:57.153846 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca5a22ea-d27f-415c-b49a-73544a21096d-kube-api-access-jb2ks\" (UniqueName: \"kubernetes.io/projected/ca5a22ea-d27f-415c-b49a-73544a21096d-kube-api-access-jb2ks\") pod \"ca5a22ea-d27f-415c-b49a-73544a21096d\" (UID: \"ca5a22ea-d27f-415c-b49a-73544a21096d\") " Oct 30 13:25:57.154092 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:57.154070 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca5a22ea-d27f-415c-b49a-73544a21096d-maas-api-openshift-ca" pod "ca5a22ea-d27f-415c-b49a-73544a21096d" (UID: "ca5a22ea-d27f-415c-b49a-73544a21096d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:25:57.154176 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:57.154151 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca5a22ea-d27f-415c-b49a-73544a21096d-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:25:57.155800 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:57.155773 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca5a22ea-d27f-415c-b49a-73544a21096d-maas-api-xks-ca" pod "ca5a22ea-d27f-415c-b49a-73544a21096d" (UID: "ca5a22ea-d27f-415c-b49a-73544a21096d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:25:57.155923 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:57.155846 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca5a22ea-d27f-415c-b49a-73544a21096d-kube-api-access-jb2ks" pod "ca5a22ea-d27f-415c-b49a-73544a21096d" (UID: "ca5a22ea-d27f-415c-b49a-73544a21096d"). InnerVolumeSpecName "kube-api-access-jb2ks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:57.255414 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:57.255389 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca5a22ea-d27f-415c-b49a-73544a21096d-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:25:57.255414 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:57.255411 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb2ks\" (UniqueName: \"kubernetes.io/projected/ca5a22ea-d27f-415c-b49a-73544a21096d-kube-api-access-jb2ks\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:25:57.946457 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:57.946426 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-wznwd" Oct 30 13:25:57.946639 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:57.946450 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-wznwd" event={"ID":"ca5a22ea-d27f-415c-b49a-73544a21096d","Type":"ContainerDied","Data":"d819463db1833a21a8b38ff4b5d0b5d6cead7d5ddd488a9a021bce3975dfb9d6"} Oct 30 13:25:57.946639 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:25:57.946497 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d819463db1833a21a8b38ff4b5d0b5d6cead7d5ddd488a9a021bce3975dfb9d6" Oct 30 13:26:04.623027 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:04.622995 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fpnhl"] Oct 30 13:26:04.623501 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:04.623484 5922 cpu_manager.go:414] "containerMap: removing container" podUID="ca5a22ea-d27f-415c-b49a-73544a21096d" containerName="revoke-keys" Oct 30 13:26:04.623548 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:04.623503 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca5a22ea-d27f-415c-b49a-73544a21096d" containerName="revoke-keys" Oct 30 13:26:04.623636 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:04.623625 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca5a22ea-d27f-415c-b49a-73544a21096d" containerName="revoke-keys" Oct 30 13:26:04.626940 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:04.626916 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fpnhl" Oct 30 13:26:04.646293 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:04.646259 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fpnhl"] Oct 30 13:26:04.656273 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:04.656221 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:26:04.656273 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:04.656235 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 13:26:04.656273 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:04.656258 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 13:26:04.656456 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:04.656307 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:26:04.725659 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:04.725629 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f186ed68-13b4-4849-ab58-62a2d93339b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-fpnhl\" (UID: \"f186ed68-13b4-4849-ab58-62a2d93339b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fpnhl" Oct 30 13:26:04.725804 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:04.725669 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f186ed68-13b4-4849-ab58-62a2d93339b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-fpnhl\" (UID: \"f186ed68-13b4-4849-ab58-62a2d93339b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fpnhl" Oct 30 13:26:04.725845 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:04.725821 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm9n9\" (UniqueName: \"kubernetes.io/projected/f186ed68-13b4-4849-ab58-62a2d93339b3-kube-api-access-pm9n9\") pod \"maas-api-revoke-sub-6b19338f-fpnhl\" (UID: \"f186ed68-13b4-4849-ab58-62a2d93339b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fpnhl" Oct 30 13:26:04.826296 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:04.826272 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f186ed68-13b4-4849-ab58-62a2d93339b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-fpnhl\" (UID: \"f186ed68-13b4-4849-ab58-62a2d93339b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fpnhl" Oct 30 13:26:04.826427 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:04.826300 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f186ed68-13b4-4849-ab58-62a2d93339b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-fpnhl\" (UID: \"f186ed68-13b4-4849-ab58-62a2d93339b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fpnhl" Oct 30 13:26:04.826427 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:04.826374 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm9n9\" (UniqueName: \"kubernetes.io/projected/f186ed68-13b4-4849-ab58-62a2d93339b3-kube-api-access-pm9n9\") pod \"maas-api-revoke-sub-6b19338f-fpnhl\" (UID: \"f186ed68-13b4-4849-ab58-62a2d93339b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fpnhl" Oct 30 13:26:04.827764 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:04.827735 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f186ed68-13b4-4849-ab58-62a2d93339b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-fpnhl\" (UID: \"f186ed68-13b4-4849-ab58-62a2d93339b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fpnhl" Oct 30 13:26:04.835457 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:04.835434 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f186ed68-13b4-4849-ab58-62a2d93339b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-fpnhl\" (UID: \"f186ed68-13b4-4849-ab58-62a2d93339b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fpnhl" Oct 30 13:26:04.851065 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:04.851041 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm9n9\" (UniqueName: \"kubernetes.io/projected/f186ed68-13b4-4849-ab58-62a2d93339b3-kube-api-access-pm9n9\") pod \"maas-api-revoke-sub-6b19338f-fpnhl\" (UID: \"f186ed68-13b4-4849-ab58-62a2d93339b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fpnhl" Oct 30 13:26:04.940375 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:04.940317 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fpnhl" Oct 30 13:26:05.245700 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:05.245664 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fpnhl"] Oct 30 13:26:05.249454 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:26:05.249418 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf186ed68_13b4_4849_ab58_62a2d93339b3.slice/crio-2d8bfd70941843d6595b74de2f2841f9e8295fea59398aff50614c9c46c97d5a WatchSource:0}: Error finding container 2d8bfd70941843d6595b74de2f2841f9e8295fea59398aff50614c9c46c97d5a: Status 404 returned error can't find the container with id 2d8bfd70941843d6595b74de2f2841f9e8295fea59398aff50614c9c46c97d5a Oct 30 13:26:05.986877 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:05.986846 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fpnhl" event={"ID":"f186ed68-13b4-4849-ab58-62a2d93339b3","Type":"ContainerStarted","Data":"1b7447f0fc41d56d9482a30729a741c27199f769537f401c26bdb1dae0582bf6"} Oct 30 13:26:05.987209 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:05.986904 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fpnhl" event={"ID":"f186ed68-13b4-4849-ab58-62a2d93339b3","Type":"ContainerStarted","Data":"2d8bfd70941843d6595b74de2f2841f9e8295fea59398aff50614c9c46c97d5a"} Oct 30 13:26:06.044914 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:06.044864 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fpnhl" podStartSLOduration=2.044850944 podStartE2EDuration="2.044850944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:26:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:26:06.032039042 +0000 UTC m=+2022.501454786" watchObservedRunningTime="2026-10-30 13:26:06.044850944 +0000 UTC m=+2022.514266690" Oct 30 13:26:06.992972 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:06.992940 5922 generic.go:356] "Generic (PLEG): container finished" podID="f186ed68-13b4-4849-ab58-62a2d93339b3" containerID="1b7447f0fc41d56d9482a30729a741c27199f769537f401c26bdb1dae0582bf6" exitCode=0 Oct 30 13:26:06.993350 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:06.993031 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fpnhl" event={"ID":"f186ed68-13b4-4849-ab58-62a2d93339b3","Type":"ContainerDied","Data":"1b7447f0fc41d56d9482a30729a741c27199f769537f401c26bdb1dae0582bf6"} Oct 30 13:26:08.135030 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:08.135005 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fpnhl" Oct 30 13:26:08.156757 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:08.156731 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f186ed68-13b4-4849-ab58-62a2d93339b3-kube-api-access-pm9n9\" (UniqueName: \"kubernetes.io/projected/f186ed68-13b4-4849-ab58-62a2d93339b3-kube-api-access-pm9n9\") pod \"f186ed68-13b4-4849-ab58-62a2d93339b3\" (UID: \"f186ed68-13b4-4849-ab58-62a2d93339b3\") " Oct 30 13:26:08.156870 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:08.156769 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f186ed68-13b4-4849-ab58-62a2d93339b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f186ed68-13b4-4849-ab58-62a2d93339b3-maas-api-xks-ca\") pod \"f186ed68-13b4-4849-ab58-62a2d93339b3\" (UID: \"f186ed68-13b4-4849-ab58-62a2d93339b3\") " Oct 30 13:26:08.156870 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:08.156791 5922 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f186ed68-13b4-4849-ab58-62a2d93339b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f186ed68-13b4-4849-ab58-62a2d93339b3-maas-api-openshift-ca\") pod \"f186ed68-13b4-4849-ab58-62a2d93339b3\" (UID: \"f186ed68-13b4-4849-ab58-62a2d93339b3\") " Oct 30 13:26:08.157165 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:08.157143 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f186ed68-13b4-4849-ab58-62a2d93339b3-maas-api-openshift-ca" pod "f186ed68-13b4-4849-ab58-62a2d93339b3" (UID: "f186ed68-13b4-4849-ab58-62a2d93339b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:26:08.158756 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:08.158733 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f186ed68-13b4-4849-ab58-62a2d93339b3-kube-api-access-pm9n9" pod "f186ed68-13b4-4849-ab58-62a2d93339b3" (UID: "f186ed68-13b4-4849-ab58-62a2d93339b3"). InnerVolumeSpecName "kube-api-access-pm9n9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:26:08.158869 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:08.158733 5922 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f186ed68-13b4-4849-ab58-62a2d93339b3-maas-api-xks-ca" pod "f186ed68-13b4-4849-ab58-62a2d93339b3" (UID: "f186ed68-13b4-4849-ab58-62a2d93339b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:26:08.257818 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:08.257762 5922 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm9n9\" (UniqueName: \"kubernetes.io/projected/f186ed68-13b4-4849-ab58-62a2d93339b3-kube-api-access-pm9n9\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:26:08.257818 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:08.257784 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f186ed68-13b4-4849-ab58-62a2d93339b3-maas-api-xks-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:26:08.257818 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:08.257794 5922 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f186ed68-13b4-4849-ab58-62a2d93339b3-maas-api-openshift-ca\") on node \"ip-10-0-137-21.ec2.internal\" DevicePath \"\"" Oct 30 13:26:09.003545 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:09.003517 5922 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fpnhl" Oct 30 13:26:09.003811 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:09.003516 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fpnhl" event={"ID":"f186ed68-13b4-4849-ab58-62a2d93339b3","Type":"ContainerDied","Data":"2d8bfd70941843d6595b74de2f2841f9e8295fea59398aff50614c9c46c97d5a"} Oct 30 13:26:09.003811 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:09.003665 5922 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d8bfd70941843d6595b74de2f2841f9e8295fea59398aff50614c9c46c97d5a" Oct 30 13:26:15.911914 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:15.911880 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5rxtm"] Oct 30 13:26:16.133479 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:16.133446 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-c755c8f75-7njmp_578b67e5-39d7-44dc-b8e6-39c364ba1c5c/authorino/0.log" Oct 30 13:26:23.551515 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:26:23.551477 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:26:25.466996 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:25.466966 5922 scope.go:122] "RemoveContainer" containerID="1cd7e086b87b95c2b1c7e05ae4684e594972633a4fa8ea2e801ed518ed8eb834" Oct 30 13:26:25.475338 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:25.475310 5922 scope.go:122] "RemoveContainer" containerID="8ac4354d5f023767eadc50d230502a059931aa1266c321d72dedc4afc662597b" Oct 30 13:26:26.288843 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:26.288808 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:26.769270 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:26.769241 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6f5bb998fd-4c485_8ec441fb-e6b7-471b-bc1e-d418d1a516b1/manager/0.log" Oct 30 13:26:28.275790 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:28.275756 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-dmsjc_18db6c7c-bd7b-4342-aa25-b184aca06d96/manager/0.log" Oct 30 13:26:28.760915 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:28.760879 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn_40764e2d-8e5e-427b-adb5-e3c0bf47e6c8/copy-manifests/0.log" Oct 30 13:26:28.796390 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:28.796366 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn_40764e2d-8e5e-427b-adb5-e3c0bf47e6c8/manager/0.log" Oct 30 13:26:30.677266 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:30.677237 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-wznwd_ca5a22ea-d27f-415c-b49a-73544a21096d/revoke-keys/0.log" Oct 30 13:26:31.223627 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:31.223585 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-52369a7d-kkvmr_5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb/revoke-keys/0.log" Oct 30 13:26:31.416791 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:31.416761 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-fpnhl_f186ed68-13b4-4849-ab58-62a2d93339b3/revoke-keys/0.log" Oct 30 13:26:31.601244 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:31.601181 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-9c27731c-wnfxv_bf64f284-eceb-44e1-a9da-9209a50ae78b/revoke-keys/0.log" Oct 30 13:26:31.789569 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:31.789545 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-wcd82_0aece694-bc22-4290-a02d-8dd36aaf4b0e/revoke-keys/0.log" Oct 30 13:26:32.160193 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:32.160166 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-qg8cf_3ab5a960-ff93-432e-adf6-cf8a1887b965/revoke-keys/0.log" Oct 30 13:26:33.808039 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:33.808003 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-c755c8f75-7njmp_578b67e5-39d7-44dc-b8e6-39c364ba1c5c/authorino/0.log" Oct 30 13:26:34.051358 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:34.051328 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-dn2k2_4c349358-62be-42fe-a686-ec417fd3d841/manager/0.log" Oct 30 13:26:34.248124 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:34.248091 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-6v28d_8ab008c5-fd4d-42b2-8f7e-67b35e9aae15/manager/0.log" Oct 30 13:26:34.887957 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:34.887928 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-5rxtm_4480c6ba-b6e6-4f0d-bc25-439bb9dcde74/limitador/0.log" Oct 30 13:26:35.156677 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:35.156577 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:36.430438 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:36.430405 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d59dd5dc7-rc4c8_3108a560-3472-4283-b17e-2099fd70bd78/kube-auth-proxy/0.log" Oct 30 13:26:36.627942 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:36.627911 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-5x6r5_3584258d-ca3c-4574-a39e-a70cc78d0ce6/istio-proxy/0.log" Oct 30 13:26:37.254656 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:37.254607 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-bmhqs_3577d48e-eeaf-4608-ac46-99d2667ea4c0/istio-proxy/0.log" Oct 30 13:26:37.443708 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:37.443678 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-cxwwl_c29858c0-d396-4092-9e19-b76069383a82/istio-proxy/0.log" Oct 30 13:26:38.661162 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:38.661138 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-rh4bz_bd96022a-32a7-4b5d-a59b-b3678075d1f6/payload-pre-processing/0.log" Oct 30 13:26:38.852344 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:38.852315 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-wdr7j_a103895f-e01d-4661-b522-73301acf6e22/payload-processing/0.log" Oct 30 13:26:40.391374 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:40.391344 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-p22vk_6870ecd9-9f61-4a6f-a753-1ca41fe77166/main/0.log" Oct 30 13:26:41.154517 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:41.154482 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz_14ae0aea-473f-4ca0-bcc0-f617b9839c8c/main/0.log" Oct 30 13:26:42.030652 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:42.030607 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-c755c8f75-7njmp_578b67e5-39d7-44dc-b8e6-39c364ba1c5c/authorino/0.log" Oct 30 13:26:50.984009 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:50.983974 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6f5bb998fd-4c485_8ec441fb-e6b7-471b-bc1e-d418d1a516b1/manager/0.log" Oct 30 13:26:52.303024 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:52.302991 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-dmsjc_18db6c7c-bd7b-4342-aa25-b184aca06d96/manager/0.log" Oct 30 13:26:52.760143 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:52.760111 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn_40764e2d-8e5e-427b-adb5-e3c0bf47e6c8/copy-manifests/0.log" Oct 30 13:26:52.800434 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:52.800404 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7bb4ff5b5c-nnvwn_40764e2d-8e5e-427b-adb5-e3c0bf47e6c8/manager/0.log" Oct 30 13:26:53.552508 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:26:53.552466 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:26:54.468858 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:54.468827 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-wznwd_ca5a22ea-d27f-415c-b49a-73544a21096d/revoke-keys/0.log" Oct 30 13:26:55.061799 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:55.061768 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-52369a7d-kkvmr_5ea58b8c-9fdd-48ae-9bf3-36ce2fadaaeb/revoke-keys/0.log" Oct 30 13:26:55.244621 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:55.244595 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-fpnhl_f186ed68-13b4-4849-ab58-62a2d93339b3/revoke-keys/0.log" Oct 30 13:26:55.439082 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:55.439005 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-9c27731c-wnfxv_bf64f284-eceb-44e1-a9da-9209a50ae78b/revoke-keys/0.log" Oct 30 13:26:55.643098 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:55.643066 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-wcd82_0aece694-bc22-4290-a02d-8dd36aaf4b0e/revoke-keys/0.log" Oct 30 13:26:56.043423 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:56.043392 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-qg8cf_3ab5a960-ff93-432e-adf6-cf8a1887b965/revoke-keys/0.log" Oct 30 13:26:57.292846 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:57.292814 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-c755c8f75-7njmp_578b67e5-39d7-44dc-b8e6-39c364ba1c5c/authorino/0.log" Oct 30 13:26:57.546770 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:57.546673 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-dn2k2_4c349358-62be-42fe-a686-ec417fd3d841/manager/0.log" Oct 30 13:26:57.748815 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:57.748790 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-6v28d_8ab008c5-fd4d-42b2-8f7e-67b35e9aae15/manager/0.log" Oct 30 13:26:58.402284 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:58.402253 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-5rxtm_4480c6ba-b6e6-4f0d-bc25-439bb9dcde74/limitador/0.log" Oct 30 13:26:59.700882 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:59.700855 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d59dd5dc7-rc4c8_3108a560-3472-4283-b17e-2099fd70bd78/kube-auth-proxy/0.log" Oct 30 13:26:59.900555 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:26:59.900523 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-5x6r5_3584258d-ca3c-4574-a39e-a70cc78d0ce6/istio-proxy/0.log" Oct 30 13:27:00.485430 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:27:00.485397 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-bmhqs_3577d48e-eeaf-4608-ac46-99d2667ea4c0/istio-proxy/0.log" Oct 30 13:27:00.699691 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:27:00.699657 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-cxwwl_c29858c0-d396-4092-9e19-b76069383a82/istio-proxy/0.log" Oct 30 13:27:01.913338 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:27:01.913306 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-rh4bz_bd96022a-32a7-4b5d-a59b-b3678075d1f6/payload-pre-processing/0.log" Oct 30 13:27:02.092924 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:27:02.092889 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-wdr7j_a103895f-e01d-4661-b522-73301acf6e22/payload-processing/0.log" Oct 30 13:27:03.175259 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:27:03.175230 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-p22vk_6870ecd9-9f61-4a6f-a753-1ca41fe77166/main/0.log" Oct 30 13:27:03.930526 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:27:03.930500 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-tx4fz_14ae0aea-473f-4ca0-bcc0-f617b9839c8c/main/0.log" Oct 30 13:27:20.039005 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:27:20.038925 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wcd82"] Oct 30 13:27:20.051158 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:27:20.051129 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wcd82"] Oct 30 13:27:21.643456 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:27:21.643415 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0aece694-bc22-4290-a02d-8dd36aaf4b0e" path="/var/lib/kubelet/pods/0aece694-bc22-4290-a02d-8dd36aaf4b0e/volumes" Oct 30 13:27:23.552184 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:27:23.552153 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:27:35.373956 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:27:35.373915 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:40.820375 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:27:40.820344 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-wdr7j_a103895f-e01d-4661-b522-73301acf6e22/payload-processing/0.log" Oct 30 13:27:41.000541 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:27:41.000514 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-wdr7j_a103895f-e01d-4661-b522-73301acf6e22/payload-processing/0.log" Oct 30 13:27:41.878564 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:27:41.878527 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:41.991086 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:27:41.991058 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:53.551893 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:27:53.551852 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:28:13.034762 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:13.034730 5922 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c27731c-wnfxv"] Oct 30 13:28:13.046521 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:13.046483 5922 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c27731c-wnfxv"] Oct 30 13:28:13.642240 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:13.642207 5922 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf64f284-eceb-44e1-a9da-9209a50ae78b" path="/var/lib/kubelet/pods/bf64f284-eceb-44e1-a9da-9209a50ae78b/volumes" Oct 30 13:28:23.552432 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:28:23.552393 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:28:24.802042 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:24.802008 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-wdr7j_a103895f-e01d-4661-b522-73301acf6e22/payload-processing/0.log" Oct 30 13:28:24.984864 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:24.984836 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-wdr7j_a103895f-e01d-4661-b522-73301acf6e22/payload-processing/0.log" Oct 30 13:28:25.287589 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:25.287553 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-z6jh4/must-gather-27vzg"] Oct 30 13:28:25.288045 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:25.288025 5922 cpu_manager.go:414] "containerMap: removing container" podUID="f186ed68-13b4-4849-ab58-62a2d93339b3" containerName="revoke-keys" Oct 30 13:28:25.288137 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:25.288047 5922 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f186ed68-13b4-4849-ab58-62a2d93339b3" containerName="revoke-keys" Oct 30 13:28:25.288197 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:25.288164 5922 memory_manager.go:365] "RemoveStaleState removing state" podUID="f186ed68-13b4-4849-ab58-62a2d93339b3" containerName="revoke-keys" Oct 30 13:28:25.291464 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:25.291443 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z6jh4/must-gather-27vzg" Oct 30 13:28:25.294649 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:25.294622 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:25.307569 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:25.307544 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z6jh4\"/\"openshift-service-ca.crt\"" Oct 30 13:28:25.313332 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:25.313312 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7whjh\" (UniqueName: \"kubernetes.io/projected/5fa5c79c-1359-4bb2-8f5e-c1707cccf7f5-kube-api-access-7whjh\") pod \"must-gather-27vzg\" (UID: \"5fa5c79c-1359-4bb2-8f5e-c1707cccf7f5\") " pod="openshift-must-gather-z6jh4/must-gather-27vzg" Oct 30 13:28:25.313434 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:25.313382 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5fa5c79c-1359-4bb2-8f5e-c1707cccf7f5-must-gather-output\") pod \"must-gather-27vzg\" (UID: \"5fa5c79c-1359-4bb2-8f5e-c1707cccf7f5\") " pod="openshift-must-gather-z6jh4/must-gather-27vzg" Oct 30 13:28:25.318660 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:25.318638 5922 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z6jh4\"/\"kube-root-ca.crt\"" Oct 30 13:28:25.318770 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:25.318733 5922 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-z6jh4\"/\"default-dockercfg-hfvsn\"" Oct 30 13:28:25.320365 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:25.320343 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z6jh4/must-gather-27vzg"] Oct 30 13:28:25.406172 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:25.406149 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:25.414446 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:25.414424 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5fa5c79c-1359-4bb2-8f5e-c1707cccf7f5-must-gather-output\") pod \"must-gather-27vzg\" (UID: \"5fa5c79c-1359-4bb2-8f5e-c1707cccf7f5\") " pod="openshift-must-gather-z6jh4/must-gather-27vzg" Oct 30 13:28:25.414575 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:25.414477 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7whjh\" (UniqueName: \"kubernetes.io/projected/5fa5c79c-1359-4bb2-8f5e-c1707cccf7f5-kube-api-access-7whjh\") pod \"must-gather-27vzg\" (UID: \"5fa5c79c-1359-4bb2-8f5e-c1707cccf7f5\") " pod="openshift-must-gather-z6jh4/must-gather-27vzg" Oct 30 13:28:25.414802 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:25.414782 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5fa5c79c-1359-4bb2-8f5e-c1707cccf7f5-must-gather-output\") pod \"must-gather-27vzg\" (UID: \"5fa5c79c-1359-4bb2-8f5e-c1707cccf7f5\") " pod="openshift-must-gather-z6jh4/must-gather-27vzg" Oct 30 13:28:25.439006 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:25.438980 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7whjh\" (UniqueName: \"kubernetes.io/projected/5fa5c79c-1359-4bb2-8f5e-c1707cccf7f5-kube-api-access-7whjh\") pod \"must-gather-27vzg\" (UID: \"5fa5c79c-1359-4bb2-8f5e-c1707cccf7f5\") " pod="openshift-must-gather-z6jh4/must-gather-27vzg" Oct 30 13:28:25.503840 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:25.503820 5922 scope.go:122] "RemoveContainer" containerID="614b7bc07bcdc7704aefd89d6f4a812593afb57877404b91c94b61541d3d32e3" Oct 30 13:28:25.605439 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:25.605383 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z6jh4/must-gather-27vzg" Oct 30 13:28:25.903123 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:25.903070 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z6jh4/must-gather-27vzg"] Oct 30 13:28:25.905989 ip-10-0-137-21 kubenswrapper[5922]: W1030 13:28:25.905962 5922 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5fa5c79c_1359_4bb2_8f5e_c1707cccf7f5.slice/crio-a2d99339fce98af5b25d971d66ffa5e15a467e587d8d7efe90c6944ff34ccf84 WatchSource:0}: Error finding container a2d99339fce98af5b25d971d66ffa5e15a467e587d8d7efe90c6944ff34ccf84: Status 404 returned error can't find the container with id a2d99339fce98af5b25d971d66ffa5e15a467e587d8d7efe90c6944ff34ccf84 Oct 30 13:28:25.908286 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:25.908267 5922 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:28:26.658338 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:26.658310 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z6jh4/must-gather-27vzg" event={"ID":"5fa5c79c-1359-4bb2-8f5e-c1707cccf7f5","Type":"ContainerStarted","Data":"a2d99339fce98af5b25d971d66ffa5e15a467e587d8d7efe90c6944ff34ccf84"} Oct 30 13:28:27.665061 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:27.664558 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z6jh4/must-gather-27vzg" event={"ID":"5fa5c79c-1359-4bb2-8f5e-c1707cccf7f5","Type":"ContainerStarted","Data":"289ae59bc662495e70998fbd0778dff5b25fbf36347cb93105c5a40ce7387ace"} Oct 30 13:28:27.665061 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:27.664647 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z6jh4/must-gather-27vzg" event={"ID":"5fa5c79c-1359-4bb2-8f5e-c1707cccf7f5","Type":"ContainerStarted","Data":"e85514597e1e0f9877b28ce5b9c5039e5899526a8bbac00c8511bf803b798388"} Oct 30 13:28:27.714360 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:27.714309 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-z6jh4/must-gather-27vzg" podStartSLOduration=2.021341625 podStartE2EDuration="2.714294783s" totalImagesPullingTime="692.953158ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:28:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:28:27.712689045 +0000 UTC m=+2164.182104788" watchObservedRunningTime="2026-10-30 13:28:27.714294783 +0000 UTC m=+2164.183710529" Oct 30 13:28:30.318633 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:30.317247 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-s7fsp_50bec85b-5592-437d-aa6f-4899aeb494a5/global-pull-secret-syncer/0.log" Oct 30 13:28:30.963589 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:30.963549 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" Oct 30 13:28:31.502641 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:31.501576 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-21.ec2.internal_100438b9fc00ed25c10ae1227a1de159/haproxy/0.log" Oct 30 13:28:35.408782 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:35.408688 5922 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nqhp_3f38af1e-cd92-4cd8-a643-151b10b8da12/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:37.780313 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:37.780267 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-c755c8f75-7njmp_578b67e5-39d7-44dc-b8e6-39c364ba1c5c/authorino/0.log" Oct 30 13:28:37.879597 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:37.879568 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-dn2k2_4c349358-62be-42fe-a686-ec417fd3d841/manager/0.log" Oct 30 13:28:37.995470 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:37.995442 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-6v28d_8ab008c5-fd4d-42b2-8f7e-67b35e9aae15/manager/0.log" Oct 30 13:28:38.416867 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:38.416750 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-5rxtm_4480c6ba-b6e6-4f0d-bc25-439bb9dcde74/limitador/0.log" Oct 30 13:28:39.126198 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:39.126161 5922 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkn7x"] Oct 30 13:28:39.139630 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:39.139313 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkn7x" Oct 30 13:28:39.154651 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:39.153738 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkn7x"] Oct 30 13:28:39.273952 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:39.273913 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2599acbd-33f9-4c86-9e18-dbc96baca4a9-podres\") pod \"perf-node-gather-daemonset-rkn7x\" (UID: \"2599acbd-33f9-4c86-9e18-dbc96baca4a9\") " pod="openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkn7x" Oct 30 13:28:39.274157 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:39.274054 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2599acbd-33f9-4c86-9e18-dbc96baca4a9-proc\") pod \"perf-node-gather-daemonset-rkn7x\" (UID: \"2599acbd-33f9-4c86-9e18-dbc96baca4a9\") " pod="openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkn7x" Oct 30 13:28:39.274157 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:39.274104 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d6q6\" (UniqueName: \"kubernetes.io/projected/2599acbd-33f9-4c86-9e18-dbc96baca4a9-kube-api-access-6d6q6\") pod \"perf-node-gather-daemonset-rkn7x\" (UID: \"2599acbd-33f9-4c86-9e18-dbc96baca4a9\") " pod="openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkn7x" Oct 30 13:28:39.274157 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:39.274138 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2599acbd-33f9-4c86-9e18-dbc96baca4a9-sys\") pod \"perf-node-gather-daemonset-rkn7x\" (UID: \"2599acbd-33f9-4c86-9e18-dbc96baca4a9\") " pod="openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkn7x" Oct 30 13:28:39.274338 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:39.274303 5922 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2599acbd-33f9-4c86-9e18-dbc96baca4a9-lib-modules\") pod \"perf-node-gather-daemonset-rkn7x\" (UID: \"2599acbd-33f9-4c86-9e18-dbc96baca4a9\") " pod="openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkn7x" Oct 30 13:28:39.375773 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:39.375722 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2599acbd-33f9-4c86-9e18-dbc96baca4a9-proc\") pod \"perf-node-gather-daemonset-rkn7x\" (UID: \"2599acbd-33f9-4c86-9e18-dbc96baca4a9\") " pod="openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkn7x" Oct 30 13:28:39.375977 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:39.375783 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d6q6\" (UniqueName: \"kubernetes.io/projected/2599acbd-33f9-4c86-9e18-dbc96baca4a9-kube-api-access-6d6q6\") pod \"perf-node-gather-daemonset-rkn7x\" (UID: \"2599acbd-33f9-4c86-9e18-dbc96baca4a9\") " pod="openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkn7x" Oct 30 13:28:39.375977 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:39.375817 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2599acbd-33f9-4c86-9e18-dbc96baca4a9-sys\") pod \"perf-node-gather-daemonset-rkn7x\" (UID: \"2599acbd-33f9-4c86-9e18-dbc96baca4a9\") " pod="openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkn7x" Oct 30 13:28:39.375977 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:39.375871 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2599acbd-33f9-4c86-9e18-dbc96baca4a9-lib-modules\") pod \"perf-node-gather-daemonset-rkn7x\" (UID: \"2599acbd-33f9-4c86-9e18-dbc96baca4a9\") " pod="openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkn7x" Oct 30 13:28:39.375977 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:39.375918 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2599acbd-33f9-4c86-9e18-dbc96baca4a9-sys\") pod \"perf-node-gather-daemonset-rkn7x\" (UID: \"2599acbd-33f9-4c86-9e18-dbc96baca4a9\") " pod="openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkn7x" Oct 30 13:28:39.375977 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:39.375868 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2599acbd-33f9-4c86-9e18-dbc96baca4a9-proc\") pod \"perf-node-gather-daemonset-rkn7x\" (UID: \"2599acbd-33f9-4c86-9e18-dbc96baca4a9\") " pod="openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkn7x" Oct 30 13:28:39.376274 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:39.375998 5922 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2599acbd-33f9-4c86-9e18-dbc96baca4a9-podres\") pod \"perf-node-gather-daemonset-rkn7x\" (UID: \"2599acbd-33f9-4c86-9e18-dbc96baca4a9\") " pod="openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkn7x" Oct 30 13:28:39.376274 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:39.376004 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2599acbd-33f9-4c86-9e18-dbc96baca4a9-lib-modules\") pod \"perf-node-gather-daemonset-rkn7x\" (UID: \"2599acbd-33f9-4c86-9e18-dbc96baca4a9\") " pod="openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkn7x" Oct 30 13:28:39.376274 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:39.376074 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2599acbd-33f9-4c86-9e18-dbc96baca4a9-podres\") pod \"perf-node-gather-daemonset-rkn7x\" (UID: \"2599acbd-33f9-4c86-9e18-dbc96baca4a9\") " pod="openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkn7x" Oct 30 13:28:39.400716 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:39.400680 5922 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d6q6\" (UniqueName: \"kubernetes.io/projected/2599acbd-33f9-4c86-9e18-dbc96baca4a9-kube-api-access-6d6q6\") pod \"perf-node-gather-daemonset-rkn7x\" (UID: \"2599acbd-33f9-4c86-9e18-dbc96baca4a9\") " pod="openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkn7x" Oct 30 13:28:39.482940 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:39.482900 5922 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkn7x" Oct 30 13:28:39.809308 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:39.809269 5922 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkn7x"] Oct 30 13:28:40.785626 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:40.785573 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkn7x" event={"ID":"2599acbd-33f9-4c86-9e18-dbc96baca4a9","Type":"ContainerStarted","Data":"6b97b5d72a998884dcda3dc40947395821806be4004c0073a09415c48d72db7a"} Oct 30 13:28:40.786030 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:40.785661 5922 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkn7x" event={"ID":"2599acbd-33f9-4c86-9e18-dbc96baca4a9","Type":"ContainerStarted","Data":"2e1b179c65d999729b06f1bf58c8fec88afc27b8c09968d21471a49835840174"} Oct 30 13:28:40.786030 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:40.785803 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkn7x" Oct 30 13:28:40.829044 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:40.829000 5922 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkn7x" podStartSLOduration=1.828986985 podStartE2EDuration="1.828986985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:28:40.827417392 +0000 UTC m=+2177.296833147" watchObservedRunningTime="2026-10-30 13:28:40.828986985 +0000 UTC m=+2177.298402731" Oct 30 13:28:40.970608 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:40.970577 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-4tq96_dd3d929e-3197-4794-9b31-794427a55693/kube-state-metrics/0.log" Oct 30 13:28:41.077280 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:41.077201 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-4tq96_dd3d929e-3197-4794-9b31-794427a55693/kube-rbac-proxy-main/0.log" Oct 30 13:28:41.191697 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:41.191669 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-4tq96_dd3d929e-3197-4794-9b31-794427a55693/kube-rbac-proxy-self/0.log" Oct 30 13:28:41.313345 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:41.313310 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6d7f54bf5f-jkjtm_c44f520d-f602-40e0-94d4-23277e56d67e/metrics-server/0.log" Oct 30 13:28:41.453105 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:41.452945 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-spsvf_14c0845c-1947-44b0-ada5-4b68ff9974f2/monitoring-plugin/0.log" Oct 30 13:28:41.575074 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:41.575047 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c4jgj_8fd0fdb6-c254-446d-8832-5a2fb284f03d/node-exporter/0.log" Oct 30 13:28:41.688453 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:41.688427 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c4jgj_8fd0fdb6-c254-446d-8832-5a2fb284f03d/kube-rbac-proxy/0.log" Oct 30 13:28:41.802450 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:41.802377 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c4jgj_8fd0fdb6-c254-446d-8832-5a2fb284f03d/init-textfile/0.log" Oct 30 13:28:42.943596 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:42.943564 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_83be9513-9c26-43a6-b2bc-485b5f0e386e/prometheus/0.log" Oct 30 13:28:43.056838 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:43.056803 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_83be9513-9c26-43a6-b2bc-485b5f0e386e/config-reloader/0.log" Oct 30 13:28:43.171391 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:43.171357 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_83be9513-9c26-43a6-b2bc-485b5f0e386e/thanos-sidecar/0.log" Oct 30 13:28:43.288828 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:43.288751 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_83be9513-9c26-43a6-b2bc-485b5f0e386e/kube-rbac-proxy-web/0.log" Oct 30 13:28:43.407115 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:43.407082 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_83be9513-9c26-43a6-b2bc-485b5f0e386e/kube-rbac-proxy/0.log" Oct 30 13:28:43.522920 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:43.522889 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_83be9513-9c26-43a6-b2bc-485b5f0e386e/kube-rbac-proxy-thanos/0.log" Oct 30 13:28:43.636072 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:43.636002 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_83be9513-9c26-43a6-b2bc-485b5f0e386e/init-config-reloader/0.log" Oct 30 13:28:43.759400 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:43.759366 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-xxb2q_56928a4d-7923-46e6-97bb-6690514f89a4/prometheus-operator/0.log" Oct 30 13:28:43.868757 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:43.868728 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-xxb2q_56928a4d-7923-46e6-97bb-6690514f89a4/kube-rbac-proxy/0.log" Oct 30 13:28:44.107767 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:44.107740 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-76d946b6b9-5lvf8_b0992e81-44ba-41c3-8079-c801a2d40aa4/telemeter-client/0.log" Oct 30 13:28:44.222294 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:44.222267 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-76d946b6b9-5lvf8_b0992e81-44ba-41c3-8079-c801a2d40aa4/reload/0.log" Oct 30 13:28:44.337570 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:44.337540 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-76d946b6b9-5lvf8_b0992e81-44ba-41c3-8079-c801a2d40aa4/kube-rbac-proxy/0.log" Oct 30 13:28:46.801547 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:46.801518 5922 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkn7x" Oct 30 13:28:48.482722 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:48.482686 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-74d99cdf4-q476v_f7e5b498-e68d-4c37-afae-5ee9e28d0fca/console/0.log" Oct 30 13:28:52.444190 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:52.444156 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v7skv_51a542f5-796e-46b1-95bd-31751d50b52b/dns/0.log" Oct 30 13:28:52.561814 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:52.561774 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v7skv_51a542f5-796e-46b1-95bd-31751d50b52b/kube-rbac-proxy/0.log" Oct 30 13:28:52.799056 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:52.798982 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9szkr_fe19a15d-5112-4b76-a226-4b8462e2fb33/dns-node-resolver/0.log" Oct 30 13:28:53.558329 ip-10-0-137-21 kubenswrapper[5922]: E1030 13:28:53.558288 5922 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cfgs4" containerName="extractor" Oct 30 13:28:53.824490 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:53.824422 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d59dd5dc7-rc4c8_3108a560-3472-4283-b17e-2099fd70bd78/kube-auth-proxy/0.log" Oct 30 13:28:53.899199 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:53.899169 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-5x6r5_3584258d-ca3c-4574-a39e-a70cc78d0ce6/istio-proxy/0.log" Oct 30 13:28:54.049396 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:54.049368 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bcbg8_66172ef8-aaad-4875-b3b3-bd81b9e9ba59/node-ca/0.log" Oct 30 13:28:54.099407 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:54.099339 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-bmhqs_3577d48e-eeaf-4608-ac46-99d2667ea4c0/istio-proxy/0.log" Oct 30 13:28:54.172323 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:54.172284 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-cxwwl_c29858c0-d396-4092-9e19-b76069383a82/istio-proxy/0.log" Oct 30 13:28:54.545545 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:54.545514 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-rh4bz_bd96022a-32a7-4b5d-a59b-b3678075d1f6/payload-pre-processing/0.log" Oct 30 13:28:54.610690 ip-10-0-137-21 kubenswrapper[5922]: I1030 13:28:54.610663 5922 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-wdr7j_a103895f-e01d-4661-b522-73301acf6e22/payload-processing/0.log"