Oct 30 19:44:43.356657 ip-10-0-132-173 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 19:44:43.356667 ip-10-0-132-173 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 19:44:43.356674 ip-10-0-132-173 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 19:44:43.356900 ip-10-0-132-173 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 19:44:53.404252 ip-10-0-132-173 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 19:44:53.404266 ip-10-0-132-173 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot fb82e4d909ae4b6d8184120a39cd7047 -- Oct 30 19:46:36.495639 ip-10-0-132-173 systemd[1]: Starting Kubernetes Kubelet... Oct 30 19:46:36.877272 ip-10-0-132-173 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:46:36.877272 ip-10-0-132-173 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 19:46:36.877272 ip-10-0-132-173 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:46:36.877272 ip-10-0-132-173 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:46:36.882235 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882158 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:46:36.882235 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882235 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:46:36.882235 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882239 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:46:36.882324 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882242 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:46:36.882324 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882244 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:46:36.882324 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882247 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:46:36.882324 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882249 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:46:36.882324 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882252 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:46:36.882324 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882255 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:46:36.882324 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882257 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:46:36.882324 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882260 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:46:36.882324 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882262 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:46:36.882324 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882265 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:46:36.882324 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882270 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:46:36.882324 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882274 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:46:36.882324 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882278 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:46:36.882324 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882281 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:46:36.882324 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882289 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:46:36.882324 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882292 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:46:36.882324 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882295 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:46:36.882324 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882298 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:46:36.882324 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882300 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:46:36.882857 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882303 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:46:36.882857 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882305 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:46:36.882857 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882308 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:46:36.882857 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882310 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:46:36.882857 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882313 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:46:36.882857 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882316 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:46:36.882857 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882318 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:46:36.882857 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882321 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:46:36.882857 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882323 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:46:36.882857 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882326 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:46:36.882857 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882329 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:46:36.882857 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882331 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:46:36.882857 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882335 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:46:36.882857 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882338 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:46:36.882857 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882340 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:46:36.882857 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882342 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:46:36.882857 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882345 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:46:36.882857 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882348 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:46:36.882857 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882350 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:46:36.883293 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882353 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:46:36.883293 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882355 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:46:36.883293 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882357 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:46:36.883293 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882360 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:46:36.883293 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882362 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:46:36.883293 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882365 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:46:36.883293 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882368 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:46:36.883293 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882372 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:46:36.883293 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882375 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:46:36.883293 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882390 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:46:36.883293 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882393 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:46:36.883293 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882396 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:46:36.883293 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882399 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:46:36.883293 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882402 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:46:36.883293 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882405 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:46:36.883293 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882408 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:46:36.883293 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882411 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:46:36.883293 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882413 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:46:36.883293 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882416 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:46:36.883771 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882419 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:46:36.883771 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882422 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:46:36.883771 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882424 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:46:36.883771 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882426 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:46:36.883771 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882429 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:46:36.883771 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882432 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:46:36.883771 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882434 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:46:36.883771 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882437 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:46:36.883771 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882439 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:46:36.883771 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882442 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:46:36.883771 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882444 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:46:36.883771 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882447 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:46:36.883771 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882449 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:46:36.883771 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882452 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:46:36.883771 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882454 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:46:36.883771 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882457 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:46:36.883771 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882459 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:46:36.883771 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882462 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:46:36.883771 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882464 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:46:36.883771 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882467 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:46:36.884230 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882469 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:46:36.884230 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882472 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:46:36.884230 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882475 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:46:36.884230 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882477 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:46:36.884230 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882480 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:46:36.884230 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882482 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:46:36.884230 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882485 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:46:36.884230 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882488 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:46:36.884230 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882491 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:46:36.884230 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882493 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:46:36.884230 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882496 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:46:36.884230 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882499 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:46:36.884230 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882502 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:46:36.884230 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882506 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:46:36.884230 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882508 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:46:36.884230 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882511 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:46:36.884230 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882513 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:46:36.884230 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882515 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:46:36.884230 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882518 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:46:36.884701 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882520 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:46:36.884701 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882523 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:46:36.884701 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882525 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:46:36.884701 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882527 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:46:36.884701 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882530 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:46:36.884701 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882532 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:46:36.884701 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.882534 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:46:36.884701 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883297 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:46:36.884701 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883304 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:46:36.884701 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883307 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:46:36.884701 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883310 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:46:36.884701 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883313 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:46:36.884701 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883315 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:46:36.884701 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883318 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:46:36.884701 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883321 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:46:36.884701 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883323 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:46:36.884701 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883326 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:46:36.884701 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883330 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:46:36.884701 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883333 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:46:36.884701 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883337 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:46:36.885339 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883341 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:46:36.885339 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883344 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:46:36.885339 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883347 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:46:36.885339 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883350 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:46:36.885339 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883353 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:46:36.885339 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883356 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:46:36.885339 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883358 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:46:36.885339 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883361 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:46:36.885339 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883363 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:46:36.885339 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883366 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:46:36.885339 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883368 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:46:36.885339 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883371 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:46:36.885339 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883374 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:46:36.885339 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883391 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:46:36.885339 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883394 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:46:36.885339 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883396 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:46:36.885339 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883399 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:46:36.885339 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883415 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:46:36.885339 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883419 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:46:36.885889 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883422 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:46:36.885889 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883425 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:46:36.885889 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883428 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:46:36.885889 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883431 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:46:36.885889 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883433 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:46:36.885889 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883436 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:46:36.885889 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883438 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:46:36.885889 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883441 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:46:36.885889 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883443 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:46:36.885889 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883446 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:46:36.885889 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883449 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:46:36.885889 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883452 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:46:36.885889 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883455 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:46:36.885889 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883458 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:46:36.885889 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883461 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:46:36.885889 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883464 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:46:36.885889 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883467 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:46:36.885889 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883469 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:46:36.885889 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883472 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:46:36.886447 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883474 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:46:36.886447 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883477 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:46:36.886447 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883479 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:46:36.886447 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883482 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:46:36.886447 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883484 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:46:36.886447 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883486 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:46:36.886447 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883489 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:46:36.886447 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883492 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:46:36.886447 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883494 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:46:36.886447 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883497 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:46:36.886447 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883499 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:46:36.886447 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883502 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:46:36.886447 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883506 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:46:36.886447 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883510 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:46:36.886447 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883512 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:46:36.886447 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883515 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:46:36.886447 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883517 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:46:36.886447 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883520 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:46:36.886447 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883522 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:46:36.886447 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883525 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:46:36.886951 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883527 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:46:36.886951 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883529 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:46:36.886951 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883531 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:46:36.886951 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883535 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:46:36.886951 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883538 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:46:36.886951 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883540 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:46:36.886951 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883545 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:46:36.886951 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883547 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:46:36.886951 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883550 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:46:36.886951 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883553 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:46:36.886951 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883555 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:46:36.886951 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883558 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:46:36.886951 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883561 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:46:36.886951 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883563 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:46:36.886951 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883566 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:46:36.886951 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883568 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:46:36.886951 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883571 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:46:36.886951 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883573 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:46:36.886951 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883575 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:46:36.887389 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883578 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:46:36.887389 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883581 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:46:36.887389 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883583 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:46:36.887389 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883586 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:46:36.887389 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883589 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:46:36.887389 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883591 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:46:36.887389 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883594 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:46:36.887389 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883597 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:46:36.887389 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883599 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:46:36.887389 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883602 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:46:36.887389 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883604 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:46:36.887389 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883606 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:46:36.887389 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883609 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:46:36.887389 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883611 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:46:36.887389 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883614 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:46:36.887389 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.883617 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:46:36.887389 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886144 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:46:36.887389 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886163 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:46:36.887389 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886178 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:46:36.887389 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886310 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:46:36.887870 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886314 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:46:36.887870 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886318 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:46:36.887870 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886328 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:46:36.887870 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886334 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:46:36.887870 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886337 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:46:36.887870 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886340 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:46:36.887870 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886343 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:46:36.887870 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886346 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:46:36.887870 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886349 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:46:36.887870 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886351 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:46:36.887870 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886354 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:46:36.887870 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886356 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:46:36.887870 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886359 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:46:36.887870 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886362 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:46:36.887870 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886364 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:46:36.887870 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886367 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:46:36.887870 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886371 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:46:36.887870 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886393 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:46:36.887870 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886397 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:46:36.888313 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886400 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:46:36.888313 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886403 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:46:36.888313 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886405 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:46:36.888313 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886408 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:46:36.888313 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886410 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:46:36.888313 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886413 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:46:36.888313 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886415 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:46:36.888313 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886417 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:46:36.888313 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886420 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:46:36.888313 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886423 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:46:36.888313 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886425 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:46:36.888313 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886428 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:46:36.888313 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886431 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:46:36.888313 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886433 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:46:36.888313 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886436 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:46:36.888313 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886439 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:46:36.888313 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886442 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:46:36.888313 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886445 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:46:36.888313 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886447 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:46:36.888313 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886450 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:46:36.888782 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886452 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:46:36.888782 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886455 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:46:36.888782 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886458 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:46:36.888782 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886460 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:46:36.888782 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886463 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:46:36.888782 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886466 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:46:36.888782 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886468 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:46:36.888782 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886471 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:46:36.888782 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886473 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:46:36.888782 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886476 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:46:36.888782 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886479 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:46:36.888782 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886484 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:46:36.888782 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886487 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:46:36.888782 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886489 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:46:36.888782 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886492 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:46:36.888782 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886494 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:46:36.888782 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886497 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:46:36.888782 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886499 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:46:36.888782 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886502 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:46:36.888782 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886505 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:46:36.889240 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886507 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:46:36.889240 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886510 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:46:36.889240 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886513 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:46:36.889240 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886515 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:46:36.889240 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886518 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:46:36.889240 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886520 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:46:36.889240 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886523 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:46:36.889240 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886526 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:46:36.889240 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886528 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:46:36.889240 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886531 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:46:36.889240 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886533 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:46:36.889240 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886536 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:46:36.889240 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886538 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:46:36.889240 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886540 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:46:36.889240 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886543 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:46:36.889240 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886546 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:46:36.889240 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886549 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:46:36.889240 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886552 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:46:36.889240 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886554 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:46:36.889240 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886557 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:46:36.889729 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886560 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:46:36.889729 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886563 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:46:36.889729 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886565 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:46:36.889729 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886568 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:46:36.889729 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886571 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:46:36.889729 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886573 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:46:36.889729 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886576 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:46:36.889729 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886578 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:46:36.889729 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886582 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:46:36.889729 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886585 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:46:36.889729 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886587 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:46:36.889729 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886590 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:46:36.889729 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886593 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:46:36.889729 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886595 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:46:36.889729 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886597 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:46:36.889729 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886600 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:46:36.889729 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886602 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:46:36.889729 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886605 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:46:36.889729 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886608 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:46:36.890184 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886610 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:46:36.890184 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886613 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:46:36.890184 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886615 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:46:36.890184 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.886618 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:46:36.890184 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887548 2578 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 19:46:36.890184 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887556 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 19:46:36.890184 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887564 2578 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 19:46:36.890184 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887572 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 19:46:36.890184 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887577 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 19:46:36.890184 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887580 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 19:46:36.890184 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887584 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 19:46:36.890184 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887589 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 19:46:36.890184 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887592 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 19:46:36.890184 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887595 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 19:46:36.890184 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887599 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 19:46:36.890184 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887603 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 19:46:36.890184 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887606 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 19:46:36.890184 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887609 2578 flags.go:64] FLAG: --cgroup-root="" Oct 30 19:46:36.890184 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887612 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 19:46:36.890184 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887615 2578 flags.go:64] FLAG: --client-ca-file="" Oct 30 19:46:36.890184 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887618 2578 flags.go:64] FLAG: --cloud-provider="external" Oct 30 19:46:36.890184 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887620 2578 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 19:46:36.890184 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887625 2578 flags.go:64] FLAG: --cluster-domain="" Oct 30 19:46:36.890715 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887628 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 19:46:36.890715 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887631 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 19:46:36.890715 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887634 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 19:46:36.890715 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887637 2578 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 19:46:36.890715 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887641 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 19:46:36.890715 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887644 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 19:46:36.890715 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887647 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 19:46:36.890715 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887650 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 19:46:36.890715 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887653 2578 flags.go:64] FLAG: --contention-profiling="false" Oct 30 19:46:36.890715 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887656 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 19:46:36.890715 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887659 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 19:46:36.890715 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887662 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 19:46:36.890715 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887666 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 19:46:36.890715 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887670 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 19:46:36.890715 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887672 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 19:46:36.890715 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887675 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 19:46:36.890715 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887678 2578 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 19:46:36.890715 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887681 2578 flags.go:64] FLAG: --enable-server="true" Oct 30 19:46:36.890715 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887684 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 19:46:36.890715 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887689 2578 flags.go:64] FLAG: --event-burst="100" Oct 30 19:46:36.890715 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887692 2578 flags.go:64] FLAG: --event-qps="50" Oct 30 19:46:36.890715 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887695 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 19:46:36.890715 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887698 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 19:46:36.890715 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887701 2578 flags.go:64] FLAG: --eviction-hard="" Oct 30 19:46:36.891320 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887706 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 19:46:36.891320 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887709 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 19:46:36.891320 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887712 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 19:46:36.891320 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887715 2578 flags.go:64] FLAG: --eviction-soft="" Oct 30 19:46:36.891320 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887718 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 19:46:36.891320 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887721 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 19:46:36.891320 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887724 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 19:46:36.891320 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887727 2578 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 19:46:36.891320 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887730 2578 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 19:46:36.891320 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887733 2578 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 19:46:36.891320 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887735 2578 flags.go:64] FLAG: --feature-gates="" Oct 30 19:46:36.891320 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887739 2578 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 19:46:36.891320 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887742 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 19:46:36.891320 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887745 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 19:46:36.891320 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887749 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 19:46:36.891320 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887752 2578 flags.go:64] FLAG: --healthz-port="10248" Oct 30 19:46:36.891320 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887755 2578 flags.go:64] FLAG: --help="false" Oct 30 19:46:36.891320 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887758 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-132-173.ec2.internal" Oct 30 19:46:36.891320 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887762 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 19:46:36.891320 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887765 2578 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 19:46:36.891320 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887768 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 19:46:36.891320 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887771 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 19:46:36.891320 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887775 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 19:46:36.891855 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887778 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 19:46:36.891855 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887781 2578 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 19:46:36.891855 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887784 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 19:46:36.891855 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887787 2578 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 19:46:36.891855 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887790 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 19:46:36.891855 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887793 2578 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 19:46:36.891855 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887796 2578 flags.go:64] FLAG: --kube-reserved="" Oct 30 19:46:36.891855 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887799 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 19:46:36.891855 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887801 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 19:46:36.891855 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887805 2578 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 19:46:36.891855 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887814 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 19:46:36.891855 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887817 2578 flags.go:64] FLAG: --lock-file="" Oct 30 19:46:36.891855 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887820 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 19:46:36.891855 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887823 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 19:46:36.891855 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887826 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 19:46:36.891855 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887837 2578 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 19:46:36.891855 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887840 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 19:46:36.891855 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887843 2578 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 19:46:36.891855 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887846 2578 flags.go:64] FLAG: --logging-format="text" Oct 30 19:46:36.891855 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887849 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 19:46:36.891855 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887852 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 19:46:36.891855 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887855 2578 flags.go:64] FLAG: --manifest-url="" Oct 30 19:46:36.891855 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887857 2578 flags.go:64] FLAG: --manifest-url-header="" Oct 30 19:46:36.891855 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887862 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 19:46:36.891855 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887865 2578 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 19:46:36.892435 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887868 2578 flags.go:64] FLAG: --max-pods="110" Oct 30 19:46:36.892435 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887871 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 19:46:36.892435 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887874 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 19:46:36.892435 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887878 2578 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 19:46:36.892435 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887880 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 19:46:36.892435 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887885 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 19:46:36.892435 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887888 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 19:46:36.892435 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887891 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 19:46:36.892435 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887899 2578 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 19:46:36.892435 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887902 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 19:46:36.892435 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887908 2578 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 19:46:36.892435 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887911 2578 flags.go:64] FLAG: --pod-cidr="" Oct 30 19:46:36.892435 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887914 2578 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 19:46:36.892435 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887916 2578 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 19:46:36.892435 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887919 2578 flags.go:64] FLAG: --pods-per-core="0" Oct 30 19:46:36.892435 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887922 2578 flags.go:64] FLAG: --port="10250" Oct 30 19:46:36.892435 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887925 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 19:46:36.892435 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887928 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06712a67f28d34232" Oct 30 19:46:36.892435 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887931 2578 flags.go:64] FLAG: --qos-reserved="" Oct 30 19:46:36.892435 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887935 2578 flags.go:64] FLAG: --read-only-port="10255" Oct 30 19:46:36.892435 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887938 2578 flags.go:64] FLAG: --register-node="true" Oct 30 19:46:36.892435 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887940 2578 flags.go:64] FLAG: --register-with-taints="" Oct 30 19:46:36.892435 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887944 2578 flags.go:64] FLAG: --registry-burst="10" Oct 30 19:46:36.892435 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887947 2578 flags.go:64] FLAG: --registry-qps="5" Oct 30 19:46:36.892435 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887950 2578 flags.go:64] FLAG: --reserved-cpus="" Oct 30 19:46:36.892435 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887953 2578 flags.go:64] FLAG: --reserved-memory="" Oct 30 19:46:36.893030 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887957 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 19:46:36.893030 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887960 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 19:46:36.893030 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887963 2578 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 19:46:36.893030 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887965 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 19:46:36.893030 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887968 2578 flags.go:64] FLAG: --runonce="false" Oct 30 19:46:36.893030 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887971 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 19:46:36.893030 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887974 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 19:46:36.893030 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887977 2578 flags.go:64] FLAG: --seccomp-default="false" Oct 30 19:46:36.893030 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887980 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 19:46:36.893030 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887983 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 19:46:36.893030 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887986 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 19:46:36.893030 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887989 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 19:46:36.893030 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887992 2578 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 19:46:36.893030 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887994 2578 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 19:46:36.893030 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.887997 2578 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 19:46:36.893030 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.888001 2578 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 19:46:36.893030 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.888005 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 19:46:36.893030 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.888008 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 19:46:36.893030 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.888011 2578 flags.go:64] FLAG: --system-cgroups="" Oct 30 19:46:36.893030 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.888013 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 19:46:36.893030 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.888019 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 19:46:36.893030 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.888022 2578 flags.go:64] FLAG: --tls-cert-file="" Oct 30 19:46:36.893030 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.888025 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 19:46:36.893030 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.888029 2578 flags.go:64] FLAG: --tls-min-version="" Oct 30 19:46:36.893030 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.888032 2578 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 19:46:36.893614 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.888036 2578 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 19:46:36.893614 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.888039 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 19:46:36.893614 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.888042 2578 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 19:46:36.893614 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.888045 2578 flags.go:64] FLAG: --v="2" Oct 30 19:46:36.893614 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.888050 2578 flags.go:64] FLAG: --version="false" Oct 30 19:46:36.893614 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.888053 2578 flags.go:64] FLAG: --vmodule="" Oct 30 19:46:36.893614 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.888057 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 19:46:36.893614 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.888060 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 19:46:36.893614 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888199 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:46:36.893614 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888203 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:46:36.893614 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888205 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:46:36.893614 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888208 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:46:36.893614 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888210 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:46:36.893614 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888213 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:46:36.893614 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888216 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:46:36.893614 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888219 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:46:36.893614 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888221 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:46:36.893614 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888223 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:46:36.893614 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888226 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:46:36.893614 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888229 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:46:36.893614 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888231 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:46:36.893614 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888234 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:46:36.894142 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888236 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:46:36.894142 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888240 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:46:36.894142 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888242 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:46:36.894142 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888245 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:46:36.894142 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888247 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:46:36.894142 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888250 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:46:36.894142 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888252 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:46:36.894142 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888255 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:46:36.894142 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888257 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:46:36.894142 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888260 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:46:36.894142 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888262 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:46:36.894142 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888265 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:46:36.894142 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888268 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:46:36.894142 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888271 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:46:36.894142 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888273 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:46:36.894142 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888275 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:46:36.894142 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888278 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:46:36.894142 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888280 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:46:36.894142 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888283 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:46:36.894844 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888285 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:46:36.894844 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888288 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:46:36.894844 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888290 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:46:36.894844 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888292 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:46:36.894844 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888295 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:46:36.894844 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888297 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:46:36.894844 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888300 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:46:36.894844 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888302 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:46:36.894844 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888305 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:46:36.894844 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888307 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:46:36.894844 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888310 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:46:36.894844 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888312 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:46:36.894844 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888315 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:46:36.894844 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888317 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:46:36.894844 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888321 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:46:36.894844 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888324 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:46:36.894844 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888327 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:46:36.894844 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888329 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:46:36.894844 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888331 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:46:36.895676 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888334 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:46:36.895676 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888336 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:46:36.895676 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888339 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:46:36.895676 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888341 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:46:36.895676 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888344 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:46:36.895676 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888346 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:46:36.895676 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888349 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:46:36.895676 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888352 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:46:36.895676 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888354 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:46:36.895676 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888358 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:46:36.895676 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888361 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:46:36.895676 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888363 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:46:36.895676 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888366 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:46:36.895676 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888368 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:46:36.895676 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888371 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:46:36.895676 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888374 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:46:36.895676 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888389 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:46:36.895676 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888394 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:46:36.895676 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888398 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:46:36.896525 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888402 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:46:36.896525 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888406 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:46:36.896525 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888408 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:46:36.896525 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888411 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:46:36.896525 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888413 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:46:36.896525 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888416 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:46:36.896525 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888418 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:46:36.896525 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888420 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:46:36.896525 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888424 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:46:36.896525 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888427 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:46:36.896525 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888431 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:46:36.896525 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888435 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:46:36.896525 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888438 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:46:36.896525 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888441 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:46:36.896525 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888444 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:46:36.896525 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888447 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:46:36.896525 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888450 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:46:36.896525 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888452 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:46:36.896525 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888455 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:46:36.897301 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888459 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:46:36.897301 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888463 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:46:36.897301 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888465 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:46:36.897301 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888468 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:46:36.897301 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888471 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:46:36.897301 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888473 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:46:36.897301 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888476 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:46:36.897301 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888478 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:46:36.897301 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888481 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:46:36.897301 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888483 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:46:36.897301 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888486 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:46:36.897301 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888488 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:46:36.897301 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888491 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:46:36.897301 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888493 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:46:36.897301 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888496 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:46:36.897301 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888499 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:46:36.897994 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.888526 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:46:36.897994 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888562 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:46:36.897994 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888566 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:46:36.897994 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888569 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:46:36.897994 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888572 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:46:36.897994 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888576 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:46:36.897994 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888579 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:46:36.897994 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888581 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:46:36.897994 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888584 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:46:36.897994 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888586 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:46:36.897994 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888589 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:46:36.897994 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888592 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:46:36.897994 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888596 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:46:36.897994 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888598 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:46:36.897994 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888601 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:46:36.897994 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888604 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:46:36.898690 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888606 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:46:36.898690 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888610 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:46:36.898690 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888613 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:46:36.898690 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888616 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:46:36.898690 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888619 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:46:36.898690 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888622 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:46:36.898690 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888625 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:46:36.898690 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888627 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:46:36.898690 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888629 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:46:36.898690 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888632 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:46:36.898690 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888635 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:46:36.898690 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888637 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:46:36.898690 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888640 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:46:36.898690 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888642 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:46:36.898690 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888645 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:46:36.898690 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888648 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:46:36.898690 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888652 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:46:36.898690 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888655 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:46:36.898690 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888658 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:46:36.899207 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888661 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:46:36.899207 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888664 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:46:36.899207 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888667 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:46:36.899207 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888670 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:46:36.899207 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888673 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:46:36.899207 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888675 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:46:36.899207 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888678 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:46:36.899207 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888680 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:46:36.899207 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888683 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:46:36.899207 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888686 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:46:36.899207 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888688 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:46:36.899207 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888691 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:46:36.899207 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888693 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:46:36.899207 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888695 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:46:36.899207 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888699 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:46:36.899207 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888702 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:46:36.899207 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888706 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:46:36.899207 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888709 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:46:36.899207 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888711 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:46:36.899207 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888713 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:46:36.899823 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888716 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:46:36.899823 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888719 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:46:36.899823 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888721 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:46:36.899823 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888724 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:46:36.899823 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888726 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:46:36.899823 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888729 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:46:36.899823 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888731 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:46:36.899823 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888734 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:46:36.899823 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888737 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:46:36.899823 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888739 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:46:36.899823 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888741 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:46:36.899823 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888744 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:46:36.899823 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888746 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:46:36.899823 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888749 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:46:36.899823 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888753 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:46:36.899823 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888755 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:46:36.899823 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888757 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:46:36.899823 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888760 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:46:36.899823 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888762 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:46:36.899823 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888765 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:46:36.900558 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888767 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:46:36.900558 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888770 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:46:36.900558 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888772 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:46:36.900558 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888775 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:46:36.900558 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888777 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:46:36.900558 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888780 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:46:36.900558 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888784 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:46:36.900558 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888786 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:46:36.900558 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888789 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:46:36.900558 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888792 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:46:36.900558 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888795 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:46:36.900558 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888798 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:46:36.900558 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888801 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:46:36.900558 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888804 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:46:36.900558 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888807 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:46:36.900558 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888809 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:46:36.900558 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888812 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:46:36.900558 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888814 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:46:36.900558 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888817 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:46:36.901023 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888819 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:46:36.901023 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888822 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:46:36.901023 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888824 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:46:36.901023 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888827 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:46:36.901023 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888829 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:46:36.901023 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888832 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:46:36.901023 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888834 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:46:36.901023 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888837 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:46:36.901023 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888840 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:46:36.901023 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888842 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:46:36.901023 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888845 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:46:36.901023 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888847 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:46:36.901023 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.888850 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:46:36.901023 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.896674 2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 19:46:36.901023 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.896692 2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 19:46:36.901023 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.896707 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 19:46:36.901023 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.896715 2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 19:46:36.901023 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.896806 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:46:36.901023 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.896816 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:46:36.901486 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.896820 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:46:36.901486 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.896826 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:46:36.901486 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.896831 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:46:36.901486 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.896836 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:46:36.901486 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.896840 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:46:36.901486 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.896845 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:46:36.901486 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.896849 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:46:36.901486 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.896853 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:46:36.901486 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.896857 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:46:36.901486 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.896862 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:46:36.901486 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.896866 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:46:36.901486 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.896870 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:46:36.901486 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.896874 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:46:36.901486 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.896879 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:46:36.901486 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.896883 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:46:36.901486 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.896887 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:46:36.901486 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.896892 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:46:36.901486 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.896896 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:46:36.901486 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.896900 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:46:36.902065 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.896905 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:46:36.902065 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.896909 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:46:36.902065 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.896913 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:46:36.902065 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.896917 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:46:36.902065 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.896922 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:46:36.902065 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.896927 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:46:36.902065 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.896931 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:46:36.902065 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.896935 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:46:36.902065 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.896941 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:46:36.902065 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.896945 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:46:36.902065 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.896949 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:46:36.902065 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.896953 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:46:36.902065 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.896958 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:46:36.902065 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.896962 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:46:36.902065 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.896966 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:46:36.902065 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.896970 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:46:36.902065 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.896974 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:46:36.902065 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.896978 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:46:36.902065 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.896982 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:46:36.902065 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.896987 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:46:36.902677 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.896991 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:46:36.902677 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.896996 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:46:36.902677 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897000 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:46:36.902677 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897004 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:46:36.902677 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897008 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:46:36.902677 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897013 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:46:36.902677 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897017 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:46:36.902677 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897023 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:46:36.902677 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897030 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:46:36.902677 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897034 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:46:36.902677 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897039 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:46:36.902677 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897043 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:46:36.902677 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897047 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:46:36.902677 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897052 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:46:36.902677 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897056 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:46:36.902677 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897060 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:46:36.902677 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897064 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:46:36.902677 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897069 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:46:36.902677 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897073 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:46:36.903155 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897077 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:46:36.903155 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897081 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:46:36.903155 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897086 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:46:36.903155 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897091 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:46:36.903155 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897095 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:46:36.903155 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897099 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:46:36.903155 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897103 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:46:36.903155 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897107 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:46:36.903155 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897112 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:46:36.903155 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897116 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:46:36.903155 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897120 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:46:36.903155 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897124 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:46:36.903155 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897129 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:46:36.903155 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897133 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:46:36.903155 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897137 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:46:36.903155 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897141 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:46:36.903155 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897145 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:46:36.903155 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897149 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:46:36.903155 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897155 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:46:36.903787 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897162 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:46:36.903787 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897168 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:46:36.903787 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897172 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:46:36.903787 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897177 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:46:36.903787 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897182 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:46:36.903787 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897186 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:46:36.903787 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897191 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:46:36.903787 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897195 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:46:36.903787 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897200 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:46:36.903787 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897203 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:46:36.903787 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897208 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:46:36.903787 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897212 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:46:36.903787 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897216 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:46:36.903787 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897220 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:46:36.903787 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897225 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:46:36.903787 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897229 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:46:36.903787 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897234 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:46:36.903787 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897238 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:46:36.903787 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897243 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:46:36.903787 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897247 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:46:36.904334 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897251 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:46:36.904334 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897255 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:46:36.904334 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897259 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:46:36.904334 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897263 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:46:36.904334 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897267 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:46:36.904334 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897272 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:46:36.904334 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897276 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:46:36.904334 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.897313 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:46:36.904334 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897602 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:46:36.904334 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897618 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:46:36.904334 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897624 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:46:36.904334 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897628 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:46:36.904334 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897633 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:46:36.904334 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897638 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:46:36.904334 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897643 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:46:36.904334 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897647 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:46:36.904904 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897651 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:46:36.904904 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897655 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:46:36.904904 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897660 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:46:36.904904 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897664 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:46:36.904904 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897668 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:46:36.904904 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897673 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:46:36.904904 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897677 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:46:36.904904 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897682 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:46:36.904904 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897686 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:46:36.904904 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897690 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:46:36.904904 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897694 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:46:36.904904 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897699 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:46:36.904904 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897703 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:46:36.904904 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897708 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:46:36.904904 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897712 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:46:36.904904 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897717 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:46:36.904904 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897721 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:46:36.904904 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897725 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:46:36.904904 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897730 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:46:36.904904 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897734 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:46:36.905400 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897738 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:46:36.905400 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897743 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:46:36.905400 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897747 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:46:36.905400 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897751 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:46:36.905400 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897758 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:46:36.905400 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897765 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:46:36.905400 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897773 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:46:36.905400 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897778 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:46:36.905400 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897782 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:46:36.905400 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897787 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:46:36.905400 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897791 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:46:36.905400 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897795 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:46:36.905400 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897800 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:46:36.905400 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897804 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:46:36.905400 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897808 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:46:36.905400 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897812 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:46:36.905400 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897817 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:46:36.905400 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897821 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:46:36.905400 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897825 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:46:36.905882 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897829 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:46:36.905882 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897833 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:46:36.905882 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897837 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:46:36.905882 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897841 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:46:36.905882 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897846 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:46:36.905882 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897850 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:46:36.905882 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897854 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:46:36.905882 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897859 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:46:36.905882 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897863 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:46:36.905882 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897867 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:46:36.905882 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897871 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:46:36.905882 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897875 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:46:36.905882 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897880 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:46:36.905882 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897884 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:46:36.905882 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897888 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:46:36.905882 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897892 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:46:36.905882 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897896 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:46:36.905882 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897900 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:46:36.905882 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897905 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:46:36.906374 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897909 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:46:36.906374 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897913 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:46:36.906374 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897917 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:46:36.906374 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897921 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:46:36.906374 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897925 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:46:36.906374 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897930 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:46:36.906374 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897934 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:46:36.906374 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897938 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:46:36.906374 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897942 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:46:36.906374 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897946 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:46:36.906374 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897950 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:46:36.906374 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897955 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:46:36.906374 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897959 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:46:36.906374 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897963 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:46:36.906374 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897967 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:46:36.906374 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897972 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:46:36.906374 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897976 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:46:36.906374 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897980 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:46:36.906374 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897984 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:46:36.906374 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897988 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:46:36.906863 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897992 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:46:36.906863 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.897997 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:46:36.906863 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898001 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:46:36.906863 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898005 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:46:36.906863 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898008 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:46:36.906863 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898012 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:46:36.906863 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898016 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:46:36.906863 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898020 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:46:36.906863 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898024 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:46:36.906863 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898028 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:46:36.906863 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898032 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:46:36.906863 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898036 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:46:36.906863 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898040 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:46:36.906863 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898045 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:46:36.906863 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898049 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:46:36.906863 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898053 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:46:36.906863 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898057 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:46:36.906863 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898061 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:46:36.906863 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898065 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:46:36.907312 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898069 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:46:36.907312 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.898107 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:46:36.907312 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898170 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:46:36.907312 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898179 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:46:36.907312 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898184 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:46:36.907312 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898189 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:46:36.907312 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898194 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:46:36.907312 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898198 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:46:36.907312 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898202 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:46:36.907312 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898206 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:46:36.907312 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898210 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:46:36.907312 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898214 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:46:36.907312 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898219 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:46:36.907312 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898223 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:46:36.907312 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898227 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:46:36.907312 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898232 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:46:36.907956 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898235 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:46:36.907956 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898240 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:46:36.907956 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898243 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:46:36.907956 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898248 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:46:36.907956 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898252 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:46:36.907956 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898256 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:46:36.907956 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898260 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:46:36.907956 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898264 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:46:36.907956 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898268 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:46:36.907956 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898272 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:46:36.907956 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898277 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:46:36.907956 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898281 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:46:36.907956 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898285 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:46:36.907956 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898290 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:46:36.907956 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898294 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:46:36.907956 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898298 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:46:36.907956 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898304 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:46:36.907956 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898311 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:46:36.907956 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898317 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:46:36.908714 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898322 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:46:36.908714 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898327 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:46:36.908714 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898332 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:46:36.908714 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898336 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:46:36.908714 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898341 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:46:36.908714 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898344 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:46:36.908714 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898348 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:46:36.908714 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898353 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:46:36.908714 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898357 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:46:36.908714 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898361 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:46:36.908714 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898365 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:46:36.908714 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898369 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:46:36.908714 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898373 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:46:36.908714 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898393 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:46:36.908714 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898397 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:46:36.908714 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898401 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:46:36.908714 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898405 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:46:36.908714 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898409 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:46:36.908714 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898414 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:46:36.909459 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898417 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:46:36.909459 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898421 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:46:36.909459 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898425 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:46:36.909459 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898430 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:46:36.909459 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898438 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:46:36.909459 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898442 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:46:36.909459 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898447 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:46:36.909459 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898451 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:46:36.909459 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898455 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:46:36.909459 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898460 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:46:36.909459 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898464 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:46:36.909459 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898468 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:46:36.909459 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898472 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:46:36.909459 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898476 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:46:36.909459 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898480 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:46:36.909459 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898485 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:46:36.909459 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898490 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:46:36.909459 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898494 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:46:36.909459 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898498 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:46:36.909912 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898502 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:46:36.909912 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898507 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:46:36.909912 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898511 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:46:36.909912 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898515 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:46:36.909912 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898519 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:46:36.909912 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898523 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:46:36.909912 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898528 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:46:36.909912 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898532 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:46:36.909912 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898537 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:46:36.909912 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898541 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:46:36.909912 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898545 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:46:36.909912 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898549 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:46:36.909912 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898554 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:46:36.909912 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898558 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:46:36.909912 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898562 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:46:36.909912 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898567 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:46:36.909912 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898571 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:46:36.909912 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898577 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:46:36.909912 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898581 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:46:36.909912 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898585 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:46:36.910523 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898589 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:46:36.910523 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898593 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:46:36.910523 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898598 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:46:36.910523 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898602 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:46:36.910523 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898606 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:46:36.910523 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898610 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:46:36.910523 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898615 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:46:36.910523 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898619 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:46:36.910523 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898624 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:46:36.910523 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898628 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:46:36.910523 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898632 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:46:36.910523 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898636 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:46:36.910523 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898640 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:46:36.910523 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898644 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:46:36.910523 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:36.898648 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:46:36.910523 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.898728 2578 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 19:46:36.910523 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.904272 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 19:46:36.910523 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.905428 2578 server.go:1014] "Starting client certificate rotation" Oct 30 19:46:36.910523 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.905548 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 19:46:36.911268 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.905577 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 19:46:36.911268 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.908201 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:46:36.911268 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.911210 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:46:36.922867 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.922850 2578 log.go:25] "Validated CRI v1 runtime API" Oct 30 19:46:36.926836 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.926819 2578 log.go:25] "Validated CRI v1 image API" Oct 30 19:46:36.927646 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.927622 2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 19:46:36.930280 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.930260 2578 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7a701ec6-e7d6-469d-afe7-54d9727c599a:/dev/nvme0n1p4 968cc63f-7bdd-4a21-ac44-654993fbf10c:/dev/nvme0n1p3] Oct 30 19:46:36.930355 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.930280 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 19:46:36.936097 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.935991 2578 manager.go:217] Machine: {Timestamp:2026-10-30 19:46:36.933785583 +0000 UTC m=+0.356228713 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3199445 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25aaa95da08431c7fc2d9571a19a09 SystemUUID:ec25aaa9-5da0-8431-c7fc-2d9571a19a09 BootID:fb82e4d9-09ae-4b6d-8184-120a39cd7047 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c3:8c:38:d1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c3:8c:38:d1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:7e:a0:df:f3:42 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 19:46:36.936097 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.936096 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 19:46:36.936194 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.936169 2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 19:46:36.937415 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.937390 2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 19:46:36.937542 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.937417 2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-173.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 19:46:36.937581 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.937544 2578 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 19:46:36.937581 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.937551 2578 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 19:46:36.937640 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.937597 2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:46:36.937640 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.937622 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:46:36.938949 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.938939 2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 19:46:36.943641 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.943629 2578 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 19:46:36.943760 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.943751 2578 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 19:46:36.944538 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.944528 2578 kubelet.go:493] "Attempting to sync node with API server" Oct 30 19:46:36.944583 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.944540 2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 19:46:36.944583 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.944555 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 19:46:36.944583 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.944563 2578 kubelet.go:405] "Adding apiserver pod source" Oct 30 19:46:36.944583 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.944571 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 19:46:36.944894 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.944881 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:46:36.947276 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.947252 2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 19:46:36.954749 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.954738 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:46:36.955688 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.955676 2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 19:46:36.955728 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.955705 2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 19:46:36.956662 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.956652 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 19:46:36.956693 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.956669 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 19:46:36.956693 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.956675 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 19:46:36.956693 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.956680 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 19:46:36.956693 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.956686 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 19:46:36.956693 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.956692 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 19:46:36.956816 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.956700 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 19:46:36.956816 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.956705 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 19:46:36.956816 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.956712 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 19:46:36.956816 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.956718 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 19:46:36.956816 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.956727 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 19:46:36.956816 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.956735 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 19:46:36.957514 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.957496 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 19:46:36.957514 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.957515 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 19:46:36.961494 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.961474 2578 server.go:1263] "Started kubelet" Oct 30 19:46:36.961637 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.961587 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 19:46:36.961637 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.961591 2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 19:46:36.961754 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.961655 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 19:46:36.962229 ip-10-0-132-173 systemd[1]: Started Kubernetes Kubelet. Oct 30 19:46:36.963110 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.963093 2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 19:46:36.968675 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.968658 2578 server.go:317] "Adding debug handlers to kubelet server" Oct 30 19:46:36.968864 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:36.968833 2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 19:46:36.974280 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.974267 2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 19:46:36.974353 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.974288 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 19:46:36.974417 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:36.974357 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-173.ec2.internal\" not found" Oct 30 19:46:36.975147 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.975124 2578 factory.go:55] Registering systemd factory Oct 30 19:46:36.975147 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.975145 2578 factory.go:223] Registration of the systemd container factory successfully Oct 30 19:46:36.975147 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.975145 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 19:46:36.975392 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.975362 2578 factory.go:153] Registering CRI-O factory Oct 30 19:46:36.975392 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.975390 2578 factory.go:223] Registration of the crio container factory successfully Oct 30 19:46:36.975471 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.975453 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 19:46:36.975518 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.975472 2578 factory.go:103] Registering Raw factory Oct 30 19:46:36.975518 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.975489 2578 manager.go:1199] Started watching for new ooms in manager Oct 30 19:46:36.975569 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.975558 2578 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 19:46:36.975595 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.975571 2578 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 19:46:36.975698 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.975688 2578 reconstruct.go:97] "Volume reconstruction finished" Oct 30 19:46:36.975726 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.975700 2578 reconciler.go:29] "Reconciler: start to sync state" Oct 30 19:46:36.976462 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.976441 2578 manager.go:319] Starting recovery of all containers Oct 30 19:46:36.986230 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.986212 2578 manager.go:324] Recovery completed Oct 30 19:46:36.987892 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:36.987822 2578 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Oct 30 19:46:36.992229 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.992216 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:46:36.993964 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.993946 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:46:36.994023 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.993988 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:46:36.994023 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.993998 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:46:36.994493 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.994479 2578 cpu_manager.go:226] "Starting" policy="none" Oct 30 19:46:36.994493 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.994490 2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 19:46:36.994619 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.994488 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:46:36.994619 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.994508 2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 19:46:36.995172 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.995158 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:46:36.995230 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.995181 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:46:36.995230 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.995191 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:46:36.996814 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.996796 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-173.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 19:46:36.996896 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:36.996800 2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-173.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 19:46:36.996997 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.996983 2578 policy_none.go:50] "Start" Oct 30 19:46:36.997044 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.997003 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 19:46:36.997044 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.997016 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 19:46:36.997642 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.997623 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 19:46:36.998927 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:36.996830 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-173.ec2.internal.18e366562ec4a24b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-173.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-173.ec2.internal,},FirstTimestamp:2026-10-30 19:46:36.961448523 +0000 UTC m=+0.383891653,LastTimestamp:2026-10-30 19:46:36.961448523 +0000 UTC m=+0.383891653,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-173.ec2.internal,}" Oct 30 19:46:36.999169 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:36.999157 2578 policy_none.go:44] "Start" Oct 30 19:46:37.020876 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:37.020759 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 19:46:37.020948 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:37.020764 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-173.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 19:46:37.020948 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:37.020803 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 19:46:37.020948 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:37.020753 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-173.ec2.internal.18e3665630b4e76c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-173.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-173.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-173.ec2.internal,},FirstTimestamp:2026-10-30 19:46:36.993972076 +0000 UTC m=+0.416415207,LastTimestamp:2026-10-30 19:46:36.993972076 +0000 UTC m=+0.416415207,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-173.ec2.internal,}" Oct 30 19:46:37.026037 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.026004 2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 19:46:37.029277 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.029262 2578 manager.go:341] "Starting Device Plugin manager" Oct 30 19:46:37.052191 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:37.029294 2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 19:46:37.052191 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.029306 2578 server.go:87] "Starting device plugin registration server" Oct 30 19:46:37.052191 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.029585 2578 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 19:46:37.052191 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.029597 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 19:46:37.052191 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.029725 2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 19:46:37.052191 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.029847 2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 19:46:37.052191 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.029860 2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 19:46:37.052191 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.030245 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7hxqh" Oct 30 19:46:37.052191 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:37.030311 2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 19:46:37.052191 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:37.030353 2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-173.ec2.internal\" not found" Oct 30 19:46:37.052191 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.035228 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7hxqh" Oct 30 19:46:37.052191 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.045836 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 19:46:37.052191 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:37.045987 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-173.ec2.internal.18e3665630b53b07 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-173.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-173.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-173.ec2.internal,},FirstTimestamp:2026-10-30 19:46:36.993993479 +0000 UTC m=+0.416436608,LastTimestamp:2026-10-30 19:46:36.993993479 +0000 UTC m=+0.416436608,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-173.ec2.internal,}" Oct 30 19:46:37.052191 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.047041 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 19:46:37.052191 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.047074 2578 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 19:46:37.052191 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.047094 2578 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 19:46:37.052191 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:37.047132 2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 19:46:37.077250 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.077226 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:46:37.129978 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.129898 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:46:37.130896 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.130881 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:46:37.130980 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.130913 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:46:37.130980 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.130928 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:46:37.130980 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.130963 2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-173.ec2.internal" Oct 30 19:46:37.147450 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.147427 2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal"] Oct 30 19:46:37.147523 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.147494 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:46:37.148542 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.148527 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:46:37.148641 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.148558 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:46:37.148641 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.148573 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:46:37.150879 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.150865 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:46:37.151044 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.151029 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" Oct 30 19:46:37.151090 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.151060 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:46:37.151641 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.151616 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:46:37.151740 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.151628 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:46:37.151740 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.151657 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:46:37.151740 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.151673 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:46:37.151740 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.151675 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:46:37.151740 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.151694 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:46:37.153643 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.153628 2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-173.ec2.internal" Oct 30 19:46:37.153712 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:37.153650 2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-173.ec2.internal\": node \"ip-10-0-132-173.ec2.internal\" not found" Oct 30 19:46:37.153834 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.153818 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal" Oct 30 19:46:37.153899 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.153850 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:46:37.154701 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.154685 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:46:37.154779 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.154707 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:46:37.154779 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.154716 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:46:37.173845 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:37.173828 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-173.ec2.internal\" not found" node="ip-10-0-132-173.ec2.internal" Oct 30 19:46:37.175881 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:37.175867 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-173.ec2.internal\" not found" node="ip-10-0-132-173.ec2.internal" Oct 30 19:46:37.176611 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.176596 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6ad70715366d9db6ff14ef77cdc50313-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal\" (UID: \"6ad70715366d9db6ff14ef77cdc50313\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" Oct 30 19:46:37.176665 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.176620 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ad70715366d9db6ff14ef77cdc50313-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal\" (UID: \"6ad70715366d9db6ff14ef77cdc50313\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" Oct 30 19:46:37.176665 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.176636 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5b1dc20b8a7bd35e96a3f207cff8defc-config\") pod \"kube-apiserver-proxy-ip-10-0-132-173.ec2.internal\" (UID: \"5b1dc20b8a7bd35e96a3f207cff8defc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal" Oct 30 19:46:37.190533 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:37.190518 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-173.ec2.internal\" not found" Oct 30 19:46:37.277065 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.277036 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6ad70715366d9db6ff14ef77cdc50313-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal\" (UID: \"6ad70715366d9db6ff14ef77cdc50313\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" Oct 30 19:46:37.277065 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.277065 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ad70715366d9db6ff14ef77cdc50313-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal\" (UID: \"6ad70715366d9db6ff14ef77cdc50313\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" Oct 30 19:46:37.277225 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.277082 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5b1dc20b8a7bd35e96a3f207cff8defc-config\") pod \"kube-apiserver-proxy-ip-10-0-132-173.ec2.internal\" (UID: \"5b1dc20b8a7bd35e96a3f207cff8defc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal" Oct 30 19:46:37.277225 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.277126 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5b1dc20b8a7bd35e96a3f207cff8defc-config\") pod \"kube-apiserver-proxy-ip-10-0-132-173.ec2.internal\" (UID: \"5b1dc20b8a7bd35e96a3f207cff8defc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal" Oct 30 19:46:37.277225 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.277136 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6ad70715366d9db6ff14ef77cdc50313-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal\" (UID: \"6ad70715366d9db6ff14ef77cdc50313\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" Oct 30 19:46:37.277225 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.277136 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ad70715366d9db6ff14ef77cdc50313-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal\" (UID: \"6ad70715366d9db6ff14ef77cdc50313\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" Oct 30 19:46:37.291450 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:37.291431 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-173.ec2.internal\" not found" Oct 30 19:46:37.392327 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:37.392263 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-173.ec2.internal\" not found" Oct 30 19:46:37.478298 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.478277 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" Oct 30 19:46:37.481447 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.481430 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal" Oct 30 19:46:37.492753 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:37.492738 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-173.ec2.internal\" not found" Oct 30 19:46:37.593205 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:37.593169 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-173.ec2.internal\" not found" Oct 30 19:46:37.693657 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:37.693629 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-173.ec2.internal\" not found" Oct 30 19:46:37.794222 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:37.794186 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-173.ec2.internal\" not found" Oct 30 19:46:37.862888 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.862861 2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:46:37.874900 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:37.874878 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" Oct 30 19:46:39.202676 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.202652 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:46:39.204638 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.204605 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal" Oct 30 19:46:39.204772 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.204755 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 19:46:39.204913 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:39.204887 2578 kubelet.go:3370] "Failed creating a mirror pod" err="Post \"https://ab962f218708a4a3a94563c5003e2683-35dcd4391a080576.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/kube-system/pods\": read tcp 10.0.132.173:47992->3.134.56.84:6443: use of closed network connection" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal" Oct 30 19:46:39.204967 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.204906 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 19:46:39.244360 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.244306 2578 apiserver.go:52] "Watching apiserver" Oct 30 19:46:39.273983 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.273960 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 19:46:39.297764 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.296660 2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 19:46:39.297764 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.297331 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-7f4jl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal","openshift-network-operator/iptables-alerter-882qh","kube-system/global-pull-secret-syncer-hz6jn","kube-system/konnectivity-agent-djkzp","openshift-image-registry/node-ca-hkvn6","openshift-multus/multus-additional-cni-plugins-2hfrk","openshift-multus/multus-76qp7","openshift-network-diagnostics/network-check-target-kvw9t","openshift-ovn-kubernetes/ovnkube-node-xpjpn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn","openshift-cluster-node-tuning-operator/tuned-rggcq","openshift-dns/node-resolver-l9fpp"] Oct 30 19:46:39.300598 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.299727 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2hfrk" Oct 30 19:46:39.302479 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.302460 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-882qh" Oct 30 19:46:39.304684 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.304670 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hz6jn" Oct 30 19:46:39.304744 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:39.304722 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hz6jn" podUID="73bb0a1f-597a-4078-8a01-20d0ed24715b" Oct 30 19:46:39.307521 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.307504 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-djkzp" Oct 30 19:46:39.309505 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.309488 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hkvn6" Oct 30 19:46:39.311719 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.311549 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7f4jl" Oct 30 19:46:39.311719 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:39.311634 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7f4jl" podUID="57ddd2a0-f081-4f90-95a6-0f6f8df2f433" Oct 30 19:46:39.313912 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.313896 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.316089 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.316076 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvw9t" Oct 30 19:46:39.316154 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:39.316127 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvw9t" podUID="116a1241-c9a3-4664-8217-60c2b6aa8b76" Oct 30 19:46:39.319031 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.319011 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.320634 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.320601 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-n5k26\"" Oct 30 19:46:39.321012 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.320992 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 19:46:39.321090 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.321074 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 19:46:39.321090 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.321085 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 19:46:39.321173 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.321124 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 19:46:39.321173 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.321141 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 19:46:39.321173 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.321148 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 19:46:39.321954 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.321940 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" Oct 30 19:46:39.324191 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.324175 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:46:39.326882 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.326864 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l9fpp" Oct 30 19:46:39.340600 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.340543 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:41:38 +0000 UTC" deadline="2028-07-18 14:41:17.339517502 +0000 UTC" Oct 30 19:46:39.340600 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.340599 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15042h54m37.998922092s" Oct 30 19:46:39.342911 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.342896 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 19:46:39.342987 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.342941 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-lcqb4" Oct 30 19:46:39.365403 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:39.365376 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b1dc20b8a7bd35e96a3f207cff8defc.slice/crio-8ab94cd9a89ec8b7e8f48c21784a2d597264034894e149a4d80accb109d73fb2 WatchSource:0}: Error finding container 8ab94cd9a89ec8b7e8f48c21784a2d597264034894e149a4d80accb109d73fb2: Status 404 returned error can't find the container with id 8ab94cd9a89ec8b7e8f48c21784a2d597264034894e149a4d80accb109d73fb2 Oct 30 19:46:39.365913 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:39.365896 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ad70715366d9db6ff14ef77cdc50313.slice/crio-23213dbd30843595194cc488e41405260a4e82391bf97b3fec6e67059dd7cdc8 WatchSource:0}: Error finding container 23213dbd30843595194cc488e41405260a4e82391bf97b3fec6e67059dd7cdc8: Status 404 returned error can't find the container with id 23213dbd30843595194cc488e41405260a4e82391bf97b3fec6e67059dd7cdc8 Oct 30 19:46:39.370560 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.370545 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:46:39.375887 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.375873 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 19:46:39.381111 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.381086 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d808d253-3f30-48e8-918c-bd3880c2af8f-iptables-alerter-script\") pod \"iptables-alerter-882qh\" (UID: \"d808d253-3f30-48e8-918c-bd3880c2af8f\") " pod="openshift-network-operator/iptables-alerter-882qh" Oct 30 19:46:39.381192 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.381130 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/08b5a0ca-cfe1-4366-a404-5ebbaeb7c149-hosts-file\") pod \"node-resolver-l9fpp\" (UID: \"08b5a0ca-cfe1-4366-a404-5ebbaeb7c149\") " pod="openshift-dns/node-resolver-l9fpp" Oct 30 19:46:39.381192 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.381171 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-host-var-lib-cni-multus\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.381300 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.381199 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2e03cb52-fd85-4e35-87b4-38833cbc7d74-serviceca\") pod \"node-ca-hkvn6\" (UID: \"2e03cb52-fd85-4e35-87b4-38833cbc7d74\") " pod="openshift-image-registry/node-ca-hkvn6" Oct 30 19:46:39.381300 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.381228 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/688d079a-932d-4f63-97da-5cd6222a03e1-env-overrides\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.381300 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.381251 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-etc-sysctl-d\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:46:39.381300 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.381278 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1afac97f-a617-4113-b8e8-c668caed54f9-cni-binary-copy\") pod \"multus-additional-cni-plugins-2hfrk\" (UID: \"1afac97f-a617-4113-b8e8-c668caed54f9\") " pod="openshift-multus/multus-additional-cni-plugins-2hfrk" Oct 30 19:46:39.381474 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.381306 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1afac97f-a617-4113-b8e8-c668caed54f9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2hfrk\" (UID: \"1afac97f-a617-4113-b8e8-c668caed54f9\") " pod="openshift-multus/multus-additional-cni-plugins-2hfrk" Oct 30 19:46:39.381474 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.381340 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-host-run-netns\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.381474 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.381369 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-multus-conf-dir\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.381474 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.381394 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-systemd-units\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.381474 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.381428 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-sys\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:46:39.381474 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.381455 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1afac97f-a617-4113-b8e8-c668caed54f9-system-cni-dir\") pod \"multus-additional-cni-plugins-2hfrk\" (UID: \"1afac97f-a617-4113-b8e8-c668caed54f9\") " pod="openshift-multus/multus-additional-cni-plugins-2hfrk" Oct 30 19:46:39.382570 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.382548 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr5dn\" (UniqueName: \"kubernetes.io/projected/1afac97f-a617-4113-b8e8-c668caed54f9-kube-api-access-nr5dn\") pod \"multus-additional-cni-plugins-2hfrk\" (UID: \"1afac97f-a617-4113-b8e8-c668caed54f9\") " pod="openshift-multus/multus-additional-cni-plugins-2hfrk" Oct 30 19:46:39.382677 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.382592 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d808d253-3f30-48e8-918c-bd3880c2af8f-host-slash\") pod \"iptables-alerter-882qh\" (UID: \"d808d253-3f30-48e8-918c-bd3880c2af8f\") " pod="openshift-network-operator/iptables-alerter-882qh" Oct 30 19:46:39.382677 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.382642 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-multus-cni-dir\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.382746 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.382677 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94r98\" (UniqueName: \"kubernetes.io/projected/2e03cb52-fd85-4e35-87b4-38833cbc7d74-kube-api-access-94r98\") pod \"node-ca-hkvn6\" (UID: \"2e03cb52-fd85-4e35-87b4-38833cbc7d74\") " pod="openshift-image-registry/node-ca-hkvn6" Oct 30 19:46:39.382746 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.382724 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/688d079a-932d-4f63-97da-5cd6222a03e1-ovn-node-metrics-cert\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.382815 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.382788 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d1dca14b-4840-4dd6-a7cd-7d795adb9532-registration-dir\") pod \"aws-ebs-csi-driver-node-glksn\" (UID: \"d1dca14b-4840-4dd6-a7cd-7d795adb9532\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" Oct 30 19:46:39.382874 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.382860 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-etc-kubernetes\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.382928 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.382914 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5rr5\" (UniqueName: \"kubernetes.io/projected/66999038-807b-406b-9242-c68c04e521f3-kube-api-access-t5rr5\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.382986 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.382965 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtqp8\" (UniqueName: \"kubernetes.io/projected/d1dca14b-4840-4dd6-a7cd-7d795adb9532-kube-api-access-xtqp8\") pod \"aws-ebs-csi-driver-node-glksn\" (UID: \"d1dca14b-4840-4dd6-a7cd-7d795adb9532\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" Oct 30 19:46:39.383045 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.382994 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm7rk\" (UniqueName: \"kubernetes.io/projected/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-kube-api-access-cm7rk\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:46:39.383045 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.383034 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/13962162-194c-4e37-ac6b-b3bbc810cedb-konnectivity-ca\") pod \"konnectivity-agent-djkzp\" (UID: \"13962162-194c-4e37-ac6b-b3bbc810cedb\") " pod="kube-system/konnectivity-agent-djkzp" Oct 30 19:46:39.383150 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.383069 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-host-run-k8s-cni-cncf-io\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.383150 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.383091 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-host-var-lib-cni-bin\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.383240 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.383152 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lph46\" (UniqueName: \"kubernetes.io/projected/116a1241-c9a3-4664-8217-60c2b6aa8b76-kube-api-access-lph46\") pod \"network-check-target-kvw9t\" (UID: \"116a1241-c9a3-4664-8217-60c2b6aa8b76\") " pod="openshift-network-diagnostics/network-check-target-kvw9t" Oct 30 19:46:39.383240 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.383177 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e03cb52-fd85-4e35-87b4-38833cbc7d74-host\") pod \"node-ca-hkvn6\" (UID: \"2e03cb52-fd85-4e35-87b4-38833cbc7d74\") " pod="openshift-image-registry/node-ca-hkvn6" Oct 30 19:46:39.383240 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.383236 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-host-slash\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.383380 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.383287 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/13962162-194c-4e37-ac6b-b3bbc810cedb-agent-certs\") pod \"konnectivity-agent-djkzp\" (UID: \"13962162-194c-4e37-ac6b-b3bbc810cedb\") " pod="kube-system/konnectivity-agent-djkzp" Oct 30 19:46:39.383380 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.383317 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57ddd2a0-f081-4f90-95a6-0f6f8df2f433-metrics-certs\") pod \"network-metrics-daemon-7f4jl\" (UID: \"57ddd2a0-f081-4f90-95a6-0f6f8df2f433\") " pod="openshift-multus/network-metrics-daemon-7f4jl" Oct 30 19:46:39.383380 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.383349 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-log-socket\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.383520 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.383396 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-host-run-ovn-kubernetes\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.383520 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.383460 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/73bb0a1f-597a-4078-8a01-20d0ed24715b-dbus\") pod \"global-pull-secret-syncer-hz6jn\" (UID: \"73bb0a1f-597a-4078-8a01-20d0ed24715b\") " pod="kube-system/global-pull-secret-syncer-hz6jn" Oct 30 19:46:39.383520 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.383483 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb8sb\" (UniqueName: \"kubernetes.io/projected/57ddd2a0-f081-4f90-95a6-0f6f8df2f433-kube-api-access-wb8sb\") pod \"network-metrics-daemon-7f4jl\" (UID: \"57ddd2a0-f081-4f90-95a6-0f6f8df2f433\") " pod="openshift-multus/network-metrics-daemon-7f4jl" Oct 30 19:46:39.383688 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.383528 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-host-run-multus-certs\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.383688 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.383558 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-run-systemd\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.383688 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.383597 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-var-lib-openvswitch\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.383688 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.383635 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-run-openvswitch\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.383688 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.383659 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-node-log\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.383935 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.383702 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-etc-systemd\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:46:39.383935 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.383798 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66999038-807b-406b-9242-c68c04e521f3-cni-binary-copy\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.383935 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.383832 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-hostroot\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.383935 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.383860 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-host-cni-netd\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.383935 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.383903 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/688d079a-932d-4f63-97da-5cd6222a03e1-ovnkube-script-lib\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.383935 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.383932 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d1dca14b-4840-4dd6-a7cd-7d795adb9532-socket-dir\") pod \"aws-ebs-csi-driver-node-glksn\" (UID: \"d1dca14b-4840-4dd6-a7cd-7d795adb9532\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" Oct 30 19:46:39.384207 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.383965 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d1dca14b-4840-4dd6-a7cd-7d795adb9532-etc-selinux\") pod \"aws-ebs-csi-driver-node-glksn\" (UID: \"d1dca14b-4840-4dd6-a7cd-7d795adb9532\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" Oct 30 19:46:39.384207 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.383990 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-run\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:46:39.384207 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.384042 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73bb0a1f-597a-4078-8a01-20d0ed24715b-original-pull-secret\") pod \"global-pull-secret-syncer-hz6jn\" (UID: \"73bb0a1f-597a-4078-8a01-20d0ed24715b\") " pod="kube-system/global-pull-secret-syncer-hz6jn" Oct 30 19:46:39.384207 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.384072 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-host-run-netns\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.384207 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.384102 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.384207 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.384145 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-host\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:46:39.384207 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.384179 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49lpb\" (UniqueName: \"kubernetes.io/projected/08b5a0ca-cfe1-4366-a404-5ebbaeb7c149-kube-api-access-49lpb\") pod \"node-resolver-l9fpp\" (UID: \"08b5a0ca-cfe1-4366-a404-5ebbaeb7c149\") " pod="openshift-dns/node-resolver-l9fpp" Oct 30 19:46:39.384207 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.384207 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-multus-socket-dir-parent\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.384599 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.384239 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/66999038-807b-406b-9242-c68c04e521f3-multus-daemon-config\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.384599 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.384267 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-run-ovn\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.384599 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.384341 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfbsr\" (UniqueName: \"kubernetes.io/projected/688d079a-932d-4f63-97da-5cd6222a03e1-kube-api-access-tfbsr\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.384599 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.384376 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d1dca14b-4840-4dd6-a7cd-7d795adb9532-device-dir\") pod \"aws-ebs-csi-driver-node-glksn\" (UID: \"d1dca14b-4840-4dd6-a7cd-7d795adb9532\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" Oct 30 19:46:39.384599 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.384407 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-etc-tuned\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:46:39.384599 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.384436 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1afac97f-a617-4113-b8e8-c668caed54f9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2hfrk\" (UID: \"1afac97f-a617-4113-b8e8-c668caed54f9\") " pod="openshift-multus/multus-additional-cni-plugins-2hfrk" Oct 30 19:46:39.384599 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.384464 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1afac97f-a617-4113-b8e8-c668caed54f9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2hfrk\" (UID: \"1afac97f-a617-4113-b8e8-c668caed54f9\") " pod="openshift-multus/multus-additional-cni-plugins-2hfrk" Oct 30 19:46:39.384599 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.384486 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-cnibin\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.384599 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.384513 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-host-cni-bin\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.384599 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.384540 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-etc-sysconfig\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:46:39.384599 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.384569 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-lib-modules\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:46:39.384599 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.384593 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-system-cni-dir\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.385060 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.384636 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-host-var-lib-kubelet\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.385060 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.384664 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d1dca14b-4840-4dd6-a7cd-7d795adb9532-kubelet-dir\") pod \"aws-ebs-csi-driver-node-glksn\" (UID: \"d1dca14b-4840-4dd6-a7cd-7d795adb9532\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" Oct 30 19:46:39.385060 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.384723 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-tmp\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:46:39.385060 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.384756 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08b5a0ca-cfe1-4366-a404-5ebbaeb7c149-tmp-dir\") pod \"node-resolver-l9fpp\" (UID: \"08b5a0ca-cfe1-4366-a404-5ebbaeb7c149\") " pod="openshift-dns/node-resolver-l9fpp" Oct 30 19:46:39.385060 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.384781 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/73bb0a1f-597a-4078-8a01-20d0ed24715b-kubelet-config\") pod \"global-pull-secret-syncer-hz6jn\" (UID: \"73bb0a1f-597a-4078-8a01-20d0ed24715b\") " pod="kube-system/global-pull-secret-syncer-hz6jn" Oct 30 19:46:39.385060 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.384808 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d1dca14b-4840-4dd6-a7cd-7d795adb9532-sys-fs\") pod \"aws-ebs-csi-driver-node-glksn\" (UID: \"d1dca14b-4840-4dd6-a7cd-7d795adb9532\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" Oct 30 19:46:39.385060 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.384923 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-etc-kubernetes\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:46:39.385060 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.384956 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-etc-sysctl-conf\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:46:39.385060 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.384986 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkmdz\" (UniqueName: \"kubernetes.io/projected/d808d253-3f30-48e8-918c-bd3880c2af8f-kube-api-access-mkmdz\") pod \"iptables-alerter-882qh\" (UID: \"d808d253-3f30-48e8-918c-bd3880c2af8f\") " pod="openshift-network-operator/iptables-alerter-882qh" Oct 30 19:46:39.385060 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.385022 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-host-kubelet\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.385060 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.385053 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/688d079a-932d-4f63-97da-5cd6222a03e1-ovnkube-config\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.385505 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.385083 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1afac97f-a617-4113-b8e8-c668caed54f9-cnibin\") pod \"multus-additional-cni-plugins-2hfrk\" (UID: \"1afac97f-a617-4113-b8e8-c668caed54f9\") " pod="openshift-multus/multus-additional-cni-plugins-2hfrk" Oct 30 19:46:39.385505 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.385122 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-etc-openvswitch\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.385505 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.385163 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-etc-modprobe-d\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:46:39.385505 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.385216 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-var-lib-kubelet\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:46:39.424206 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.424181 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-grfkh\"" Oct 30 19:46:39.424206 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.424198 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 19:46:39.424482 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.424467 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:46:39.424745 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.424727 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 19:46:39.424826 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.424773 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 19:46:39.424826 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.424773 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 19:46:39.424936 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.424835 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 19:46:39.424936 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.424837 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qlg5x\"" Oct 30 19:46:39.424936 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.424867 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 19:46:39.424936 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.424838 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 19:46:39.424936 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.424877 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 19:46:39.424936 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.424869 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-bvhff\"" Oct 30 19:46:39.424936 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.424846 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 19:46:39.424936 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.424922 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 19:46:39.425193 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.424954 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 19:46:39.425193 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.425031 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 19:46:39.425193 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.425088 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:46:39.425193 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.425092 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 19:46:39.425193 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.425114 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 19:46:39.427326 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.427313 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 19:46:39.427369 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.427339 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 19:46:39.427400 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.427369 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 19:46:39.438784 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.438769 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-d5xr8\"" Oct 30 19:46:39.438784 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.438777 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-h4pgj\"" Oct 30 19:46:39.439663 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.439650 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 19:46:39.439716 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.439650 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wgtz9\"" Oct 30 19:46:39.439716 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.439695 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-97cb9\"" Oct 30 19:46:39.439772 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.439721 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 19:46:39.439772 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.439759 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cscqg\"" Oct 30 19:46:39.485755 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.485731 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr5dn\" (UniqueName: \"kubernetes.io/projected/1afac97f-a617-4113-b8e8-c668caed54f9-kube-api-access-nr5dn\") pod \"multus-additional-cni-plugins-2hfrk\" (UID: \"1afac97f-a617-4113-b8e8-c668caed54f9\") " pod="openshift-multus/multus-additional-cni-plugins-2hfrk" Oct 30 19:46:39.485824 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.485759 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d808d253-3f30-48e8-918c-bd3880c2af8f-host-slash\") pod \"iptables-alerter-882qh\" (UID: \"d808d253-3f30-48e8-918c-bd3880c2af8f\") " pod="openshift-network-operator/iptables-alerter-882qh" Oct 30 19:46:39.485824 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.485776 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-multus-cni-dir\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.485824 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.485811 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94r98\" (UniqueName: \"kubernetes.io/projected/2e03cb52-fd85-4e35-87b4-38833cbc7d74-kube-api-access-94r98\") pod \"node-ca-hkvn6\" (UID: \"2e03cb52-fd85-4e35-87b4-38833cbc7d74\") " pod="openshift-image-registry/node-ca-hkvn6" Oct 30 19:46:39.485948 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.485832 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/688d079a-932d-4f63-97da-5cd6222a03e1-ovn-node-metrics-cert\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.485948 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.485849 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d1dca14b-4840-4dd6-a7cd-7d795adb9532-registration-dir\") pod \"aws-ebs-csi-driver-node-glksn\" (UID: \"d1dca14b-4840-4dd6-a7cd-7d795adb9532\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" Oct 30 19:46:39.485948 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.485850 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d808d253-3f30-48e8-918c-bd3880c2af8f-host-slash\") pod \"iptables-alerter-882qh\" (UID: \"d808d253-3f30-48e8-918c-bd3880c2af8f\") " pod="openshift-network-operator/iptables-alerter-882qh" Oct 30 19:46:39.485948 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.485865 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-etc-kubernetes\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.485948 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.485885 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5rr5\" (UniqueName: \"kubernetes.io/projected/66999038-807b-406b-9242-c68c04e521f3-kube-api-access-t5rr5\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.485948 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.485908 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtqp8\" (UniqueName: \"kubernetes.io/projected/d1dca14b-4840-4dd6-a7cd-7d795adb9532-kube-api-access-xtqp8\") pod \"aws-ebs-csi-driver-node-glksn\" (UID: \"d1dca14b-4840-4dd6-a7cd-7d795adb9532\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" Oct 30 19:46:39.485948 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.485939 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d1dca14b-4840-4dd6-a7cd-7d795adb9532-registration-dir\") pod \"aws-ebs-csi-driver-node-glksn\" (UID: \"d1dca14b-4840-4dd6-a7cd-7d795adb9532\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" Oct 30 19:46:39.485948 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.485930 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm7rk\" (UniqueName: \"kubernetes.io/projected/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-kube-api-access-cm7rk\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:46:39.486277 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.485952 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-etc-kubernetes\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.486277 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.485975 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/13962162-194c-4e37-ac6b-b3bbc810cedb-konnectivity-ca\") pod \"konnectivity-agent-djkzp\" (UID: \"13962162-194c-4e37-ac6b-b3bbc810cedb\") " pod="kube-system/konnectivity-agent-djkzp" Oct 30 19:46:39.486277 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.486006 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-multus-cni-dir\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.486277 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.486111 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-host-run-k8s-cni-cncf-io\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.486277 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.486136 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-host-var-lib-cni-bin\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.486277 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.486153 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lph46\" (UniqueName: \"kubernetes.io/projected/116a1241-c9a3-4664-8217-60c2b6aa8b76-kube-api-access-lph46\") pod \"network-check-target-kvw9t\" (UID: \"116a1241-c9a3-4664-8217-60c2b6aa8b76\") " pod="openshift-network-diagnostics/network-check-target-kvw9t" Oct 30 19:46:39.486277 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.486180 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e03cb52-fd85-4e35-87b4-38833cbc7d74-host\") pod \"node-ca-hkvn6\" (UID: \"2e03cb52-fd85-4e35-87b4-38833cbc7d74\") " pod="openshift-image-registry/node-ca-hkvn6" Oct 30 19:46:39.486277 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.486182 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-host-run-k8s-cni-cncf-io\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.486277 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.486204 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-host-slash\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.486277 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.486220 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-host-var-lib-cni-bin\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.486277 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.486203 2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 19:46:39.486277 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.486231 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/13962162-194c-4e37-ac6b-b3bbc810cedb-agent-certs\") pod \"konnectivity-agent-djkzp\" (UID: \"13962162-194c-4e37-ac6b-b3bbc810cedb\") " pod="kube-system/konnectivity-agent-djkzp" Oct 30 19:46:39.486277 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.486270 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-host-slash\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.486776 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.486285 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57ddd2a0-f081-4f90-95a6-0f6f8df2f433-metrics-certs\") pod \"network-metrics-daemon-7f4jl\" (UID: \"57ddd2a0-f081-4f90-95a6-0f6f8df2f433\") " pod="openshift-multus/network-metrics-daemon-7f4jl" Oct 30 19:46:39.486776 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.486233 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e03cb52-fd85-4e35-87b4-38833cbc7d74-host\") pod \"node-ca-hkvn6\" (UID: \"2e03cb52-fd85-4e35-87b4-38833cbc7d74\") " pod="openshift-image-registry/node-ca-hkvn6" Oct 30 19:46:39.486776 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.486348 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-log-socket\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.486776 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.486305 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-log-socket\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.486776 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:39.486387 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:46:39.486776 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.486388 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-host-run-ovn-kubernetes\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.486776 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.486419 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-host-run-ovn-kubernetes\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.486776 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:39.486454 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57ddd2a0-f081-4f90-95a6-0f6f8df2f433-metrics-certs podName:57ddd2a0-f081-4f90-95a6-0f6f8df2f433 nodeName:}" failed. No retries permitted until 2026-10-30 19:46:39.986429011 +0000 UTC m=+2.109640326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57ddd2a0-f081-4f90-95a6-0f6f8df2f433-metrics-certs") pod "network-metrics-daemon-7f4jl" (UID: "57ddd2a0-f081-4f90-95a6-0f6f8df2f433") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:46:39.486776 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.486515 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/73bb0a1f-597a-4078-8a01-20d0ed24715b-dbus\") pod \"global-pull-secret-syncer-hz6jn\" (UID: \"73bb0a1f-597a-4078-8a01-20d0ed24715b\") " pod="kube-system/global-pull-secret-syncer-hz6jn" Oct 30 19:46:39.486776 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.486513 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/13962162-194c-4e37-ac6b-b3bbc810cedb-konnectivity-ca\") pod \"konnectivity-agent-djkzp\" (UID: \"13962162-194c-4e37-ac6b-b3bbc810cedb\") " pod="kube-system/konnectivity-agent-djkzp" Oct 30 19:46:39.486776 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.486581 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/73bb0a1f-597a-4078-8a01-20d0ed24715b-dbus\") pod \"global-pull-secret-syncer-hz6jn\" (UID: \"73bb0a1f-597a-4078-8a01-20d0ed24715b\") " pod="kube-system/global-pull-secret-syncer-hz6jn" Oct 30 19:46:39.486776 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.486632 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb8sb\" (UniqueName: \"kubernetes.io/projected/57ddd2a0-f081-4f90-95a6-0f6f8df2f433-kube-api-access-wb8sb\") pod \"network-metrics-daemon-7f4jl\" (UID: \"57ddd2a0-f081-4f90-95a6-0f6f8df2f433\") " pod="openshift-multus/network-metrics-daemon-7f4jl" Oct 30 19:46:39.486776 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.486658 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-host-run-multus-certs\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.486776 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.486679 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-run-systemd\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.486776 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.486699 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-var-lib-openvswitch\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.486776 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.486735 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-host-run-multus-certs\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.486776 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.486746 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-run-systemd\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.487470 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.486776 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-run-openvswitch\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.487470 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.486803 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-run-openvswitch\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.487470 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.486801 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-var-lib-openvswitch\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.487470 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.486821 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-node-log\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.487470 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.486846 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-etc-systemd\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:46:39.487470 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.486856 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-node-log\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.487470 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.486866 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66999038-807b-406b-9242-c68c04e521f3-cni-binary-copy\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.487470 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.486885 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-hostroot\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.487470 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.486895 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-etc-systemd\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:46:39.487470 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.486907 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-host-cni-netd\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.487470 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.486927 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/688d079a-932d-4f63-97da-5cd6222a03e1-ovnkube-script-lib\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.487470 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.486949 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d1dca14b-4840-4dd6-a7cd-7d795adb9532-socket-dir\") pod \"aws-ebs-csi-driver-node-glksn\" (UID: \"d1dca14b-4840-4dd6-a7cd-7d795adb9532\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" Oct 30 19:46:39.487470 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.486973 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-hostroot\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.487470 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.486986 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-host-cni-netd\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.487470 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.487007 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d1dca14b-4840-4dd6-a7cd-7d795adb9532-etc-selinux\") pod \"aws-ebs-csi-driver-node-glksn\" (UID: \"d1dca14b-4840-4dd6-a7cd-7d795adb9532\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" Oct 30 19:46:39.487470 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.487034 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-run\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:46:39.487470 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.487050 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d1dca14b-4840-4dd6-a7cd-7d795adb9532-socket-dir\") pod \"aws-ebs-csi-driver-node-glksn\" (UID: \"d1dca14b-4840-4dd6-a7cd-7d795adb9532\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" Oct 30 19:46:39.487470 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.487081 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-run\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:46:39.488349 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.487108 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73bb0a1f-597a-4078-8a01-20d0ed24715b-original-pull-secret\") pod \"global-pull-secret-syncer-hz6jn\" (UID: \"73bb0a1f-597a-4078-8a01-20d0ed24715b\") " pod="kube-system/global-pull-secret-syncer-hz6jn" Oct 30 19:46:39.488349 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.487127 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d1dca14b-4840-4dd6-a7cd-7d795adb9532-etc-selinux\") pod \"aws-ebs-csi-driver-node-glksn\" (UID: \"d1dca14b-4840-4dd6-a7cd-7d795adb9532\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" Oct 30 19:46:39.488349 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.487144 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-host-run-netns\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.488349 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:39.487206 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:46:39.488349 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:39.487256 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73bb0a1f-597a-4078-8a01-20d0ed24715b-original-pull-secret podName:73bb0a1f-597a-4078-8a01-20d0ed24715b nodeName:}" failed. No retries permitted until 2026-10-30 19:46:39.987241742 +0000 UTC m=+2.110453054 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/73bb0a1f-597a-4078-8a01-20d0ed24715b-original-pull-secret") pod "global-pull-secret-syncer-hz6jn" (UID: "73bb0a1f-597a-4078-8a01-20d0ed24715b") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:46:39.488349 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.487248 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.488349 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.487262 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-host-run-netns\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.488349 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.487281 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.488349 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.487288 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-host\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:46:39.488349 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.487318 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49lpb\" (UniqueName: \"kubernetes.io/projected/08b5a0ca-cfe1-4366-a404-5ebbaeb7c149-kube-api-access-49lpb\") pod \"node-resolver-l9fpp\" (UID: \"08b5a0ca-cfe1-4366-a404-5ebbaeb7c149\") " pod="openshift-dns/node-resolver-l9fpp" Oct 30 19:46:39.488349 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.487345 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-multus-socket-dir-parent\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.488349 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.487346 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-host\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:46:39.488349 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.487369 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/66999038-807b-406b-9242-c68c04e521f3-multus-daemon-config\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.488349 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.487392 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-run-ovn\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.488349 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.487408 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-multus-socket-dir-parent\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.488349 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.487416 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfbsr\" (UniqueName: \"kubernetes.io/projected/688d079a-932d-4f63-97da-5cd6222a03e1-kube-api-access-tfbsr\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.488349 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.487439 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d1dca14b-4840-4dd6-a7cd-7d795adb9532-device-dir\") pod \"aws-ebs-csi-driver-node-glksn\" (UID: \"d1dca14b-4840-4dd6-a7cd-7d795adb9532\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" Oct 30 19:46:39.489001 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.487463 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-etc-tuned\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:46:39.489001 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.487451 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-run-ovn\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.489001 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.487486 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1afac97f-a617-4113-b8e8-c668caed54f9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2hfrk\" (UID: \"1afac97f-a617-4113-b8e8-c668caed54f9\") " pod="openshift-multus/multus-additional-cni-plugins-2hfrk" Oct 30 19:46:39.489001 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.487501 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/688d079a-932d-4f63-97da-5cd6222a03e1-ovnkube-script-lib\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.489001 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.487511 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1afac97f-a617-4113-b8e8-c668caed54f9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2hfrk\" (UID: \"1afac97f-a617-4113-b8e8-c668caed54f9\") " pod="openshift-multus/multus-additional-cni-plugins-2hfrk" Oct 30 19:46:39.489001 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.487549 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-cnibin\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.489001 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.487391 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66999038-807b-406b-9242-c68c04e521f3-cni-binary-copy\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.489001 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.487579 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-host-cni-bin\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.489001 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.487603 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-etc-sysconfig\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:46:39.489001 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.487644 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-lib-modules\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:46:39.489001 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.487668 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-system-cni-dir\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.489001 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.487689 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-host-var-lib-kubelet\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.489001 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.487734 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-host-var-lib-kubelet\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.489001 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.487782 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-cnibin\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.489001 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.487823 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-host-cni-bin\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.489001 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.487845 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1afac97f-a617-4113-b8e8-c668caed54f9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2hfrk\" (UID: \"1afac97f-a617-4113-b8e8-c668caed54f9\") " pod="openshift-multus/multus-additional-cni-plugins-2hfrk" Oct 30 19:46:39.489001 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.487854 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/66999038-807b-406b-9242-c68c04e521f3-multus-daemon-config\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.489001 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.487878 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d1dca14b-4840-4dd6-a7cd-7d795adb9532-kubelet-dir\") pod \"aws-ebs-csi-driver-node-glksn\" (UID: \"d1dca14b-4840-4dd6-a7cd-7d795adb9532\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" Oct 30 19:46:39.489666 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.487888 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-etc-sysconfig\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:46:39.489666 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.487910 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-tmp\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:46:39.489666 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.487932 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d1dca14b-4840-4dd6-a7cd-7d795adb9532-device-dir\") pod \"aws-ebs-csi-driver-node-glksn\" (UID: \"d1dca14b-4840-4dd6-a7cd-7d795adb9532\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" Oct 30 19:46:39.489666 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.487937 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08b5a0ca-cfe1-4366-a404-5ebbaeb7c149-tmp-dir\") pod \"node-resolver-l9fpp\" (UID: \"08b5a0ca-cfe1-4366-a404-5ebbaeb7c149\") " pod="openshift-dns/node-resolver-l9fpp" Oct 30 19:46:39.489666 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.487976 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d1dca14b-4840-4dd6-a7cd-7d795adb9532-kubelet-dir\") pod \"aws-ebs-csi-driver-node-glksn\" (UID: \"d1dca14b-4840-4dd6-a7cd-7d795adb9532\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" Oct 30 19:46:39.489666 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.487986 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-system-cni-dir\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.489666 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.487975 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-lib-modules\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:46:39.489666 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.488009 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/73bb0a1f-597a-4078-8a01-20d0ed24715b-kubelet-config\") pod \"global-pull-secret-syncer-hz6jn\" (UID: \"73bb0a1f-597a-4078-8a01-20d0ed24715b\") " pod="kube-system/global-pull-secret-syncer-hz6jn" Oct 30 19:46:39.489666 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.488033 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d1dca14b-4840-4dd6-a7cd-7d795adb9532-sys-fs\") pod \"aws-ebs-csi-driver-node-glksn\" (UID: \"d1dca14b-4840-4dd6-a7cd-7d795adb9532\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" Oct 30 19:46:39.489666 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.488054 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-etc-kubernetes\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:46:39.489666 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.488077 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-etc-sysctl-conf\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:46:39.489666 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.488084 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1afac97f-a617-4113-b8e8-c668caed54f9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2hfrk\" (UID: \"1afac97f-a617-4113-b8e8-c668caed54f9\") " pod="openshift-multus/multus-additional-cni-plugins-2hfrk" Oct 30 19:46:39.489666 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.488102 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkmdz\" (UniqueName: \"kubernetes.io/projected/d808d253-3f30-48e8-918c-bd3880c2af8f-kube-api-access-mkmdz\") pod \"iptables-alerter-882qh\" (UID: \"d808d253-3f30-48e8-918c-bd3880c2af8f\") " pod="openshift-network-operator/iptables-alerter-882qh" Oct 30 19:46:39.489666 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.488129 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-host-kubelet\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.489666 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.488150 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d1dca14b-4840-4dd6-a7cd-7d795adb9532-sys-fs\") pod \"aws-ebs-csi-driver-node-glksn\" (UID: \"d1dca14b-4840-4dd6-a7cd-7d795adb9532\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" Oct 30 19:46:39.489666 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.488151 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-etc-kubernetes\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:46:39.489666 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.488087 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/73bb0a1f-597a-4078-8a01-20d0ed24715b-kubelet-config\") pod \"global-pull-secret-syncer-hz6jn\" (UID: \"73bb0a1f-597a-4078-8a01-20d0ed24715b\") " pod="kube-system/global-pull-secret-syncer-hz6jn" Oct 30 19:46:39.490231 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.488153 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/688d079a-932d-4f63-97da-5cd6222a03e1-ovnkube-config\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.490231 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.488207 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-host-kubelet\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.490231 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.488221 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1afac97f-a617-4113-b8e8-c668caed54f9-cnibin\") pod \"multus-additional-cni-plugins-2hfrk\" (UID: \"1afac97f-a617-4113-b8e8-c668caed54f9\") " pod="openshift-multus/multus-additional-cni-plugins-2hfrk" Oct 30 19:46:39.490231 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.488229 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08b5a0ca-cfe1-4366-a404-5ebbaeb7c149-tmp-dir\") pod \"node-resolver-l9fpp\" (UID: \"08b5a0ca-cfe1-4366-a404-5ebbaeb7c149\") " pod="openshift-dns/node-resolver-l9fpp" Oct 30 19:46:39.490231 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.488253 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-etc-sysctl-conf\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:46:39.490231 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.488247 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-etc-openvswitch\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.490231 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.488291 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-etc-modprobe-d\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:46:39.490231 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.488315 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-var-lib-kubelet\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:46:39.490231 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.488348 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-etc-openvswitch\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.490231 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.488358 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d808d253-3f30-48e8-918c-bd3880c2af8f-iptables-alerter-script\") pod \"iptables-alerter-882qh\" (UID: \"d808d253-3f30-48e8-918c-bd3880c2af8f\") " pod="openshift-network-operator/iptables-alerter-882qh" Oct 30 19:46:39.490231 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.488383 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/08b5a0ca-cfe1-4366-a404-5ebbaeb7c149-hosts-file\") pod \"node-resolver-l9fpp\" (UID: \"08b5a0ca-cfe1-4366-a404-5ebbaeb7c149\") " pod="openshift-dns/node-resolver-l9fpp" Oct 30 19:46:39.490231 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.488396 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-var-lib-kubelet\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:46:39.490231 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.488405 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-host-var-lib-cni-multus\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.490231 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.488464 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-etc-modprobe-d\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:46:39.490231 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.488508 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/08b5a0ca-cfe1-4366-a404-5ebbaeb7c149-hosts-file\") pod \"node-resolver-l9fpp\" (UID: \"08b5a0ca-cfe1-4366-a404-5ebbaeb7c149\") " pod="openshift-dns/node-resolver-l9fpp" Oct 30 19:46:39.490231 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.488535 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2e03cb52-fd85-4e35-87b4-38833cbc7d74-serviceca\") pod \"node-ca-hkvn6\" (UID: \"2e03cb52-fd85-4e35-87b4-38833cbc7d74\") " pod="openshift-image-registry/node-ca-hkvn6" Oct 30 19:46:39.490231 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.488546 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-host-var-lib-cni-multus\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.490231 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.488559 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/688d079a-932d-4f63-97da-5cd6222a03e1-env-overrides\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.490793 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.488582 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-etc-sysctl-d\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:46:39.490793 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.488606 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1afac97f-a617-4113-b8e8-c668caed54f9-cni-binary-copy\") pod \"multus-additional-cni-plugins-2hfrk\" (UID: \"1afac97f-a617-4113-b8e8-c668caed54f9\") " pod="openshift-multus/multus-additional-cni-plugins-2hfrk" Oct 30 19:46:39.490793 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.488647 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1afac97f-a617-4113-b8e8-c668caed54f9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2hfrk\" (UID: \"1afac97f-a617-4113-b8e8-c668caed54f9\") " pod="openshift-multus/multus-additional-cni-plugins-2hfrk" Oct 30 19:46:39.490793 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.488693 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-host-run-netns\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.490793 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.488690 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/688d079a-932d-4f63-97da-5cd6222a03e1-ovnkube-config\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.490793 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.488800 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-etc-sysctl-d\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:46:39.490793 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.488851 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-multus-conf-dir\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.490793 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.488892 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-systemd-units\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.490793 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.488917 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-sys\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:46:39.490793 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.488940 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1afac97f-a617-4113-b8e8-c668caed54f9-system-cni-dir\") pod \"multus-additional-cni-plugins-2hfrk\" (UID: \"1afac97f-a617-4113-b8e8-c668caed54f9\") " pod="openshift-multus/multus-additional-cni-plugins-2hfrk" Oct 30 19:46:39.490793 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.489011 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1afac97f-a617-4113-b8e8-c668caed54f9-system-cni-dir\") pod \"multus-additional-cni-plugins-2hfrk\" (UID: \"1afac97f-a617-4113-b8e8-c668caed54f9\") " pod="openshift-multus/multus-additional-cni-plugins-2hfrk" Oct 30 19:46:39.490793 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.489132 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-multus-conf-dir\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.490793 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.488316 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1afac97f-a617-4113-b8e8-c668caed54f9-cnibin\") pod \"multus-additional-cni-plugins-2hfrk\" (UID: \"1afac97f-a617-4113-b8e8-c668caed54f9\") " pod="openshift-multus/multus-additional-cni-plugins-2hfrk" Oct 30 19:46:39.490793 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.489179 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-systemd-units\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.490793 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.489209 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-host-run-netns\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.490793 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.489227 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-sys\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:46:39.490793 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.489806 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2e03cb52-fd85-4e35-87b4-38833cbc7d74-serviceca\") pod \"node-ca-hkvn6\" (UID: \"2e03cb52-fd85-4e35-87b4-38833cbc7d74\") " pod="openshift-image-registry/node-ca-hkvn6" Oct 30 19:46:39.490793 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.489860 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d808d253-3f30-48e8-918c-bd3880c2af8f-iptables-alerter-script\") pod \"iptables-alerter-882qh\" (UID: \"d808d253-3f30-48e8-918c-bd3880c2af8f\") " pod="openshift-network-operator/iptables-alerter-882qh" Oct 30 19:46:39.491376 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.490781 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-etc-tuned\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:46:39.491376 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.490811 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-tmp\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:46:39.491376 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.490830 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/688d079a-932d-4f63-97da-5cd6222a03e1-ovn-node-metrics-cert\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.491376 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.490878 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/688d079a-932d-4f63-97da-5cd6222a03e1-env-overrides\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.491376 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.490927 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1afac97f-a617-4113-b8e8-c668caed54f9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2hfrk\" (UID: \"1afac97f-a617-4113-b8e8-c668caed54f9\") " pod="openshift-multus/multus-additional-cni-plugins-2hfrk" Oct 30 19:46:39.491376 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.490946 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1afac97f-a617-4113-b8e8-c668caed54f9-cni-binary-copy\") pod \"multus-additional-cni-plugins-2hfrk\" (UID: \"1afac97f-a617-4113-b8e8-c668caed54f9\") " pod="openshift-multus/multus-additional-cni-plugins-2hfrk" Oct 30 19:46:39.491376 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.490959 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/13962162-194c-4e37-ac6b-b3bbc810cedb-agent-certs\") pod \"konnectivity-agent-djkzp\" (UID: \"13962162-194c-4e37-ac6b-b3bbc810cedb\") " pod="kube-system/konnectivity-agent-djkzp" Oct 30 19:46:39.511221 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:39.511155 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:46:39.511221 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:39.511180 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:46:39.511221 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:39.511193 2578 projected.go:196] Error preparing data for projected volume kube-api-access-lph46 for pod openshift-network-diagnostics/network-check-target-kvw9t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:46:39.511414 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:39.511243 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/116a1241-c9a3-4664-8217-60c2b6aa8b76-kube-api-access-lph46 podName:116a1241-c9a3-4664-8217-60c2b6aa8b76 nodeName:}" failed. No retries permitted until 2026-10-30 19:46:40.011227955 +0000 UTC m=+2.134439271 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lph46" (UniqueName: "kubernetes.io/projected/116a1241-c9a3-4664-8217-60c2b6aa8b76-kube-api-access-lph46") pod "network-check-target-kvw9t" (UID: "116a1241-c9a3-4664-8217-60c2b6aa8b76") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:46:39.513729 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.513703 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr5dn\" (UniqueName: \"kubernetes.io/projected/1afac97f-a617-4113-b8e8-c668caed54f9-kube-api-access-nr5dn\") pod \"multus-additional-cni-plugins-2hfrk\" (UID: \"1afac97f-a617-4113-b8e8-c668caed54f9\") " pod="openshift-multus/multus-additional-cni-plugins-2hfrk" Oct 30 19:46:39.513823 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.513736 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94r98\" (UniqueName: \"kubernetes.io/projected/2e03cb52-fd85-4e35-87b4-38833cbc7d74-kube-api-access-94r98\") pod \"node-ca-hkvn6\" (UID: \"2e03cb52-fd85-4e35-87b4-38833cbc7d74\") " pod="openshift-image-registry/node-ca-hkvn6" Oct 30 19:46:39.513823 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.513748 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm7rk\" (UniqueName: \"kubernetes.io/projected/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-kube-api-access-cm7rk\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:46:39.513823 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.513760 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5rr5\" (UniqueName: \"kubernetes.io/projected/66999038-807b-406b-9242-c68c04e521f3-kube-api-access-t5rr5\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.513823 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.513748 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtqp8\" (UniqueName: \"kubernetes.io/projected/d1dca14b-4840-4dd6-a7cd-7d795adb9532-kube-api-access-xtqp8\") pod \"aws-ebs-csi-driver-node-glksn\" (UID: \"d1dca14b-4840-4dd6-a7cd-7d795adb9532\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" Oct 30 19:46:39.514110 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.514091 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb8sb\" (UniqueName: \"kubernetes.io/projected/57ddd2a0-f081-4f90-95a6-0f6f8df2f433-kube-api-access-wb8sb\") pod \"network-metrics-daemon-7f4jl\" (UID: \"57ddd2a0-f081-4f90-95a6-0f6f8df2f433\") " pod="openshift-multus/network-metrics-daemon-7f4jl" Oct 30 19:46:39.515585 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.515566 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfbsr\" (UniqueName: \"kubernetes.io/projected/688d079a-932d-4f63-97da-5cd6222a03e1-kube-api-access-tfbsr\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.515680 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.515665 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkmdz\" (UniqueName: \"kubernetes.io/projected/d808d253-3f30-48e8-918c-bd3880c2af8f-kube-api-access-mkmdz\") pod \"iptables-alerter-882qh\" (UID: \"d808d253-3f30-48e8-918c-bd3880c2af8f\") " pod="openshift-network-operator/iptables-alerter-882qh" Oct 30 19:46:39.517622 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.517598 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49lpb\" (UniqueName: \"kubernetes.io/projected/08b5a0ca-cfe1-4366-a404-5ebbaeb7c149-kube-api-access-49lpb\") pod \"node-resolver-l9fpp\" (UID: \"08b5a0ca-cfe1-4366-a404-5ebbaeb7c149\") " pod="openshift-dns/node-resolver-l9fpp" Oct 30 19:46:39.622715 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.622695 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:46:39.638692 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.638670 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2hfrk" Oct 30 19:46:39.642585 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.642567 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-882qh" Oct 30 19:46:39.646691 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.646675 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:46:39.651393 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:39.651367 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1afac97f_a617_4113_b8e8_c668caed54f9.slice/crio-841d60c54cb7391e4399d2d0c208ca47a923a3e8420002d84e1741ed7cabb499 WatchSource:0}: Error finding container 841d60c54cb7391e4399d2d0c208ca47a923a3e8420002d84e1741ed7cabb499: Status 404 returned error can't find the container with id 841d60c54cb7391e4399d2d0c208ca47a923a3e8420002d84e1741ed7cabb499 Oct 30 19:46:39.665754 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.665734 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-djkzp" Oct 30 19:46:39.674086 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:39.674065 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13962162_194c_4e37_ac6b_b3bbc810cedb.slice/crio-90b0f7385b5e7b323cec3734b4d8f301a608c62518a3e22a7dc8dd1a7e0dc10a WatchSource:0}: Error finding container 90b0f7385b5e7b323cec3734b4d8f301a608c62518a3e22a7dc8dd1a7e0dc10a: Status 404 returned error can't find the container with id 90b0f7385b5e7b323cec3734b4d8f301a608c62518a3e22a7dc8dd1a7e0dc10a Oct 30 19:46:39.674744 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.674724 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hkvn6" Oct 30 19:46:39.680488 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.680472 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-76qp7" Oct 30 19:46:39.681872 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:39.681851 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e03cb52_fd85_4e35_87b4_38833cbc7d74.slice/crio-e35f258ac7b615b3bfeecd1d223abab000a5302bf771b3ee5ec4da4f4ff1a869 WatchSource:0}: Error finding container e35f258ac7b615b3bfeecd1d223abab000a5302bf771b3ee5ec4da4f4ff1a869: Status 404 returned error can't find the container with id e35f258ac7b615b3bfeecd1d223abab000a5302bf771b3ee5ec4da4f4ff1a869 Oct 30 19:46:39.688778 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:39.688761 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66999038_807b_406b_9242_c68c04e521f3.slice/crio-90448fd4921147af9aa353186a47c1bc847a290dac2c8f8fa71ee2aea8aba4fc WatchSource:0}: Error finding container 90448fd4921147af9aa353186a47c1bc847a290dac2c8f8fa71ee2aea8aba4fc: Status 404 returned error can't find the container with id 90448fd4921147af9aa353186a47c1bc847a290dac2c8f8fa71ee2aea8aba4fc Oct 30 19:46:39.702693 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.702674 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:46:39.705981 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.705963 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" Oct 30 19:46:39.709481 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.709468 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l9fpp" Oct 30 19:46:39.711045 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:39.711028 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode39329dd_fe4e_4ef9_853b_d8e5fe9d0617.slice/crio-2f60f008a6832df5c7696b9c288892c310990aa3cc05fa3e0eb8d683f11ed8b7 WatchSource:0}: Error finding container 2f60f008a6832df5c7696b9c288892c310990aa3cc05fa3e0eb8d683f11ed8b7: Status 404 returned error can't find the container with id 2f60f008a6832df5c7696b9c288892c310990aa3cc05fa3e0eb8d683f11ed8b7 Oct 30 19:46:39.713061 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.713048 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:39.715514 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:39.715497 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1dca14b_4840_4dd6_a7cd_7d795adb9532.slice/crio-e368ecd6c6d37ba2ecd82cbb81f3f55544ff8fc82499f271489e82786ae6d5eb WatchSource:0}: Error finding container e368ecd6c6d37ba2ecd82cbb81f3f55544ff8fc82499f271489e82786ae6d5eb: Status 404 returned error can't find the container with id e368ecd6c6d37ba2ecd82cbb81f3f55544ff8fc82499f271489e82786ae6d5eb Oct 30 19:46:39.718171 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:39.718151 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08b5a0ca_cfe1_4366_a404_5ebbaeb7c149.slice/crio-634672d421bdf247c95c9ab51e410db3d67827bf8df3fc64e67196ec304d2060 WatchSource:0}: Error finding container 634672d421bdf247c95c9ab51e410db3d67827bf8df3fc64e67196ec304d2060: Status 404 returned error can't find the container with id 634672d421bdf247c95c9ab51e410db3d67827bf8df3fc64e67196ec304d2060 Oct 30 19:46:39.722675 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:46:39.722659 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod688d079a_932d_4f63_97da_5cd6222a03e1.slice/crio-cf7943e1f360d110b88532af1506657a5375a296969f623eb0255a1d8af8971b WatchSource:0}: Error finding container cf7943e1f360d110b88532af1506657a5375a296969f623eb0255a1d8af8971b: Status 404 returned error can't find the container with id cf7943e1f360d110b88532af1506657a5375a296969f623eb0255a1d8af8971b Oct 30 19:46:39.992221 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.992188 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57ddd2a0-f081-4f90-95a6-0f6f8df2f433-metrics-certs\") pod \"network-metrics-daemon-7f4jl\" (UID: \"57ddd2a0-f081-4f90-95a6-0f6f8df2f433\") " pod="openshift-multus/network-metrics-daemon-7f4jl" Oct 30 19:46:39.992459 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:39.992244 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73bb0a1f-597a-4078-8a01-20d0ed24715b-original-pull-secret\") pod \"global-pull-secret-syncer-hz6jn\" (UID: \"73bb0a1f-597a-4078-8a01-20d0ed24715b\") " pod="kube-system/global-pull-secret-syncer-hz6jn" Oct 30 19:46:39.992459 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:39.992359 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:46:39.992459 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:39.992418 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73bb0a1f-597a-4078-8a01-20d0ed24715b-original-pull-secret podName:73bb0a1f-597a-4078-8a01-20d0ed24715b nodeName:}" failed. No retries permitted until 2026-10-30 19:46:40.992401374 +0000 UTC m=+3.115612684 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/73bb0a1f-597a-4078-8a01-20d0ed24715b-original-pull-secret") pod "global-pull-secret-syncer-hz6jn" (UID: "73bb0a1f-597a-4078-8a01-20d0ed24715b") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:46:39.992659 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:39.992477 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:46:39.992659 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:39.992506 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57ddd2a0-f081-4f90-95a6-0f6f8df2f433-metrics-certs podName:57ddd2a0-f081-4f90-95a6-0f6f8df2f433 nodeName:}" failed. No retries permitted until 2026-10-30 19:46:40.99249572 +0000 UTC m=+3.115707030 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57ddd2a0-f081-4f90-95a6-0f6f8df2f433-metrics-certs") pod "network-metrics-daemon-7f4jl" (UID: "57ddd2a0-f081-4f90-95a6-0f6f8df2f433") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:46:40.093158 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:40.093123 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lph46\" (UniqueName: \"kubernetes.io/projected/116a1241-c9a3-4664-8217-60c2b6aa8b76-kube-api-access-lph46\") pod \"network-check-target-kvw9t\" (UID: \"116a1241-c9a3-4664-8217-60c2b6aa8b76\") " pod="openshift-network-diagnostics/network-check-target-kvw9t" Oct 30 19:46:40.093334 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:40.093311 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:46:40.093392 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:40.093340 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:46:40.093392 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:40.093354 2578 projected.go:196] Error preparing data for projected volume kube-api-access-lph46 for pod openshift-network-diagnostics/network-check-target-kvw9t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:46:40.093521 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:40.093413 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/116a1241-c9a3-4664-8217-60c2b6aa8b76-kube-api-access-lph46 podName:116a1241-c9a3-4664-8217-60c2b6aa8b76 nodeName:}" failed. No retries permitted until 2026-10-30 19:46:41.093394629 +0000 UTC m=+3.216605946 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lph46" (UniqueName: "kubernetes.io/projected/116a1241-c9a3-4664-8217-60c2b6aa8b76-kube-api-access-lph46") pod "network-check-target-kvw9t" (UID: "116a1241-c9a3-4664-8217-60c2b6aa8b76") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:46:40.345956 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:40.345827 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:41:39 +0000 UTC" deadline="2028-05-25 16:36:23.329093675 +0000 UTC" Oct 30 19:46:40.345956 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:40.345890 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13748h49m42.983207506s" Oct 30 19:46:40.362017 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:40.361966 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2hfrk" event={"ID":"1afac97f-a617-4113-b8e8-c668caed54f9","Type":"ContainerStarted","Data":"841d60c54cb7391e4399d2d0c208ca47a923a3e8420002d84e1741ed7cabb499"} Oct 30 19:46:40.370667 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:40.370640 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" event={"ID":"688d079a-932d-4f63-97da-5cd6222a03e1","Type":"ContainerStarted","Data":"cf7943e1f360d110b88532af1506657a5375a296969f623eb0255a1d8af8971b"} Oct 30 19:46:40.387189 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:40.387109 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l9fpp" event={"ID":"08b5a0ca-cfe1-4366-a404-5ebbaeb7c149","Type":"ContainerStarted","Data":"634672d421bdf247c95c9ab51e410db3d67827bf8df3fc64e67196ec304d2060"} Oct 30 19:46:40.400324 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:40.400242 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" event={"ID":"d1dca14b-4840-4dd6-a7cd-7d795adb9532","Type":"ContainerStarted","Data":"e368ecd6c6d37ba2ecd82cbb81f3f55544ff8fc82499f271489e82786ae6d5eb"} Oct 30 19:46:40.405020 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:40.404961 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hkvn6" event={"ID":"2e03cb52-fd85-4e35-87b4-38833cbc7d74","Type":"ContainerStarted","Data":"e35f258ac7b615b3bfeecd1d223abab000a5302bf771b3ee5ec4da4f4ff1a869"} Oct 30 19:46:40.412001 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:40.411977 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-882qh" event={"ID":"d808d253-3f30-48e8-918c-bd3880c2af8f","Type":"ContainerStarted","Data":"60d9e8cdd3474de1d46b40c7de021a8ecb3649b259ff50160012f1e139c2eea0"} Oct 30 19:46:40.418719 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:40.418696 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" event={"ID":"6ad70715366d9db6ff14ef77cdc50313","Type":"ContainerStarted","Data":"23213dbd30843595194cc488e41405260a4e82391bf97b3fec6e67059dd7cdc8"} Oct 30 19:46:40.422131 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:40.422109 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal" event={"ID":"5b1dc20b8a7bd35e96a3f207cff8defc","Type":"ContainerStarted","Data":"8ab94cd9a89ec8b7e8f48c21784a2d597264034894e149a4d80accb109d73fb2"} Oct 30 19:46:40.425517 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:40.425487 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rggcq" event={"ID":"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617","Type":"ContainerStarted","Data":"2f60f008a6832df5c7696b9c288892c310990aa3cc05fa3e0eb8d683f11ed8b7"} Oct 30 19:46:40.429640 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:40.429605 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-76qp7" event={"ID":"66999038-807b-406b-9242-c68c04e521f3","Type":"ContainerStarted","Data":"90448fd4921147af9aa353186a47c1bc847a290dac2c8f8fa71ee2aea8aba4fc"} Oct 30 19:46:40.433187 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:40.433124 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-djkzp" event={"ID":"13962162-194c-4e37-ac6b-b3bbc810cedb","Type":"ContainerStarted","Data":"90b0f7385b5e7b323cec3734b4d8f301a608c62518a3e22a7dc8dd1a7e0dc10a"} Oct 30 19:46:40.528151 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:40.528121 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:46:41.001783 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:41.001753 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73bb0a1f-597a-4078-8a01-20d0ed24715b-original-pull-secret\") pod \"global-pull-secret-syncer-hz6jn\" (UID: \"73bb0a1f-597a-4078-8a01-20d0ed24715b\") " pod="kube-system/global-pull-secret-syncer-hz6jn" Oct 30 19:46:41.001941 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:41.001841 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57ddd2a0-f081-4f90-95a6-0f6f8df2f433-metrics-certs\") pod \"network-metrics-daemon-7f4jl\" (UID: \"57ddd2a0-f081-4f90-95a6-0f6f8df2f433\") " pod="openshift-multus/network-metrics-daemon-7f4jl" Oct 30 19:46:41.001994 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:41.001958 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:46:41.002052 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:41.002012 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57ddd2a0-f081-4f90-95a6-0f6f8df2f433-metrics-certs podName:57ddd2a0-f081-4f90-95a6-0f6f8df2f433 nodeName:}" failed. No retries permitted until 2026-10-30 19:46:43.001995006 +0000 UTC m=+5.125206308 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57ddd2a0-f081-4f90-95a6-0f6f8df2f433-metrics-certs") pod "network-metrics-daemon-7f4jl" (UID: "57ddd2a0-f081-4f90-95a6-0f6f8df2f433") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:46:41.002415 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:41.002399 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:46:41.002487 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:41.002454 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73bb0a1f-597a-4078-8a01-20d0ed24715b-original-pull-secret podName:73bb0a1f-597a-4078-8a01-20d0ed24715b nodeName:}" failed. No retries permitted until 2026-10-30 19:46:43.002440107 +0000 UTC m=+5.125651425 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/73bb0a1f-597a-4078-8a01-20d0ed24715b-original-pull-secret") pod "global-pull-secret-syncer-hz6jn" (UID: "73bb0a1f-597a-4078-8a01-20d0ed24715b") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:46:41.102687 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:41.102111 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lph46\" (UniqueName: \"kubernetes.io/projected/116a1241-c9a3-4664-8217-60c2b6aa8b76-kube-api-access-lph46\") pod \"network-check-target-kvw9t\" (UID: \"116a1241-c9a3-4664-8217-60c2b6aa8b76\") " pod="openshift-network-diagnostics/network-check-target-kvw9t" Oct 30 19:46:41.102687 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:41.102317 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:46:41.102687 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:41.102337 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:46:41.102687 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:41.102351 2578 projected.go:196] Error preparing data for projected volume kube-api-access-lph46 for pod openshift-network-diagnostics/network-check-target-kvw9t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:46:41.102687 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:41.102406 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/116a1241-c9a3-4664-8217-60c2b6aa8b76-kube-api-access-lph46 podName:116a1241-c9a3-4664-8217-60c2b6aa8b76 nodeName:}" failed. No retries permitted until 2026-10-30 19:46:43.102387916 +0000 UTC m=+5.225599226 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lph46" (UniqueName: "kubernetes.io/projected/116a1241-c9a3-4664-8217-60c2b6aa8b76-kube-api-access-lph46") pod "network-check-target-kvw9t" (UID: "116a1241-c9a3-4664-8217-60c2b6aa8b76") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:46:41.346122 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:41.346027 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:41:39 +0000 UTC" deadline="2028-05-02 20:19:06.353164669 +0000 UTC" Oct 30 19:46:41.346122 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:41.346072 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13200h32m25.007096736s" Oct 30 19:46:41.347546 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:41.347521 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvw9t" Oct 30 19:46:41.347686 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:41.347660 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvw9t" podUID="116a1241-c9a3-4664-8217-60c2b6aa8b76" Oct 30 19:46:41.348109 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:41.348089 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7f4jl" Oct 30 19:46:41.348236 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:41.348213 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7f4jl" podUID="57ddd2a0-f081-4f90-95a6-0f6f8df2f433" Oct 30 19:46:41.348313 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:41.348293 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hz6jn" Oct 30 19:46:41.348389 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:41.348370 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hz6jn" podUID="73bb0a1f-597a-4078-8a01-20d0ed24715b" Oct 30 19:46:43.017529 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:43.017288 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73bb0a1f-597a-4078-8a01-20d0ed24715b-original-pull-secret\") pod \"global-pull-secret-syncer-hz6jn\" (UID: \"73bb0a1f-597a-4078-8a01-20d0ed24715b\") " pod="kube-system/global-pull-secret-syncer-hz6jn" Oct 30 19:46:43.017993 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:43.017591 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57ddd2a0-f081-4f90-95a6-0f6f8df2f433-metrics-certs\") pod \"network-metrics-daemon-7f4jl\" (UID: \"57ddd2a0-f081-4f90-95a6-0f6f8df2f433\") " pod="openshift-multus/network-metrics-daemon-7f4jl" Oct 30 19:46:43.017993 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:43.017732 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:46:43.017993 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:43.017799 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57ddd2a0-f081-4f90-95a6-0f6f8df2f433-metrics-certs podName:57ddd2a0-f081-4f90-95a6-0f6f8df2f433 nodeName:}" failed. No retries permitted until 2026-10-30 19:46:47.017779893 +0000 UTC m=+9.140991208 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57ddd2a0-f081-4f90-95a6-0f6f8df2f433-metrics-certs") pod "network-metrics-daemon-7f4jl" (UID: "57ddd2a0-f081-4f90-95a6-0f6f8df2f433") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:46:43.018219 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:43.018200 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:46:43.018282 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:43.018256 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73bb0a1f-597a-4078-8a01-20d0ed24715b-original-pull-secret podName:73bb0a1f-597a-4078-8a01-20d0ed24715b nodeName:}" failed. No retries permitted until 2026-10-30 19:46:47.018241579 +0000 UTC m=+9.141452888 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/73bb0a1f-597a-4078-8a01-20d0ed24715b-original-pull-secret") pod "global-pull-secret-syncer-hz6jn" (UID: "73bb0a1f-597a-4078-8a01-20d0ed24715b") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:46:43.118428 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:43.118345 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lph46\" (UniqueName: \"kubernetes.io/projected/116a1241-c9a3-4664-8217-60c2b6aa8b76-kube-api-access-lph46\") pod \"network-check-target-kvw9t\" (UID: \"116a1241-c9a3-4664-8217-60c2b6aa8b76\") " pod="openshift-network-diagnostics/network-check-target-kvw9t" Oct 30 19:46:43.118590 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:43.118571 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:46:43.118670 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:43.118595 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:46:43.118670 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:43.118622 2578 projected.go:196] Error preparing data for projected volume kube-api-access-lph46 for pod openshift-network-diagnostics/network-check-target-kvw9t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:46:43.118759 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:43.118684 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/116a1241-c9a3-4664-8217-60c2b6aa8b76-kube-api-access-lph46 podName:116a1241-c9a3-4664-8217-60c2b6aa8b76 nodeName:}" failed. No retries permitted until 2026-10-30 19:46:47.118665772 +0000 UTC m=+9.241877086 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lph46" (UniqueName: "kubernetes.io/projected/116a1241-c9a3-4664-8217-60c2b6aa8b76-kube-api-access-lph46") pod "network-check-target-kvw9t" (UID: "116a1241-c9a3-4664-8217-60c2b6aa8b76") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:46:43.349854 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:43.349362 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hz6jn" Oct 30 19:46:43.349854 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:43.349473 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hz6jn" podUID="73bb0a1f-597a-4078-8a01-20d0ed24715b" Oct 30 19:46:43.349854 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:43.349542 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7f4jl" Oct 30 19:46:43.349854 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:43.349660 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7f4jl" podUID="57ddd2a0-f081-4f90-95a6-0f6f8df2f433" Oct 30 19:46:43.349854 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:43.349720 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvw9t" Oct 30 19:46:43.349854 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:43.349781 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvw9t" podUID="116a1241-c9a3-4664-8217-60c2b6aa8b76" Oct 30 19:46:43.464505 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:43.464445 2578 generic.go:356] "Generic (PLEG): container finished" podID="6ad70715366d9db6ff14ef77cdc50313" containerID="c97983388b9679c12f0c64a9c0187402ee8ff80d0cacdb27704bcb35203c3e3b" exitCode=0 Oct 30 19:46:43.464692 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:43.464554 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" event={"ID":"6ad70715366d9db6ff14ef77cdc50313","Type":"ContainerDied","Data":"c97983388b9679c12f0c64a9c0187402ee8ff80d0cacdb27704bcb35203c3e3b"} Oct 30 19:46:43.469239 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:43.469212 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2hfrk" event={"ID":"1afac97f-a617-4113-b8e8-c668caed54f9","Type":"ContainerStarted","Data":"bc0bdcb4dbea76c6600166f5f294a2ad5cb60fe73d68f9f9a4a18a896a81f7b6"} Oct 30 19:46:44.479388 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:44.479350 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" event={"ID":"6ad70715366d9db6ff14ef77cdc50313","Type":"ContainerStarted","Data":"1b4fdb1cc924a9f616cf519cedea812417a4457c52b0e475178d3f758a84bfd8"} Oct 30 19:46:44.483585 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:44.483557 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal" event={"ID":"5b1dc20b8a7bd35e96a3f207cff8defc","Type":"ContainerStarted","Data":"6b4c6d4b4f23c6215e563503a440670518d6d93826c0c6fb7ac29bfbb0dbeb8d"} Oct 30 19:46:44.483810 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:44.483789 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal" Oct 30 19:46:44.486889 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:44.486826 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-djkzp" event={"ID":"13962162-194c-4e37-ac6b-b3bbc810cedb","Type":"ContainerStarted","Data":"a837ba8ca41afc3929ae7683825016b345cfb3c5bbcde03cdacff01451ef364b"} Oct 30 19:46:44.489590 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:44.489567 2578 generic.go:356] "Generic (PLEG): container finished" podID="1afac97f-a617-4113-b8e8-c668caed54f9" containerID="bc0bdcb4dbea76c6600166f5f294a2ad5cb60fe73d68f9f9a4a18a896a81f7b6" exitCode=0 Oct 30 19:46:44.489704 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:44.489605 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2hfrk" event={"ID":"1afac97f-a617-4113-b8e8-c668caed54f9","Type":"ContainerDied","Data":"bc0bdcb4dbea76c6600166f5f294a2ad5cb60fe73d68f9f9a4a18a896a81f7b6"} Oct 30 19:46:44.506387 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:44.506359 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal"] Oct 30 19:46:44.506534 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:44.506519 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:46:44.566654 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:44.566586 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" podStartSLOduration=5.566565997 podStartE2EDuration="5.566565997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:46:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:44.524347212 +0000 UTC m=+6.647558531" watchObservedRunningTime="2026-10-30 19:46:44.566565997 +0000 UTC m=+6.689777319" Oct 30 19:46:44.615676 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:44.615604 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal" podStartSLOduration=0.61559163 podStartE2EDuration="615.59163ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:46:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:44.567074529 +0000 UTC m=+6.690285847" watchObservedRunningTime="2026-10-30 19:46:44.61559163 +0000 UTC m=+6.738802948" Oct 30 19:46:45.347318 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:45.347285 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvw9t" Oct 30 19:46:45.347481 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:45.347418 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvw9t" podUID="116a1241-c9a3-4664-8217-60c2b6aa8b76" Oct 30 19:46:45.347845 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:45.347825 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hz6jn" Oct 30 19:46:45.347948 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:45.347926 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hz6jn" podUID="73bb0a1f-597a-4078-8a01-20d0ed24715b" Oct 30 19:46:45.348007 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:45.347995 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7f4jl" Oct 30 19:46:45.348095 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:45.348078 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7f4jl" podUID="57ddd2a0-f081-4f90-95a6-0f6f8df2f433" Oct 30 19:46:45.391479 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:45.391149 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-djkzp" Oct 30 19:46:45.391853 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:45.391834 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-djkzp" Oct 30 19:46:45.435252 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:45.435206 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-djkzp" podStartSLOduration=3.343789548 podStartE2EDuration="7.43518881s" totalImagesPullingTime="4.091399262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:44.663351287 +0000 UTC m=+6.786562608" watchObservedRunningTime="2026-10-30 19:46:45.43518881 +0000 UTC m=+7.558400128" Oct 30 19:46:45.508863 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:45.508824 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hkvn6" event={"ID":"2e03cb52-fd85-4e35-87b4-38833cbc7d74","Type":"ContainerStarted","Data":"9035420153ab6a0f8e54151d18ee4b4bb19528bcb8f6df519cc043c81eb8a8bd"} Oct 30 19:46:45.552521 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:45.552341 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hkvn6" podStartSLOduration=2.422255339 podStartE2EDuration="7.55232959s" totalImagesPullingTime="5.130074251s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:46:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:45.551662075 +0000 UTC m=+7.674873420" watchObservedRunningTime="2026-10-30 19:46:45.55232959 +0000 UTC m=+7.675540906" Oct 30 19:46:46.517170 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:46.517038 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" event={"ID":"d1dca14b-4840-4dd6-a7cd-7d795adb9532","Type":"ContainerStarted","Data":"a27cf4956382c5a218bf526b1978d0074033f22c59f535382f584176300ac92a"} Oct 30 19:46:46.517170 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:46.517136 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:46:47.014198 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:47.014168 2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 19:46:47.055472 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:47.055438 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73bb0a1f-597a-4078-8a01-20d0ed24715b-original-pull-secret\") pod \"global-pull-secret-syncer-hz6jn\" (UID: \"73bb0a1f-597a-4078-8a01-20d0ed24715b\") " pod="kube-system/global-pull-secret-syncer-hz6jn" Oct 30 19:46:47.055631 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:47.055522 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57ddd2a0-f081-4f90-95a6-0f6f8df2f433-metrics-certs\") pod \"network-metrics-daemon-7f4jl\" (UID: \"57ddd2a0-f081-4f90-95a6-0f6f8df2f433\") " pod="openshift-multus/network-metrics-daemon-7f4jl" Oct 30 19:46:47.055678 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:47.055647 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:46:47.055719 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:47.055700 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57ddd2a0-f081-4f90-95a6-0f6f8df2f433-metrics-certs podName:57ddd2a0-f081-4f90-95a6-0f6f8df2f433 nodeName:}" failed. No retries permitted until 2026-10-30 19:46:55.055682217 +0000 UTC m=+17.178893520 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57ddd2a0-f081-4f90-95a6-0f6f8df2f433-metrics-certs") pod "network-metrics-daemon-7f4jl" (UID: "57ddd2a0-f081-4f90-95a6-0f6f8df2f433") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:46:47.056054 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:47.056041 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:46:47.056092 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:47.056087 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73bb0a1f-597a-4078-8a01-20d0ed24715b-original-pull-secret podName:73bb0a1f-597a-4078-8a01-20d0ed24715b nodeName:}" failed. No retries permitted until 2026-10-30 19:46:55.056073804 +0000 UTC m=+17.179285119 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/73bb0a1f-597a-4078-8a01-20d0ed24715b-original-pull-secret") pod "global-pull-secret-syncer-hz6jn" (UID: "73bb0a1f-597a-4078-8a01-20d0ed24715b") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:46:47.156575 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:47.156503 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lph46\" (UniqueName: \"kubernetes.io/projected/116a1241-c9a3-4664-8217-60c2b6aa8b76-kube-api-access-lph46\") pod \"network-check-target-kvw9t\" (UID: \"116a1241-c9a3-4664-8217-60c2b6aa8b76\") " pod="openshift-network-diagnostics/network-check-target-kvw9t" Oct 30 19:46:47.156721 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:47.156679 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:46:47.156721 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:47.156696 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:46:47.156721 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:47.156708 2578 projected.go:196] Error preparing data for projected volume kube-api-access-lph46 for pod openshift-network-diagnostics/network-check-target-kvw9t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:46:47.156827 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:47.156758 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/116a1241-c9a3-4664-8217-60c2b6aa8b76-kube-api-access-lph46 podName:116a1241-c9a3-4664-8217-60c2b6aa8b76 nodeName:}" failed. No retries permitted until 2026-10-30 19:46:55.156742135 +0000 UTC m=+17.279953436 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lph46" (UniqueName: "kubernetes.io/projected/116a1241-c9a3-4664-8217-60c2b6aa8b76-kube-api-access-lph46") pod "network-check-target-kvw9t" (UID: "116a1241-c9a3-4664-8217-60c2b6aa8b76") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:46:47.337800 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:47.337683 2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:46:47.014187175Z","UUID":"2e645e0b-d97b-425f-a232-d7e3da97c172","Handler":null,"Name":"","Endpoint":""} Oct 30 19:46:47.339451 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:47.339432 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 19:46:47.339577 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:47.339460 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 19:46:47.347526 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:47.347505 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hz6jn" Oct 30 19:46:47.347647 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:47.347597 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hz6jn" podUID="73bb0a1f-597a-4078-8a01-20d0ed24715b" Oct 30 19:46:47.347888 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:47.347873 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7f4jl" Oct 30 19:46:47.347992 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:47.347947 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7f4jl" podUID="57ddd2a0-f081-4f90-95a6-0f6f8df2f433" Oct 30 19:46:47.348042 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:47.347992 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvw9t" Oct 30 19:46:47.348042 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:47.348034 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvw9t" podUID="116a1241-c9a3-4664-8217-60c2b6aa8b76" Oct 30 19:46:47.524249 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:47.524006 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" event={"ID":"d1dca14b-4840-4dd6-a7cd-7d795adb9532","Type":"ContainerStarted","Data":"ccb1a7b7c9207915beb507669d347530e24fc01dabbe85f3f14317a6fac52293"} Oct 30 19:46:47.526891 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:47.526179 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-76qp7" event={"ID":"66999038-807b-406b-9242-c68c04e521f3","Type":"ContainerStarted","Data":"280d4769116aa040a5e56ac0e20948e2dee843dc705e9a51f06d4661b30564dc"} Oct 30 19:46:47.574243 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:47.574171 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-76qp7" podStartSLOduration=1.8927561019999999 podStartE2EDuration="9.574154113s" totalImagesPullingTime="7.681398011s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:46:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:47.573385932 +0000 UTC m=+9.696597250" watchObservedRunningTime="2026-10-30 19:46:47.574154113 +0000 UTC m=+9.697365430" Oct 30 19:46:48.529795 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:48.529758 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l9fpp" event={"ID":"08b5a0ca-cfe1-4366-a404-5ebbaeb7c149","Type":"ContainerStarted","Data":"54f9c02577d023c15545efe11897d35f559dac71c4086ec2ff773fc6c5c2fdc3"} Oct 30 19:46:48.532274 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:48.532241 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" event={"ID":"d1dca14b-4840-4dd6-a7cd-7d795adb9532","Type":"ContainerStarted","Data":"9be43f36855a9450e65262ccbdfef9920d4f75ffe9934ebdcf3d447858d68a8c"} Oct 30 19:46:48.534363 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:48.534337 2578 generic.go:356] "Generic (PLEG): container finished" podID="1afac97f-a617-4113-b8e8-c668caed54f9" containerID="530b5dd05cf8c5aa916adc78ad54a0f0e8e0cde813e5ea127e032eefbe455a7e" exitCode=0 Oct 30 19:46:48.534469 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:48.534414 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2hfrk" event={"ID":"1afac97f-a617-4113-b8e8-c668caed54f9","Type":"ContainerDied","Data":"530b5dd05cf8c5aa916adc78ad54a0f0e8e0cde813e5ea127e032eefbe455a7e"} Oct 30 19:46:48.573100 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:48.573036 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-l9fpp" podStartSLOduration=2.57190989 podStartE2EDuration="10.573018139s" totalImagesPullingTime="8.001108249s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:46:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:48.572231374 +0000 UTC m=+10.695442691" watchObservedRunningTime="2026-10-30 19:46:48.573018139 +0000 UTC m=+10.696229458" Oct 30 19:46:48.665198 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:48.665107 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" podStartSLOduration=2.621448094 podStartE2EDuration="10.665089503s" totalImagesPullingTime="8.043641409s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:46:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:48.618205329 +0000 UTC m=+10.741416641" watchObservedRunningTime="2026-10-30 19:46:48.665089503 +0000 UTC m=+10.788300823" Oct 30 19:46:49.347523 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:49.347445 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvw9t" Oct 30 19:46:49.347682 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:49.347569 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvw9t" podUID="116a1241-c9a3-4664-8217-60c2b6aa8b76" Oct 30 19:46:49.347955 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:49.347930 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hz6jn" Oct 30 19:46:49.348063 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:49.348032 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hz6jn" podUID="73bb0a1f-597a-4078-8a01-20d0ed24715b" Oct 30 19:46:49.348108 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:49.348095 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7f4jl" Oct 30 19:46:49.348175 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:49.348158 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7f4jl" podUID="57ddd2a0-f081-4f90-95a6-0f6f8df2f433" Oct 30 19:46:49.537862 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:49.537827 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-882qh" event={"ID":"d808d253-3f30-48e8-918c-bd3880c2af8f","Type":"ContainerStarted","Data":"71474ede25cda6566cce0639eabee348dfb1a632e034ca25c6909bdd9568dd9a"} Oct 30 19:46:49.540057 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:49.540030 2578 generic.go:356] "Generic (PLEG): container finished" podID="1afac97f-a617-4113-b8e8-c668caed54f9" containerID="6df040846437c3b32e997c88e7117f1b6d533797da284bc9405555b5da9859c0" exitCode=0 Oct 30 19:46:49.540180 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:49.540116 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2hfrk" event={"ID":"1afac97f-a617-4113-b8e8-c668caed54f9","Type":"ContainerDied","Data":"6df040846437c3b32e997c88e7117f1b6d533797da284bc9405555b5da9859c0"} Oct 30 19:46:49.580827 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:49.580779 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-882qh" podStartSLOduration=3.870900959 podStartE2EDuration="11.580766552s" totalImagesPullingTime="7.709865593s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:46:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:49.580147415 +0000 UTC m=+11.703358732" watchObservedRunningTime="2026-10-30 19:46:49.580766552 +0000 UTC m=+11.703977868" Oct 30 19:46:50.544012 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:50.543789 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rggcq" event={"ID":"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617","Type":"ContainerStarted","Data":"c25ab4748b71ed6df2f0e6e1017e92458d8e7a1b87d0e401cc34f5c261c6b7a3"} Oct 30 19:46:50.547445 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:50.547417 2578 generic.go:356] "Generic (PLEG): container finished" podID="1afac97f-a617-4113-b8e8-c668caed54f9" containerID="04eb863c3af79eeb76f7bb3fae710547ff4b46691a6e51bfa0d7eb1a4949da29" exitCode=0 Oct 30 19:46:50.547568 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:50.547509 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2hfrk" event={"ID":"1afac97f-a617-4113-b8e8-c668caed54f9","Type":"ContainerDied","Data":"04eb863c3af79eeb76f7bb3fae710547ff4b46691a6e51bfa0d7eb1a4949da29"} Oct 30 19:46:50.593592 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:50.593543 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rggcq" podStartSLOduration=2.763684023 podStartE2EDuration="12.593527341s" totalImagesPullingTime="9.829843318s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:46:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:50.593117373 +0000 UTC m=+12.716328725" watchObservedRunningTime="2026-10-30 19:46:50.593527341 +0000 UTC m=+12.716738659" Oct 30 19:46:51.346821 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:51.346792 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvw9t" Oct 30 19:46:51.347016 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:51.346866 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7f4jl" Oct 30 19:46:51.347016 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:51.346884 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hz6jn" Oct 30 19:46:51.347016 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:51.346973 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvw9t" podUID="116a1241-c9a3-4664-8217-60c2b6aa8b76" Oct 30 19:46:51.347178 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:51.347034 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7f4jl" podUID="57ddd2a0-f081-4f90-95a6-0f6f8df2f433" Oct 30 19:46:51.347178 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:51.347078 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hz6jn" podUID="73bb0a1f-597a-4078-8a01-20d0ed24715b" Oct 30 19:46:52.555470 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:52.554895 2578 generic.go:356] "Generic (PLEG): container finished" podID="1afac97f-a617-4113-b8e8-c668caed54f9" containerID="cdaea81198fd578e09a5dca061b4ee1b022cc960695a31c154722fbaffd10db3" exitCode=0 Oct 30 19:46:52.555470 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:52.554938 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2hfrk" event={"ID":"1afac97f-a617-4113-b8e8-c668caed54f9","Type":"ContainerDied","Data":"cdaea81198fd578e09a5dca061b4ee1b022cc960695a31c154722fbaffd10db3"} Oct 30 19:46:53.347370 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:53.347333 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvw9t" Oct 30 19:46:53.347538 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:53.347437 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hz6jn" Oct 30 19:46:53.347538 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:53.347452 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvw9t" podUID="116a1241-c9a3-4664-8217-60c2b6aa8b76" Oct 30 19:46:53.347538 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:53.347473 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7f4jl" Oct 30 19:46:53.347717 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:53.347558 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hz6jn" podUID="73bb0a1f-597a-4078-8a01-20d0ed24715b" Oct 30 19:46:53.347717 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:53.347652 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7f4jl" podUID="57ddd2a0-f081-4f90-95a6-0f6f8df2f433" Oct 30 19:46:53.559411 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:53.559381 2578 generic.go:356] "Generic (PLEG): container finished" podID="1afac97f-a617-4113-b8e8-c668caed54f9" containerID="6ee20aae29a5db678983275618de0ae7fe0d74c6ae8803a20db88485e40de6a5" exitCode=0 Oct 30 19:46:53.559833 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:53.559422 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2hfrk" event={"ID":"1afac97f-a617-4113-b8e8-c668caed54f9","Type":"ContainerDied","Data":"6ee20aae29a5db678983275618de0ae7fe0d74c6ae8803a20db88485e40de6a5"} Oct 30 19:46:54.564743 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:54.564669 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2hfrk" event={"ID":"1afac97f-a617-4113-b8e8-c668caed54f9","Type":"ContainerStarted","Data":"773530f68c6db0bc003e2a20e543c9d54d377cef92664d82374bb220bc02ad4d"} Oct 30 19:46:54.567312 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:54.567293 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xpjpn_688d079a-932d-4f63-97da-5cd6222a03e1/ovn-acl-logging/0.log" Oct 30 19:46:54.567603 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:54.567583 2578 generic.go:356] "Generic (PLEG): container finished" podID="688d079a-932d-4f63-97da-5cd6222a03e1" containerID="5fe9a2e1adf6d1e2dcd320f39f5481d5a67b40aae630fc540688ebcc8c1845e7" exitCode=1 Oct 30 19:46:54.567675 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:54.567651 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" event={"ID":"688d079a-932d-4f63-97da-5cd6222a03e1","Type":"ContainerStarted","Data":"844d6a1cb47f40e7bbc91c4ddefac6259fc4449adeb772f77c402d3fb96625e5"} Oct 30 19:46:54.567730 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:54.567720 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" event={"ID":"688d079a-932d-4f63-97da-5cd6222a03e1","Type":"ContainerStarted","Data":"7279f4da7b133380e726d3bacd5a5daedb7ffb7a454792716fe62cecd249b767"} Oct 30 19:46:54.567762 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:54.567751 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" event={"ID":"688d079a-932d-4f63-97da-5cd6222a03e1","Type":"ContainerStarted","Data":"46ba6b27a8ebfe455be93a1141467f9e627756dafb54dd29b01f6b398673ce15"} Oct 30 19:46:54.567795 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:54.567784 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" event={"ID":"688d079a-932d-4f63-97da-5cd6222a03e1","Type":"ContainerStarted","Data":"ce054b5a11d3fe4bf149e80d131aa5b98388a5a798a8500ab597cbd51df2157b"} Oct 30 19:46:54.567828 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:54.567815 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" event={"ID":"688d079a-932d-4f63-97da-5cd6222a03e1","Type":"ContainerDied","Data":"5fe9a2e1adf6d1e2dcd320f39f5481d5a67b40aae630fc540688ebcc8c1845e7"} Oct 30 19:46:54.567856 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:54.567842 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" event={"ID":"688d079a-932d-4f63-97da-5cd6222a03e1","Type":"ContainerStarted","Data":"d1474b728c0493a587e1be211cced2adb398c4d798a26e2e96ecb67d16a22f8f"} Oct 30 19:46:54.618388 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:54.618344 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2hfrk" podStartSLOduration=7.249866797 podStartE2EDuration="16.618335526s" totalImagesPullingTime="9.368468729s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:46:38 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:54.618070311 +0000 UTC m=+16.741281628" watchObservedRunningTime="2026-10-30 19:46:54.618335526 +0000 UTC m=+16.741546843" Oct 30 19:46:55.116489 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:55.116464 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57ddd2a0-f081-4f90-95a6-0f6f8df2f433-metrics-certs\") pod \"network-metrics-daemon-7f4jl\" (UID: \"57ddd2a0-f081-4f90-95a6-0f6f8df2f433\") " pod="openshift-multus/network-metrics-daemon-7f4jl" Oct 30 19:46:55.116666 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:55.116496 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73bb0a1f-597a-4078-8a01-20d0ed24715b-original-pull-secret\") pod \"global-pull-secret-syncer-hz6jn\" (UID: \"73bb0a1f-597a-4078-8a01-20d0ed24715b\") " pod="kube-system/global-pull-secret-syncer-hz6jn" Oct 30 19:46:55.116666 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:55.116582 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:46:55.116666 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:55.116584 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:46:55.116666 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:55.116643 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73bb0a1f-597a-4078-8a01-20d0ed24715b-original-pull-secret podName:73bb0a1f-597a-4078-8a01-20d0ed24715b nodeName:}" failed. No retries permitted until 2026-10-30 19:47:11.116625782 +0000 UTC m=+33.239837094 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/73bb0a1f-597a-4078-8a01-20d0ed24715b-original-pull-secret") pod "global-pull-secret-syncer-hz6jn" (UID: "73bb0a1f-597a-4078-8a01-20d0ed24715b") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:46:55.116666 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:55.116656 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57ddd2a0-f081-4f90-95a6-0f6f8df2f433-metrics-certs podName:57ddd2a0-f081-4f90-95a6-0f6f8df2f433 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:11.116650053 +0000 UTC m=+33.239861349 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57ddd2a0-f081-4f90-95a6-0f6f8df2f433-metrics-certs") pod "network-metrics-daemon-7f4jl" (UID: "57ddd2a0-f081-4f90-95a6-0f6f8df2f433") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:46:55.217184 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:55.217156 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lph46\" (UniqueName: \"kubernetes.io/projected/116a1241-c9a3-4664-8217-60c2b6aa8b76-kube-api-access-lph46\") pod \"network-check-target-kvw9t\" (UID: \"116a1241-c9a3-4664-8217-60c2b6aa8b76\") " pod="openshift-network-diagnostics/network-check-target-kvw9t" Oct 30 19:46:55.217304 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:55.217264 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:46:55.217304 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:55.217275 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:46:55.217304 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:55.217283 2578 projected.go:196] Error preparing data for projected volume kube-api-access-lph46 for pod openshift-network-diagnostics/network-check-target-kvw9t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:46:55.217393 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:55.217317 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/116a1241-c9a3-4664-8217-60c2b6aa8b76-kube-api-access-lph46 podName:116a1241-c9a3-4664-8217-60c2b6aa8b76 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:11.2173081 +0000 UTC m=+33.340519395 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lph46" (UniqueName: "kubernetes.io/projected/116a1241-c9a3-4664-8217-60c2b6aa8b76-kube-api-access-lph46") pod "network-check-target-kvw9t" (UID: "116a1241-c9a3-4664-8217-60c2b6aa8b76") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:46:55.347113 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:55.347079 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvw9t" Oct 30 19:46:55.347240 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:55.347150 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7f4jl" Oct 30 19:46:55.347240 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:55.347230 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvw9t" podUID="116a1241-c9a3-4664-8217-60c2b6aa8b76" Oct 30 19:46:55.347363 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:55.347241 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hz6jn" Oct 30 19:46:55.347363 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:55.347333 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7f4jl" podUID="57ddd2a0-f081-4f90-95a6-0f6f8df2f433" Oct 30 19:46:55.347470 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:55.347425 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hz6jn" podUID="73bb0a1f-597a-4078-8a01-20d0ed24715b" Oct 30 19:46:56.574228 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:56.574200 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xpjpn_688d079a-932d-4f63-97da-5cd6222a03e1/ovn-acl-logging/0.log" Oct 30 19:46:56.574740 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:56.574522 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" event={"ID":"688d079a-932d-4f63-97da-5cd6222a03e1","Type":"ContainerStarted","Data":"fac417058435a7534833b1c1d6d435415b9cee7cc606a283fe18bb49dcb392b1"} Oct 30 19:46:57.346824 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:57.346791 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hz6jn" Oct 30 19:46:57.346994 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:57.346794 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7f4jl" Oct 30 19:46:57.346994 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:57.346878 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hz6jn" podUID="73bb0a1f-597a-4078-8a01-20d0ed24715b" Oct 30 19:46:57.346994 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:57.346983 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7f4jl" podUID="57ddd2a0-f081-4f90-95a6-0f6f8df2f433" Oct 30 19:46:57.347223 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:57.346797 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvw9t" Oct 30 19:46:57.347223 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:57.347074 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvw9t" podUID="116a1241-c9a3-4664-8217-60c2b6aa8b76" Oct 30 19:46:58.399869 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:58.399838 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-djkzp" Oct 30 19:46:58.400379 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:58.399931 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:46:58.400600 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:58.400584 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-djkzp" Oct 30 19:46:58.582835 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:58.582677 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xpjpn_688d079a-932d-4f63-97da-5cd6222a03e1/ovn-acl-logging/0.log" Oct 30 19:46:58.583259 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:58.583231 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" event={"ID":"688d079a-932d-4f63-97da-5cd6222a03e1","Type":"ContainerStarted","Data":"cb065ab5a16aad9cd8c460b6e1d8fa0cc0782dfaf5eb2a2e75289ee1bdba969f"} Oct 30 19:46:58.583692 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:58.583671 2578 scope.go:122] "RemoveContainer" containerID="5fe9a2e1adf6d1e2dcd320f39f5481d5a67b40aae630fc540688ebcc8c1845e7" Oct 30 19:46:59.347084 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:59.347059 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvw9t" Oct 30 19:46:59.347231 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:59.347129 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7f4jl" Oct 30 19:46:59.347231 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:59.347152 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hz6jn" Oct 30 19:46:59.347290 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:59.347226 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7f4jl" podUID="57ddd2a0-f081-4f90-95a6-0f6f8df2f433" Oct 30 19:46:59.347290 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:59.347264 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hz6jn" podUID="73bb0a1f-597a-4078-8a01-20d0ed24715b" Oct 30 19:46:59.347354 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:46:59.347317 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvw9t" podUID="116a1241-c9a3-4664-8217-60c2b6aa8b76" Oct 30 19:46:59.588381 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:59.588352 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xpjpn_688d079a-932d-4f63-97da-5cd6222a03e1/ovn-acl-logging/0.log" Oct 30 19:46:59.588755 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:59.588648 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" event={"ID":"688d079a-932d-4f63-97da-5cd6222a03e1","Type":"ContainerStarted","Data":"df5681374cf6b4a611f4ea8210f6a6a87408d8a23e17384edfc5ac2d3ea1cfc7"} Oct 30 19:46:59.588822 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:59.588809 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:59.588947 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:59.588929 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:59.589013 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:59.589001 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:59.603123 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:59.603073 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:59.603216 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:59.603163 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:46:59.669220 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:46:59.669172 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" podStartSLOduration=7.585014808 podStartE2EDuration="21.669161706s" totalImagesPullingTime="14.084146898s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:46:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:59.668923461 +0000 UTC m=+21.792134779" watchObservedRunningTime="2026-10-30 19:46:59.669161706 +0000 UTC m=+21.792373014" Oct 30 19:47:00.542778 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:00.542746 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hz6jn"] Oct 30 19:47:00.542971 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:00.542868 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hz6jn" Oct 30 19:47:00.542971 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:00.542956 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hz6jn" podUID="73bb0a1f-597a-4078-8a01-20d0ed24715b" Oct 30 19:47:00.546941 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:00.546800 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7f4jl"] Oct 30 19:47:00.547232 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:00.547208 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7f4jl" Oct 30 19:47:00.547352 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:00.547332 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7f4jl" podUID="57ddd2a0-f081-4f90-95a6-0f6f8df2f433" Oct 30 19:47:00.548815 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:00.548793 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kvw9t"] Oct 30 19:47:00.548908 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:00.548899 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvw9t" Oct 30 19:47:00.549006 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:00.548990 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvw9t" podUID="116a1241-c9a3-4664-8217-60c2b6aa8b76" Oct 30 19:47:02.347526 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:02.347496 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hz6jn" Oct 30 19:47:02.347526 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:02.347506 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7f4jl" Oct 30 19:47:02.348076 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:02.347514 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvw9t" Oct 30 19:47:02.348076 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:02.347596 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hz6jn" podUID="73bb0a1f-597a-4078-8a01-20d0ed24715b" Oct 30 19:47:02.348076 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:02.347667 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7f4jl" podUID="57ddd2a0-f081-4f90-95a6-0f6f8df2f433" Oct 30 19:47:02.348076 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:02.347738 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvw9t" podUID="116a1241-c9a3-4664-8217-60c2b6aa8b76" Oct 30 19:47:04.346969 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:04.346939 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7f4jl" Oct 30 19:47:04.346969 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:04.346962 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hz6jn" Oct 30 19:47:04.347384 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:04.346988 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvw9t" Oct 30 19:47:04.347384 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:04.347099 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hz6jn" podUID="73bb0a1f-597a-4078-8a01-20d0ed24715b" Oct 30 19:47:04.347384 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:04.347150 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7f4jl" podUID="57ddd2a0-f081-4f90-95a6-0f6f8df2f433" Oct 30 19:47:04.347384 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:04.347187 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvw9t" podUID="116a1241-c9a3-4664-8217-60c2b6aa8b76" Oct 30 19:47:06.346946 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:06.346786 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hz6jn" Oct 30 19:47:06.347427 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:06.346821 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7f4jl" Oct 30 19:47:06.347427 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:06.347028 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hz6jn" podUID="73bb0a1f-597a-4078-8a01-20d0ed24715b" Oct 30 19:47:06.347427 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:06.347130 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7f4jl" podUID="57ddd2a0-f081-4f90-95a6-0f6f8df2f433" Oct 30 19:47:06.347427 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:06.346880 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvw9t" Oct 30 19:47:06.347427 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:06.347242 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvw9t" podUID="116a1241-c9a3-4664-8217-60c2b6aa8b76" Oct 30 19:47:07.154850 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.154784 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeReady" Oct 30 19:47:07.154993 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.154876 2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 19:47:07.210626 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.210583 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-c7n9t"] Oct 30 19:47:07.218824 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.218803 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-r7djq"] Oct 30 19:47:07.218946 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.218926 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-c7n9t" Oct 30 19:47:07.220920 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.220900 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-6wqfc"] Oct 30 19:47:07.221028 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.221012 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-r7djq" Oct 30 19:47:07.223174 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.223158 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kflff"] Oct 30 19:47:07.223275 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.223259 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6wqfc" Oct 30 19:47:07.226985 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.226970 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5bc64fc876-25g7d"] Oct 30 19:47:07.227089 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.227071 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-kflff" Oct 30 19:47:07.229545 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.229527 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-bnp7s"] Oct 30 19:47:07.229672 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.229655 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:07.233044 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.233008 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-qdj4h"] Oct 30 19:47:07.233176 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.233155 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-bnp7s" Oct 30 19:47:07.236237 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.236220 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-tv4xh"] Oct 30 19:47:07.236335 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.236322 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-qdj4h" Oct 30 19:47:07.239242 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.239225 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95bqt"] Oct 30 19:47:07.239353 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.239339 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-tv4xh" Oct 30 19:47:07.240461 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.240445 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 19:47:07.240641 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.240630 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 19:47:07.241500 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.241485 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-79458f7dbc-vbb28"] Oct 30 19:47:07.241732 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.241636 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95bqt" Oct 30 19:47:07.246107 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.246086 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-qdv4s"] Oct 30 19:47:07.246206 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.246174 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:07.247076 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.247057 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 19:47:07.248638 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.248603 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 19:47:07.248761 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.248655 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 19:47:07.248761 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.248665 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 19:47:07.248761 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.248757 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4vv2n\"" Oct 30 19:47:07.249331 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.249164 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n75h8"] Oct 30 19:47:07.249896 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.249876 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qdv4s" Oct 30 19:47:07.253745 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.253725 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6796cf4b57-b9d8k"] Oct 30 19:47:07.254094 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.254078 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n75h8" Oct 30 19:47:07.256423 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.256403 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ml5lp"] Oct 30 19:47:07.256525 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.256511 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:07.260072 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.260056 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-k8mq4"] Oct 30 19:47:07.260198 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.260181 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ml5lp" Oct 30 19:47:07.263260 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.263245 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xlgtn"] Oct 30 19:47:07.263352 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.263339 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-k8mq4" Oct 30 19:47:07.265857 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.265843 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lk574"] Oct 30 19:47:07.265959 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.265945 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xlgtn" Oct 30 19:47:07.270901 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.270883 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-c7n9t"] Oct 30 19:47:07.270975 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.270911 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-6wqfc"] Oct 30 19:47:07.270975 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.270929 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kflff"] Oct 30 19:47:07.270975 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.270948 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-r7djq"] Oct 30 19:47:07.270975 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.270968 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95bqt"] Oct 30 19:47:07.271104 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.271002 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lk574" Oct 30 19:47:07.302462 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.302440 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-79458f7dbc-vbb28"] Oct 30 19:47:07.308010 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.307718 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-qdj4h"] Oct 30 19:47:07.308109 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.308088 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/74b7c739-483f-4756-b97b-e65d37387b17-config\") pod \"service-ca-operator-544d749c8c-6wqfc\" (UID: \"74b7c739-483f-4756-b97b-e65d37387b17\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6wqfc" Oct 30 19:47:07.308148 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.308108 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/74b7c739-483f-4756-b97b-e65d37387b17-serving-cert\") pod \"service-ca-operator-544d749c8c-6wqfc\" (UID: \"74b7c739-483f-4756-b97b-e65d37387b17\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6wqfc" Oct 30 19:47:07.308148 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.308125 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-image-registry-private-configuration\") pod \"image-registry-5bc64fc876-25g7d\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:07.308223 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.308153 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da850825-59ab-480f-9047-afd7d2bdc19f-trusted-ca\") pod \"console-operator-7cd56b8dbc-qdj4h\" (UID: \"da850825-59ab-480f-9047-afd7d2bdc19f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qdj4h" Oct 30 19:47:07.308223 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.308169 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-bound-sa-token\") pod \"image-registry-5bc64fc876-25g7d\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:07.308223 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.308185 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d78606e4-ccb9-421a-8ae5-8fbecbcd3831-snapshots\") pod \"insights-operator-7c4986d449-bnp7s\" (UID: \"d78606e4-ccb9-421a-8ae5-8fbecbcd3831\") " pod="openshift-insights/insights-operator-7c4986d449-bnp7s" Oct 30 19:47:07.308223 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.308199 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzg2k\" (UniqueName: \"kubernetes.io/projected/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-kube-api-access-rzg2k\") pod \"image-registry-5bc64fc876-25g7d\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:07.308223 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.308216 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/afb6fa1c-9722-42c4-a914-4941bcbcaaad-nginx-conf\") pod \"networking-console-plugin-86c478bb88-tv4xh\" (UID: \"afb6fa1c-9722-42c4-a914-4941bcbcaaad\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-tv4xh" Oct 30 19:47:07.308360 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.308234 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-trusted-ca\") pod \"image-registry-5bc64fc876-25g7d\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:07.308360 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.308250 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a421de2b-dd41-4862-9849-0a4f0256af1b-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-95bqt\" (UID: \"a421de2b-dd41-4862-9849-0a4f0256af1b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95bqt" Oct 30 19:47:07.308360 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.308264 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrg8k\" (UniqueName: \"kubernetes.io/projected/a421de2b-dd41-4862-9849-0a4f0256af1b-kube-api-access-rrg8k\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-95bqt\" (UID: \"a421de2b-dd41-4862-9849-0a4f0256af1b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95bqt" Oct 30 19:47:07.308360 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.308281 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/da850825-59ab-480f-9047-afd7d2bdc19f-serving-cert\") pod \"console-operator-7cd56b8dbc-qdj4h\" (UID: \"da850825-59ab-480f-9047-afd7d2bdc19f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qdj4h" Oct 30 19:47:07.308360 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.308297 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v26lp\" (UniqueName: \"kubernetes.io/projected/da850825-59ab-480f-9047-afd7d2bdc19f-kube-api-access-v26lp\") pod \"console-operator-7cd56b8dbc-qdj4h\" (UID: \"da850825-59ab-480f-9047-afd7d2bdc19f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qdj4h" Oct 30 19:47:07.308360 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.308312 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d78606e4-ccb9-421a-8ae5-8fbecbcd3831-service-ca-bundle\") pod \"insights-operator-7c4986d449-bnp7s\" (UID: \"d78606e4-ccb9-421a-8ae5-8fbecbcd3831\") " pod="openshift-insights/insights-operator-7c4986d449-bnp7s" Oct 30 19:47:07.308360 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.308331 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljrnk\" (UniqueName: \"kubernetes.io/projected/d78606e4-ccb9-421a-8ae5-8fbecbcd3831-kube-api-access-ljrnk\") pod \"insights-operator-7c4986d449-bnp7s\" (UID: \"d78606e4-ccb9-421a-8ae5-8fbecbcd3831\") " pod="openshift-insights/insights-operator-7c4986d449-bnp7s" Oct 30 19:47:07.308360 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.308366 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d78606e4-ccb9-421a-8ae5-8fbecbcd3831-tmp\") pod \"insights-operator-7c4986d449-bnp7s\" (UID: \"d78606e4-ccb9-421a-8ae5-8fbecbcd3831\") " pod="openshift-insights/insights-operator-7c4986d449-bnp7s" Oct 30 19:47:07.308657 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.308380 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d78606e4-ccb9-421a-8ae5-8fbecbcd3831-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-bnp7s\" (UID: \"d78606e4-ccb9-421a-8ae5-8fbecbcd3831\") " pod="openshift-insights/insights-operator-7c4986d449-bnp7s" Oct 30 19:47:07.308657 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.308396 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/da850825-59ab-480f-9047-afd7d2bdc19f-config\") pod \"console-operator-7cd56b8dbc-qdj4h\" (UID: \"da850825-59ab-480f-9047-afd7d2bdc19f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qdj4h" Oct 30 19:47:07.308657 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.308411 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-registry-certificates\") pod \"image-registry-5bc64fc876-25g7d\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:07.308657 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.308426 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg9ks\" (UniqueName: \"kubernetes.io/projected/aab2acf3-4af5-4d2e-8336-05b01a2dca1e-kube-api-access-lg9ks\") pod \"kas-connection-checker-685895c7c9-kflff\" (UID: \"aab2acf3-4af5-4d2e-8336-05b01a2dca1e\") " pod="kube-system/kas-connection-checker-685895c7c9-kflff" Oct 30 19:47:07.308657 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.308445 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbnlm\" (UniqueName: \"kubernetes.io/projected/74b7c739-483f-4756-b97b-e65d37387b17-kube-api-access-gbnlm\") pod \"service-ca-operator-544d749c8c-6wqfc\" (UID: \"74b7c739-483f-4756-b97b-e65d37387b17\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6wqfc" Oct 30 19:47:07.308657 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.308467 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a421de2b-dd41-4862-9849-0a4f0256af1b-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-95bqt\" (UID: \"a421de2b-dd41-4862-9849-0a4f0256af1b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95bqt" Oct 30 19:47:07.308657 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.308484 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkbkr\" (UniqueName: \"kubernetes.io/projected/2962e3da-8439-4316-9c3a-1679842609b7-kube-api-access-rkbkr\") pod \"kas-connection-checker-685895c7c9-r7djq\" (UID: \"2962e3da-8439-4316-9c3a-1679842609b7\") " pod="kube-system/kas-connection-checker-685895c7c9-r7djq" Oct 30 19:47:07.308657 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.308505 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d78606e4-ccb9-421a-8ae5-8fbecbcd3831-serving-cert\") pod \"insights-operator-7c4986d449-bnp7s\" (UID: \"d78606e4-ccb9-421a-8ae5-8fbecbcd3831\") " pod="openshift-insights/insights-operator-7c4986d449-bnp7s" Oct 30 19:47:07.308657 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.308524 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-installation-pull-secrets\") pod \"image-registry-5bc64fc876-25g7d\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:07.308657 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.308539 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/afb6fa1c-9722-42c4-a914-4941bcbcaaad-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-tv4xh\" (UID: \"afb6fa1c-9722-42c4-a914-4941bcbcaaad\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-tv4xh" Oct 30 19:47:07.308657 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.308555 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58q6j\" (UniqueName: \"kubernetes.io/projected/908c2874-510c-43c4-a07c-c1771b924e8d-kube-api-access-58q6j\") pod \"kas-connection-checker-685895c7c9-c7n9t\" (UID: \"908c2874-510c-43c4-a07c-c1771b924e8d\") " pod="kube-system/kas-connection-checker-685895c7c9-c7n9t" Oct 30 19:47:07.308657 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.308571 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-registry-tls\") pod \"image-registry-5bc64fc876-25g7d\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:07.308657 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.308594 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-ca-trust-extracted\") pod \"image-registry-5bc64fc876-25g7d\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:07.309151 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.308987 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-k8mq4"] Oct 30 19:47:07.309295 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.309220 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-bnp7s"] Oct 30 19:47:07.312072 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.312054 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ml5lp"] Oct 30 19:47:07.313040 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.313024 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xlgtn"] Oct 30 19:47:07.314004 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.313988 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6796cf4b57-b9d8k"] Oct 30 19:47:07.314900 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.314883 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n75h8"] Oct 30 19:47:07.315867 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.315847 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5bc64fc876-25g7d"] Oct 30 19:47:07.330019 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.329874 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lk574"] Oct 30 19:47:07.333767 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.333749 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-tv4xh"] Oct 30 19:47:07.344148 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.344126 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 19:47:07.344245 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.344135 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:47:07.344245 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.344196 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 19:47:07.346645 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.345745 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 19:47:07.346645 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.345843 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 19:47:07.346645 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.345841 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:47:07.346645 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.345875 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 19:47:07.346645 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.345952 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 19:47:07.346645 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.346004 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 19:47:07.346645 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.346048 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:47:07.346645 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.346055 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 19:47:07.346645 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.346085 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 19:47:07.346645 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.346105 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 19:47:07.346645 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.346147 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 19:47:07.346645 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.346153 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 19:47:07.346645 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.346157 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 19:47:07.346645 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.346157 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:47:07.346645 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.346187 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 19:47:07.346645 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.346188 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 19:47:07.346645 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.346255 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 19:47:07.346645 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.346281 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 19:47:07.346645 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.346412 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 19:47:07.346645 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.346447 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:47:07.346645 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.346466 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 19:47:07.346645 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.346493 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 19:47:07.346645 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.346518 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 19:47:07.346645 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.346527 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 19:47:07.348019 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.346666 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 19:47:07.348019 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.347155 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-qdv4s"] Oct 30 19:47:07.362264 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.362206 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 19:47:07.372826 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.372805 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 19:47:07.375457 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.375426 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 19:47:07.375547 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.375489 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 19:47:07.375547 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.375537 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 19:47:07.375632 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.375547 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 19:47:07.375632 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.375538 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 19:47:07.375731 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.375661 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 19:47:07.375731 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.375687 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 19:47:07.375731 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.375693 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-l5g68\"" Oct 30 19:47:07.375731 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.375715 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 19:47:07.375731 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.375716 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 19:47:07.375901 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.375746 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 19:47:07.375901 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.375759 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 19:47:07.375965 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.375927 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 19:47:07.376006 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.375978 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-h5z6r\"" Oct 30 19:47:07.376321 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.376305 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 19:47:07.409175 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.409125 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d78606e4-ccb9-421a-8ae5-8fbecbcd3831-service-ca-bundle\") pod \"insights-operator-7c4986d449-bnp7s\" (UID: \"d78606e4-ccb9-421a-8ae5-8fbecbcd3831\") " pod="openshift-insights/insights-operator-7c4986d449-bnp7s" Oct 30 19:47:07.409175 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.409156 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1342ef67-5f63-47fb-a337-64c693e11228-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n75h8\" (UID: \"1342ef67-5f63-47fb-a337-64c693e11228\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n75h8" Oct 30 19:47:07.409287 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.409190 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b1440092-9901-4eff-8a9d-6174b2dfdada-ca-trust-extracted\") pod \"image-registry-6796cf4b57-b9d8k\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:07.409287 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.409216 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljrnk\" (UniqueName: \"kubernetes.io/projected/d78606e4-ccb9-421a-8ae5-8fbecbcd3831-kube-api-access-ljrnk\") pod \"insights-operator-7c4986d449-bnp7s\" (UID: \"d78606e4-ccb9-421a-8ae5-8fbecbcd3831\") " pod="openshift-insights/insights-operator-7c4986d449-bnp7s" Oct 30 19:47:07.409287 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.409255 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b1440092-9901-4eff-8a9d-6174b2dfdada-bound-sa-token\") pod \"image-registry-6796cf4b57-b9d8k\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:07.409389 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.409286 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpkkb\" (UniqueName: \"kubernetes.io/projected/cba5fe49-b5b5-413e-9085-e35762430fa9-kube-api-access-hpkkb\") pod \"network-check-source-6f78d59f99-k8mq4\" (UID: \"cba5fe49-b5b5-413e-9085-e35762430fa9\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-k8mq4" Oct 30 19:47:07.409389 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.409377 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d78606e4-ccb9-421a-8ae5-8fbecbcd3831-tmp\") pod \"insights-operator-7c4986d449-bnp7s\" (UID: \"d78606e4-ccb9-421a-8ae5-8fbecbcd3831\") " pod="openshift-insights/insights-operator-7c4986d449-bnp7s" Oct 30 19:47:07.409467 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.409407 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d78606e4-ccb9-421a-8ae5-8fbecbcd3831-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-bnp7s\" (UID: \"d78606e4-ccb9-421a-8ae5-8fbecbcd3831\") " pod="openshift-insights/insights-operator-7c4986d449-bnp7s" Oct 30 19:47:07.409467 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.409436 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b1440092-9901-4eff-8a9d-6174b2dfdada-image-registry-private-configuration\") pod \"image-registry-6796cf4b57-b9d8k\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:07.409467 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.409464 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/da850825-59ab-480f-9047-afd7d2bdc19f-config\") pod \"console-operator-7cd56b8dbc-qdj4h\" (UID: \"da850825-59ab-480f-9047-afd7d2bdc19f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qdj4h" Oct 30 19:47:07.409598 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.409485 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b1440092-9901-4eff-8a9d-6174b2dfdada-installation-pull-secrets\") pod \"image-registry-6796cf4b57-b9d8k\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:07.409598 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.409513 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-registry-certificates\") pod \"image-registry-5bc64fc876-25g7d\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:07.409598 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.409543 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg9ks\" (UniqueName: \"kubernetes.io/projected/aab2acf3-4af5-4d2e-8336-05b01a2dca1e-kube-api-access-lg9ks\") pod \"kas-connection-checker-685895c7c9-kflff\" (UID: \"aab2acf3-4af5-4d2e-8336-05b01a2dca1e\") " pod="kube-system/kas-connection-checker-685895c7c9-kflff" Oct 30 19:47:07.409779 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.409759 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d78606e4-ccb9-421a-8ae5-8fbecbcd3831-service-ca-bundle\") pod \"insights-operator-7c4986d449-bnp7s\" (UID: \"d78606e4-ccb9-421a-8ae5-8fbecbcd3831\") " pod="openshift-insights/insights-operator-7c4986d449-bnp7s" Oct 30 19:47:07.409825 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.409761 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d78606e4-ccb9-421a-8ae5-8fbecbcd3831-tmp\") pod \"insights-operator-7c4986d449-bnp7s\" (UID: \"d78606e4-ccb9-421a-8ae5-8fbecbcd3831\") " pod="openshift-insights/insights-operator-7c4986d449-bnp7s" Oct 30 19:47:07.409825 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.409765 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbnlm\" (UniqueName: \"kubernetes.io/projected/74b7c739-483f-4756-b97b-e65d37387b17-kube-api-access-gbnlm\") pod \"service-ca-operator-544d749c8c-6wqfc\" (UID: \"74b7c739-483f-4756-b97b-e65d37387b17\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6wqfc" Oct 30 19:47:07.409915 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.409841 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj8ws\" (UniqueName: \"kubernetes.io/projected/1342ef67-5f63-47fb-a337-64c693e11228-kube-api-access-vj8ws\") pod \"cluster-samples-operator-6ddb6c8b94-n75h8\" (UID: \"1342ef67-5f63-47fb-a337-64c693e11228\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n75h8" Oct 30 19:47:07.409915 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.409881 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b1440092-9901-4eff-8a9d-6174b2dfdada-registry-certificates\") pod \"image-registry-6796cf4b57-b9d8k\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:07.409915 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.409902 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a421de2b-dd41-4862-9849-0a4f0256af1b-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-95bqt\" (UID: \"a421de2b-dd41-4862-9849-0a4f0256af1b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95bqt" Oct 30 19:47:07.410048 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.409978 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b1440092-9901-4eff-8a9d-6174b2dfdada-trusted-ca\") pod \"image-registry-6796cf4b57-b9d8k\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:07.410048 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.410021 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkbkr\" (UniqueName: \"kubernetes.io/projected/2962e3da-8439-4316-9c3a-1679842609b7-kube-api-access-rkbkr\") pod \"kas-connection-checker-685895c7c9-r7djq\" (UID: \"2962e3da-8439-4316-9c3a-1679842609b7\") " pod="kube-system/kas-connection-checker-685895c7c9-r7djq" Oct 30 19:47:07.410145 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.410053 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d78606e4-ccb9-421a-8ae5-8fbecbcd3831-serving-cert\") pod \"insights-operator-7c4986d449-bnp7s\" (UID: \"d78606e4-ccb9-421a-8ae5-8fbecbcd3831\") " pod="openshift-insights/insights-operator-7c4986d449-bnp7s" Oct 30 19:47:07.410145 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.410098 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8e3ed9d-5156-4698-82c3-df72fa728f94-metrics-tls\") pod \"dns-default-ml5lp\" (UID: \"b8e3ed9d-5156-4698-82c3-df72fa728f94\") " pod="openshift-dns/dns-default-ml5lp" Oct 30 19:47:07.410145 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.410130 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41534930-917f-44ec-bccd-86d6c6fc7c70-metrics-certs\") pod \"router-default-79458f7dbc-vbb28\" (UID: \"41534930-917f-44ec-bccd-86d6c6fc7c70\") " pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:07.410290 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.410160 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5ad8f4b0-50ba-4191-8df4-25930717acb0-cert\") pod \"ingress-canary-lk574\" (UID: \"5ad8f4b0-50ba-4191-8df4-25930717acb0\") " pod="openshift-ingress-canary/ingress-canary-lk574" Oct 30 19:47:07.410290 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.410205 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-registry-certificates\") pod \"image-registry-5bc64fc876-25g7d\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:07.410290 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.410205 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtksm\" (UniqueName: \"kubernetes.io/projected/b8e3ed9d-5156-4698-82c3-df72fa728f94-kube-api-access-xtksm\") pod \"dns-default-ml5lp\" (UID: \"b8e3ed9d-5156-4698-82c3-df72fa728f94\") " pod="openshift-dns/dns-default-ml5lp" Oct 30 19:47:07.410290 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.410223 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/da850825-59ab-480f-9047-afd7d2bdc19f-config\") pod \"console-operator-7cd56b8dbc-qdj4h\" (UID: \"da850825-59ab-480f-9047-afd7d2bdc19f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qdj4h" Oct 30 19:47:07.410290 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.410252 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rwww\" (UniqueName: \"kubernetes.io/projected/20f593c8-594e-4a64-9c5e-881311b269a3-kube-api-access-6rwww\") pod \"cluster-monitoring-operator-884576c57-qdv4s\" (UID: \"20f593c8-594e-4a64-9c5e-881311b269a3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qdv4s" Oct 30 19:47:07.410518 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.410295 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-installation-pull-secrets\") pod \"image-registry-5bc64fc876-25g7d\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:07.410518 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.410324 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/afb6fa1c-9722-42c4-a914-4941bcbcaaad-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-tv4xh\" (UID: \"afb6fa1c-9722-42c4-a914-4941bcbcaaad\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-tv4xh" Oct 30 19:47:07.410518 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.410342 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58q6j\" (UniqueName: \"kubernetes.io/projected/908c2874-510c-43c4-a07c-c1771b924e8d-kube-api-access-58q6j\") pod \"kas-connection-checker-685895c7c9-c7n9t\" (UID: \"908c2874-510c-43c4-a07c-c1771b924e8d\") " pod="kube-system/kas-connection-checker-685895c7c9-c7n9t" Oct 30 19:47:07.410518 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.410358 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-registry-tls\") pod \"image-registry-5bc64fc876-25g7d\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:07.410518 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.410377 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-ca-trust-extracted\") pod \"image-registry-5bc64fc876-25g7d\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:07.410518 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.410419 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/41534930-917f-44ec-bccd-86d6c6fc7c70-default-certificate\") pod \"router-default-79458f7dbc-vbb28\" (UID: \"41534930-917f-44ec-bccd-86d6c6fc7c70\") " pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:07.410518 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.410447 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/74b7c739-483f-4756-b97b-e65d37387b17-config\") pod \"service-ca-operator-544d749c8c-6wqfc\" (UID: \"74b7c739-483f-4756-b97b-e65d37387b17\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6wqfc" Oct 30 19:47:07.410518 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:07.410456 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 19:47:07.410518 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.410471 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1440092-9901-4eff-8a9d-6174b2dfdada-registry-tls\") pod \"image-registry-6796cf4b57-b9d8k\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:07.410518 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.410495 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/74b7c739-483f-4756-b97b-e65d37387b17-serving-cert\") pod \"service-ca-operator-544d749c8c-6wqfc\" (UID: \"74b7c739-483f-4756-b97b-e65d37387b17\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6wqfc" Oct 30 19:47:07.410518 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:07.410507 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afb6fa1c-9722-42c4-a914-4941bcbcaaad-networking-console-plugin-cert podName:afb6fa1c-9722-42c4-a914-4941bcbcaaad nodeName:}" failed. No retries permitted until 2026-10-30 19:47:07.9104915 +0000 UTC m=+30.033702796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/afb6fa1c-9722-42c4-a914-4941bcbcaaad-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-tv4xh" (UID: "afb6fa1c-9722-42c4-a914-4941bcbcaaad") : secret "networking-console-plugin-cert" not found Oct 30 19:47:07.411038 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.410534 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-image-registry-private-configuration\") pod \"image-registry-5bc64fc876-25g7d\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:07.411038 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.410560 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxq6z\" (UniqueName: \"kubernetes.io/projected/5ad8f4b0-50ba-4191-8df4-25930717acb0-kube-api-access-lxq6z\") pod \"ingress-canary-lk574\" (UID: \"5ad8f4b0-50ba-4191-8df4-25930717acb0\") " pod="openshift-ingress-canary/ingress-canary-lk574" Oct 30 19:47:07.411038 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.410585 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b8e3ed9d-5156-4698-82c3-df72fa728f94-config-volume\") pod \"dns-default-ml5lp\" (UID: \"b8e3ed9d-5156-4698-82c3-df72fa728f94\") " pod="openshift-dns/dns-default-ml5lp" Oct 30 19:47:07.411038 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.410607 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x59q2\" (UniqueName: \"kubernetes.io/projected/41534930-917f-44ec-bccd-86d6c6fc7c70-kube-api-access-x59q2\") pod \"router-default-79458f7dbc-vbb28\" (UID: \"41534930-917f-44ec-bccd-86d6c6fc7c70\") " pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:07.411038 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.410655 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv4p7\" (UniqueName: \"kubernetes.io/projected/9b6dd02b-d4bd-4959-9a06-8941d6baf1a5-kube-api-access-sv4p7\") pod \"volume-data-source-validator-58c9f5f46d-xlgtn\" (UID: \"9b6dd02b-d4bd-4959-9a06-8941d6baf1a5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xlgtn" Oct 30 19:47:07.411038 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.410697 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da850825-59ab-480f-9047-afd7d2bdc19f-trusted-ca\") pod \"console-operator-7cd56b8dbc-qdj4h\" (UID: \"da850825-59ab-480f-9047-afd7d2bdc19f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qdj4h" Oct 30 19:47:07.411038 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.410722 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-bound-sa-token\") pod \"image-registry-5bc64fc876-25g7d\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:07.411038 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.410750 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41534930-917f-44ec-bccd-86d6c6fc7c70-service-ca-bundle\") pod \"router-default-79458f7dbc-vbb28\" (UID: \"41534930-917f-44ec-bccd-86d6c6fc7c70\") " pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:07.411038 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.410774 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/41534930-917f-44ec-bccd-86d6c6fc7c70-stats-auth\") pod \"router-default-79458f7dbc-vbb28\" (UID: \"41534930-917f-44ec-bccd-86d6c6fc7c70\") " pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:07.411038 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.410797 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d78606e4-ccb9-421a-8ae5-8fbecbcd3831-snapshots\") pod \"insights-operator-7c4986d449-bnp7s\" (UID: \"d78606e4-ccb9-421a-8ae5-8fbecbcd3831\") " pod="openshift-insights/insights-operator-7c4986d449-bnp7s" Oct 30 19:47:07.411038 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.410828 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzg2k\" (UniqueName: \"kubernetes.io/projected/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-kube-api-access-rzg2k\") pod \"image-registry-5bc64fc876-25g7d\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:07.411038 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.410846 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-ca-trust-extracted\") pod \"image-registry-5bc64fc876-25g7d\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:07.411038 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.410866 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1342ef67-5f63-47fb-a337-64c693e11228-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-n75h8\" (UID: \"1342ef67-5f63-47fb-a337-64c693e11228\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n75h8" Oct 30 19:47:07.411038 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.410871 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d78606e4-ccb9-421a-8ae5-8fbecbcd3831-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-bnp7s\" (UID: \"d78606e4-ccb9-421a-8ae5-8fbecbcd3831\") " pod="openshift-insights/insights-operator-7c4986d449-bnp7s" Oct 30 19:47:07.411038 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.410892 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/afb6fa1c-9722-42c4-a914-4941bcbcaaad-nginx-conf\") pod \"networking-console-plugin-86c478bb88-tv4xh\" (UID: \"afb6fa1c-9722-42c4-a914-4941bcbcaaad\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-tv4xh" Oct 30 19:47:07.411038 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:07.410926 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:47:07.411758 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.410928 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/20f593c8-594e-4a64-9c5e-881311b269a3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qdv4s\" (UID: \"20f593c8-594e-4a64-9c5e-881311b269a3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qdv4s" Oct 30 19:47:07.411758 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:07.410937 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bc64fc876-25g7d: secret "image-registry-tls" not found Oct 30 19:47:07.411758 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.410956 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-trusted-ca\") pod \"image-registry-5bc64fc876-25g7d\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:07.411758 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.410980 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a421de2b-dd41-4862-9849-0a4f0256af1b-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-95bqt\" (UID: \"a421de2b-dd41-4862-9849-0a4f0256af1b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95bqt" Oct 30 19:47:07.411758 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:07.411004 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-registry-tls podName:9a8ef63e-bcc4-4b71-b345-3fcbc48f4396 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:07.9109908 +0000 UTC m=+30.034202095 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-registry-tls") pod "image-registry-5bc64fc876-25g7d" (UID: "9a8ef63e-bcc4-4b71-b345-3fcbc48f4396") : secret "image-registry-tls" not found Oct 30 19:47:07.411758 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.411023 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrg8k\" (UniqueName: \"kubernetes.io/projected/a421de2b-dd41-4862-9849-0a4f0256af1b-kube-api-access-rrg8k\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-95bqt\" (UID: \"a421de2b-dd41-4862-9849-0a4f0256af1b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95bqt" Oct 30 19:47:07.411758 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.411050 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8e3ed9d-5156-4698-82c3-df72fa728f94-tmp-dir\") pod \"dns-default-ml5lp\" (UID: \"b8e3ed9d-5156-4698-82c3-df72fa728f94\") " pod="openshift-dns/dns-default-ml5lp" Oct 30 19:47:07.411758 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.411075 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xtvg\" (UniqueName: \"kubernetes.io/projected/b1440092-9901-4eff-8a9d-6174b2dfdada-kube-api-access-8xtvg\") pod \"image-registry-6796cf4b57-b9d8k\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:07.411758 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.411098 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/da850825-59ab-480f-9047-afd7d2bdc19f-serving-cert\") pod \"console-operator-7cd56b8dbc-qdj4h\" (UID: \"da850825-59ab-480f-9047-afd7d2bdc19f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qdj4h" Oct 30 19:47:07.411758 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.411120 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v26lp\" (UniqueName: \"kubernetes.io/projected/da850825-59ab-480f-9047-afd7d2bdc19f-kube-api-access-v26lp\") pod \"console-operator-7cd56b8dbc-qdj4h\" (UID: \"da850825-59ab-480f-9047-afd7d2bdc19f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qdj4h" Oct 30 19:47:07.411758 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.411141 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/20f593c8-594e-4a64-9c5e-881311b269a3-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-qdv4s\" (UID: \"20f593c8-594e-4a64-9c5e-881311b269a3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qdv4s" Oct 30 19:47:07.411758 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.411421 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a421de2b-dd41-4862-9849-0a4f0256af1b-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-95bqt\" (UID: \"a421de2b-dd41-4862-9849-0a4f0256af1b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95bqt" Oct 30 19:47:07.412311 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.411935 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d78606e4-ccb9-421a-8ae5-8fbecbcd3831-snapshots\") pod \"insights-operator-7c4986d449-bnp7s\" (UID: \"d78606e4-ccb9-421a-8ae5-8fbecbcd3831\") " pod="openshift-insights/insights-operator-7c4986d449-bnp7s" Oct 30 19:47:07.412311 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.412294 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/afb6fa1c-9722-42c4-a914-4941bcbcaaad-nginx-conf\") pod \"networking-console-plugin-86c478bb88-tv4xh\" (UID: \"afb6fa1c-9722-42c4-a914-4941bcbcaaad\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-tv4xh" Oct 30 19:47:07.412943 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.412923 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da850825-59ab-480f-9047-afd7d2bdc19f-trusted-ca\") pod \"console-operator-7cd56b8dbc-qdj4h\" (UID: \"da850825-59ab-480f-9047-afd7d2bdc19f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qdj4h" Oct 30 19:47:07.413225 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.413198 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-trusted-ca\") pod \"image-registry-5bc64fc876-25g7d\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:07.413418 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.413392 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/74b7c739-483f-4756-b97b-e65d37387b17-config\") pod \"service-ca-operator-544d749c8c-6wqfc\" (UID: \"74b7c739-483f-4756-b97b-e65d37387b17\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6wqfc" Oct 30 19:47:07.414445 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.414420 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/74b7c739-483f-4756-b97b-e65d37387b17-serving-cert\") pod \"service-ca-operator-544d749c8c-6wqfc\" (UID: \"74b7c739-483f-4756-b97b-e65d37387b17\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6wqfc" Oct 30 19:47:07.414544 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.414502 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a421de2b-dd41-4862-9849-0a4f0256af1b-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-95bqt\" (UID: \"a421de2b-dd41-4862-9849-0a4f0256af1b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95bqt" Oct 30 19:47:07.414591 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.414538 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/da850825-59ab-480f-9047-afd7d2bdc19f-serving-cert\") pod \"console-operator-7cd56b8dbc-qdj4h\" (UID: \"da850825-59ab-480f-9047-afd7d2bdc19f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qdj4h" Oct 30 19:47:07.414591 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.414541 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d78606e4-ccb9-421a-8ae5-8fbecbcd3831-serving-cert\") pod \"insights-operator-7c4986d449-bnp7s\" (UID: \"d78606e4-ccb9-421a-8ae5-8fbecbcd3831\") " pod="openshift-insights/insights-operator-7c4986d449-bnp7s" Oct 30 19:47:07.414591 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.414567 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-image-registry-private-configuration\") pod \"image-registry-5bc64fc876-25g7d\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:07.414699 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.414566 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-installation-pull-secrets\") pod \"image-registry-5bc64fc876-25g7d\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:07.439954 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.439927 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljrnk\" (UniqueName: \"kubernetes.io/projected/d78606e4-ccb9-421a-8ae5-8fbecbcd3831-kube-api-access-ljrnk\") pod \"insights-operator-7c4986d449-bnp7s\" (UID: \"d78606e4-ccb9-421a-8ae5-8fbecbcd3831\") " pod="openshift-insights/insights-operator-7c4986d449-bnp7s" Oct 30 19:47:07.441345 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.441295 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbnlm\" (UniqueName: \"kubernetes.io/projected/74b7c739-483f-4756-b97b-e65d37387b17-kube-api-access-gbnlm\") pod \"service-ca-operator-544d749c8c-6wqfc\" (UID: \"74b7c739-483f-4756-b97b-e65d37387b17\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6wqfc" Oct 30 19:47:07.441949 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.441911 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkbkr\" (UniqueName: \"kubernetes.io/projected/2962e3da-8439-4316-9c3a-1679842609b7-kube-api-access-rkbkr\") pod \"kas-connection-checker-685895c7c9-r7djq\" (UID: \"2962e3da-8439-4316-9c3a-1679842609b7\") " pod="kube-system/kas-connection-checker-685895c7c9-r7djq" Oct 30 19:47:07.441949 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.441929 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrg8k\" (UniqueName: \"kubernetes.io/projected/a421de2b-dd41-4862-9849-0a4f0256af1b-kube-api-access-rrg8k\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-95bqt\" (UID: \"a421de2b-dd41-4862-9849-0a4f0256af1b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95bqt" Oct 30 19:47:07.441949 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.441919 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg9ks\" (UniqueName: \"kubernetes.io/projected/aab2acf3-4af5-4d2e-8336-05b01a2dca1e-kube-api-access-lg9ks\") pod \"kas-connection-checker-685895c7c9-kflff\" (UID: \"aab2acf3-4af5-4d2e-8336-05b01a2dca1e\") " pod="kube-system/kas-connection-checker-685895c7c9-kflff" Oct 30 19:47:07.442159 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.442013 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58q6j\" (UniqueName: \"kubernetes.io/projected/908c2874-510c-43c4-a07c-c1771b924e8d-kube-api-access-58q6j\") pod \"kas-connection-checker-685895c7c9-c7n9t\" (UID: \"908c2874-510c-43c4-a07c-c1771b924e8d\") " pod="kube-system/kas-connection-checker-685895c7c9-c7n9t" Oct 30 19:47:07.442219 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.442168 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v26lp\" (UniqueName: \"kubernetes.io/projected/da850825-59ab-480f-9047-afd7d2bdc19f-kube-api-access-v26lp\") pod \"console-operator-7cd56b8dbc-qdj4h\" (UID: \"da850825-59ab-480f-9047-afd7d2bdc19f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qdj4h" Oct 30 19:47:07.442365 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.442347 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzg2k\" (UniqueName: \"kubernetes.io/projected/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-kube-api-access-rzg2k\") pod \"image-registry-5bc64fc876-25g7d\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:07.442782 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.442764 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-bound-sa-token\") pod \"image-registry-5bc64fc876-25g7d\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:07.511849 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.511828 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b1440092-9901-4eff-8a9d-6174b2dfdada-trusted-ca\") pod \"image-registry-6796cf4b57-b9d8k\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:07.511986 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.511855 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8e3ed9d-5156-4698-82c3-df72fa728f94-metrics-tls\") pod \"dns-default-ml5lp\" (UID: \"b8e3ed9d-5156-4698-82c3-df72fa728f94\") " pod="openshift-dns/dns-default-ml5lp" Oct 30 19:47:07.511986 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.511871 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41534930-917f-44ec-bccd-86d6c6fc7c70-metrics-certs\") pod \"router-default-79458f7dbc-vbb28\" (UID: \"41534930-917f-44ec-bccd-86d6c6fc7c70\") " pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:07.511986 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.511887 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5ad8f4b0-50ba-4191-8df4-25930717acb0-cert\") pod \"ingress-canary-lk574\" (UID: \"5ad8f4b0-50ba-4191-8df4-25930717acb0\") " pod="openshift-ingress-canary/ingress-canary-lk574" Oct 30 19:47:07.511986 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.511909 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtksm\" (UniqueName: \"kubernetes.io/projected/b8e3ed9d-5156-4698-82c3-df72fa728f94-kube-api-access-xtksm\") pod \"dns-default-ml5lp\" (UID: \"b8e3ed9d-5156-4698-82c3-df72fa728f94\") " pod="openshift-dns/dns-default-ml5lp" Oct 30 19:47:07.511986 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.511929 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rwww\" (UniqueName: \"kubernetes.io/projected/20f593c8-594e-4a64-9c5e-881311b269a3-kube-api-access-6rwww\") pod \"cluster-monitoring-operator-884576c57-qdv4s\" (UID: \"20f593c8-594e-4a64-9c5e-881311b269a3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qdv4s" Oct 30 19:47:07.511986 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.511963 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/41534930-917f-44ec-bccd-86d6c6fc7c70-default-certificate\") pod \"router-default-79458f7dbc-vbb28\" (UID: \"41534930-917f-44ec-bccd-86d6c6fc7c70\") " pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:07.512257 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.511987 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1440092-9901-4eff-8a9d-6174b2dfdada-registry-tls\") pod \"image-registry-6796cf4b57-b9d8k\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:07.512257 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.512010 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxq6z\" (UniqueName: \"kubernetes.io/projected/5ad8f4b0-50ba-4191-8df4-25930717acb0-kube-api-access-lxq6z\") pod \"ingress-canary-lk574\" (UID: \"5ad8f4b0-50ba-4191-8df4-25930717acb0\") " pod="openshift-ingress-canary/ingress-canary-lk574" Oct 30 19:47:07.512257 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.512025 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b8e3ed9d-5156-4698-82c3-df72fa728f94-config-volume\") pod \"dns-default-ml5lp\" (UID: \"b8e3ed9d-5156-4698-82c3-df72fa728f94\") " pod="openshift-dns/dns-default-ml5lp" Oct 30 19:47:07.512257 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.512038 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x59q2\" (UniqueName: \"kubernetes.io/projected/41534930-917f-44ec-bccd-86d6c6fc7c70-kube-api-access-x59q2\") pod \"router-default-79458f7dbc-vbb28\" (UID: \"41534930-917f-44ec-bccd-86d6c6fc7c70\") " pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:07.512257 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.512056 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv4p7\" (UniqueName: \"kubernetes.io/projected/9b6dd02b-d4bd-4959-9a06-8941d6baf1a5-kube-api-access-sv4p7\") pod \"volume-data-source-validator-58c9f5f46d-xlgtn\" (UID: \"9b6dd02b-d4bd-4959-9a06-8941d6baf1a5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xlgtn" Oct 30 19:47:07.512257 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.512166 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41534930-917f-44ec-bccd-86d6c6fc7c70-service-ca-bundle\") pod \"router-default-79458f7dbc-vbb28\" (UID: \"41534930-917f-44ec-bccd-86d6c6fc7c70\") " pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:07.512257 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:07.512176 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 19:47:07.512257 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.512217 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/41534930-917f-44ec-bccd-86d6c6fc7c70-stats-auth\") pod \"router-default-79458f7dbc-vbb28\" (UID: \"41534930-917f-44ec-bccd-86d6c6fc7c70\") " pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:07.512257 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:07.512238 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41534930-917f-44ec-bccd-86d6c6fc7c70-metrics-certs podName:41534930-917f-44ec-bccd-86d6c6fc7c70 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:08.012220713 +0000 UTC m=+30.135432030 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/41534930-917f-44ec-bccd-86d6c6fc7c70-metrics-certs") pod "router-default-79458f7dbc-vbb28" (UID: "41534930-917f-44ec-bccd-86d6c6fc7c70") : secret "router-metrics-certs-default" not found Oct 30 19:47:07.512698 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.512290 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1342ef67-5f63-47fb-a337-64c693e11228-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-n75h8\" (UID: \"1342ef67-5f63-47fb-a337-64c693e11228\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n75h8" Oct 30 19:47:07.512698 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.512335 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/20f593c8-594e-4a64-9c5e-881311b269a3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qdv4s\" (UID: \"20f593c8-594e-4a64-9c5e-881311b269a3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qdv4s" Oct 30 19:47:07.512698 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.512374 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8e3ed9d-5156-4698-82c3-df72fa728f94-tmp-dir\") pod \"dns-default-ml5lp\" (UID: \"b8e3ed9d-5156-4698-82c3-df72fa728f94\") " pod="openshift-dns/dns-default-ml5lp" Oct 30 19:47:07.512698 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.512402 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xtvg\" (UniqueName: \"kubernetes.io/projected/b1440092-9901-4eff-8a9d-6174b2dfdada-kube-api-access-8xtvg\") pod \"image-registry-6796cf4b57-b9d8k\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:07.512698 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.512443 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/20f593c8-594e-4a64-9c5e-881311b269a3-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-qdv4s\" (UID: \"20f593c8-594e-4a64-9c5e-881311b269a3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qdv4s" Oct 30 19:47:07.512698 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.512471 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1342ef67-5f63-47fb-a337-64c693e11228-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n75h8\" (UID: \"1342ef67-5f63-47fb-a337-64c693e11228\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n75h8" Oct 30 19:47:07.512698 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.512502 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b1440092-9901-4eff-8a9d-6174b2dfdada-ca-trust-extracted\") pod \"image-registry-6796cf4b57-b9d8k\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:07.512698 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.512530 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b1440092-9901-4eff-8a9d-6174b2dfdada-bound-sa-token\") pod \"image-registry-6796cf4b57-b9d8k\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:07.512698 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.512563 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpkkb\" (UniqueName: \"kubernetes.io/projected/cba5fe49-b5b5-413e-9085-e35762430fa9-kube-api-access-hpkkb\") pod \"network-check-source-6f78d59f99-k8mq4\" (UID: \"cba5fe49-b5b5-413e-9085-e35762430fa9\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-k8mq4" Oct 30 19:47:07.512698 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.512603 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b1440092-9901-4eff-8a9d-6174b2dfdada-image-registry-private-configuration\") pod \"image-registry-6796cf4b57-b9d8k\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:07.512698 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.512652 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b1440092-9901-4eff-8a9d-6174b2dfdada-installation-pull-secrets\") pod \"image-registry-6796cf4b57-b9d8k\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:07.512698 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.512699 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj8ws\" (UniqueName: \"kubernetes.io/projected/1342ef67-5f63-47fb-a337-64c693e11228-kube-api-access-vj8ws\") pod \"cluster-samples-operator-6ddb6c8b94-n75h8\" (UID: \"1342ef67-5f63-47fb-a337-64c693e11228\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n75h8" Oct 30 19:47:07.513252 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.512741 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b1440092-9901-4eff-8a9d-6174b2dfdada-registry-certificates\") pod \"image-registry-6796cf4b57-b9d8k\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:07.513252 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.512802 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8e3ed9d-5156-4698-82c3-df72fa728f94-tmp-dir\") pod \"dns-default-ml5lp\" (UID: \"b8e3ed9d-5156-4698-82c3-df72fa728f94\") " pod="openshift-dns/dns-default-ml5lp" Oct 30 19:47:07.513252 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:07.512182 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:47:07.513252 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.512854 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b8e3ed9d-5156-4698-82c3-df72fa728f94-config-volume\") pod \"dns-default-ml5lp\" (UID: \"b8e3ed9d-5156-4698-82c3-df72fa728f94\") " pod="openshift-dns/dns-default-ml5lp" Oct 30 19:47:07.513252 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:07.512886 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8e3ed9d-5156-4698-82c3-df72fa728f94-metrics-tls podName:b8e3ed9d-5156-4698-82c3-df72fa728f94 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:08.01287055 +0000 UTC m=+30.136081854 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b8e3ed9d-5156-4698-82c3-df72fa728f94-metrics-tls") pod "dns-default-ml5lp" (UID: "b8e3ed9d-5156-4698-82c3-df72fa728f94") : secret "dns-default-metrics-tls" not found Oct 30 19:47:07.513252 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:07.513068 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/41534930-917f-44ec-bccd-86d6c6fc7c70-service-ca-bundle podName:41534930-917f-44ec-bccd-86d6c6fc7c70 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:08.013051182 +0000 UTC m=+30.136262500 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/41534930-917f-44ec-bccd-86d6c6fc7c70-service-ca-bundle") pod "router-default-79458f7dbc-vbb28" (UID: "41534930-917f-44ec-bccd-86d6c6fc7c70") : configmap references non-existent config key: service-ca.crt Oct 30 19:47:07.513543 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.513284 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b1440092-9901-4eff-8a9d-6174b2dfdada-registry-certificates\") pod \"image-registry-6796cf4b57-b9d8k\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:07.513543 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:07.513361 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 19:47:07.513543 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:07.513409 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20f593c8-594e-4a64-9c5e-881311b269a3-cluster-monitoring-operator-tls podName:20f593c8-594e-4a64-9c5e-881311b269a3 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:08.013393822 +0000 UTC m=+30.136605133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/20f593c8-594e-4a64-9c5e-881311b269a3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-qdv4s" (UID: "20f593c8-594e-4a64-9c5e-881311b269a3") : secret "cluster-monitoring-operator-tls" not found Oct 30 19:47:07.513543 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.513515 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1342ef67-5f63-47fb-a337-64c693e11228-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-n75h8\" (UID: \"1342ef67-5f63-47fb-a337-64c693e11228\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n75h8" Oct 30 19:47:07.513765 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.513627 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b1440092-9901-4eff-8a9d-6174b2dfdada-ca-trust-extracted\") pod \"image-registry-6796cf4b57-b9d8k\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:07.513765 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:07.513366 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 19:47:07.513765 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:07.513717 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1342ef67-5f63-47fb-a337-64c693e11228-samples-operator-tls podName:1342ef67-5f63-47fb-a337-64c693e11228 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:08.013703299 +0000 UTC m=+30.136914597 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1342ef67-5f63-47fb-a337-64c693e11228-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-n75h8" (UID: "1342ef67-5f63-47fb-a337-64c693e11228") : secret "samples-operator-tls" not found Oct 30 19:47:07.514247 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:07.513975 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:47:07.514247 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:07.513996 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6796cf4b57-b9d8k: secret "image-registry-tls" not found Oct 30 19:47:07.514247 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:07.514037 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1440092-9901-4eff-8a9d-6174b2dfdada-registry-tls podName:b1440092-9901-4eff-8a9d-6174b2dfdada nodeName:}" failed. No retries permitted until 2026-10-30 19:47:08.014023797 +0000 UTC m=+30.137235102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b1440092-9901-4eff-8a9d-6174b2dfdada-registry-tls") pod "image-registry-6796cf4b57-b9d8k" (UID: "b1440092-9901-4eff-8a9d-6174b2dfdada") : secret "image-registry-tls" not found Oct 30 19:47:07.514247 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.514177 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/20f593c8-594e-4a64-9c5e-881311b269a3-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-qdv4s\" (UID: \"20f593c8-594e-4a64-9c5e-881311b269a3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qdv4s" Oct 30 19:47:07.514679 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.514657 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b1440092-9901-4eff-8a9d-6174b2dfdada-trusted-ca\") pod \"image-registry-6796cf4b57-b9d8k\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:07.514763 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.514729 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/41534930-917f-44ec-bccd-86d6c6fc7c70-default-certificate\") pod \"router-default-79458f7dbc-vbb28\" (UID: \"41534930-917f-44ec-bccd-86d6c6fc7c70\") " pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:07.514930 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.514907 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/41534930-917f-44ec-bccd-86d6c6fc7c70-stats-auth\") pod \"router-default-79458f7dbc-vbb28\" (UID: \"41534930-917f-44ec-bccd-86d6c6fc7c70\") " pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:07.515359 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.515338 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5ad8f4b0-50ba-4191-8df4-25930717acb0-cert\") pod \"ingress-canary-lk574\" (UID: \"5ad8f4b0-50ba-4191-8df4-25930717acb0\") " pod="openshift-ingress-canary/ingress-canary-lk574" Oct 30 19:47:07.516026 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.516011 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b1440092-9901-4eff-8a9d-6174b2dfdada-image-registry-private-configuration\") pod \"image-registry-6796cf4b57-b9d8k\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:07.516073 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.516064 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b1440092-9901-4eff-8a9d-6174b2dfdada-installation-pull-secrets\") pod \"image-registry-6796cf4b57-b9d8k\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:07.530987 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.530969 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-c7n9t" Oct 30 19:47:07.538095 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.538076 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-r7djq" Oct 30 19:47:07.539981 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.539928 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rwww\" (UniqueName: \"kubernetes.io/projected/20f593c8-594e-4a64-9c5e-881311b269a3-kube-api-access-6rwww\") pod \"cluster-monitoring-operator-884576c57-qdv4s\" (UID: \"20f593c8-594e-4a64-9c5e-881311b269a3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qdv4s" Oct 30 19:47:07.540163 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.540108 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x59q2\" (UniqueName: \"kubernetes.io/projected/41534930-917f-44ec-bccd-86d6c6fc7c70-kube-api-access-x59q2\") pod \"router-default-79458f7dbc-vbb28\" (UID: \"41534930-917f-44ec-bccd-86d6c6fc7c70\") " pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:07.540393 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.540363 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtksm\" (UniqueName: \"kubernetes.io/projected/b8e3ed9d-5156-4698-82c3-df72fa728f94-kube-api-access-xtksm\") pod \"dns-default-ml5lp\" (UID: \"b8e3ed9d-5156-4698-82c3-df72fa728f94\") " pod="openshift-dns/dns-default-ml5lp" Oct 30 19:47:07.540556 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.540539 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv4p7\" (UniqueName: \"kubernetes.io/projected/9b6dd02b-d4bd-4959-9a06-8941d6baf1a5-kube-api-access-sv4p7\") pod \"volume-data-source-validator-58c9f5f46d-xlgtn\" (UID: \"9b6dd02b-d4bd-4959-9a06-8941d6baf1a5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xlgtn" Oct 30 19:47:07.540674 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.540654 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxq6z\" (UniqueName: \"kubernetes.io/projected/5ad8f4b0-50ba-4191-8df4-25930717acb0-kube-api-access-lxq6z\") pod \"ingress-canary-lk574\" (UID: \"5ad8f4b0-50ba-4191-8df4-25930717acb0\") " pod="openshift-ingress-canary/ingress-canary-lk574" Oct 30 19:47:07.541336 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.541304 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xtvg\" (UniqueName: \"kubernetes.io/projected/b1440092-9901-4eff-8a9d-6174b2dfdada-kube-api-access-8xtvg\") pod \"image-registry-6796cf4b57-b9d8k\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:07.541404 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.541376 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b1440092-9901-4eff-8a9d-6174b2dfdada-bound-sa-token\") pod \"image-registry-6796cf4b57-b9d8k\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:07.542450 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.542433 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpkkb\" (UniqueName: \"kubernetes.io/projected/cba5fe49-b5b5-413e-9085-e35762430fa9-kube-api-access-hpkkb\") pod \"network-check-source-6f78d59f99-k8mq4\" (UID: \"cba5fe49-b5b5-413e-9085-e35762430fa9\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-k8mq4" Oct 30 19:47:07.544764 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.544746 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj8ws\" (UniqueName: \"kubernetes.io/projected/1342ef67-5f63-47fb-a337-64c693e11228-kube-api-access-vj8ws\") pod \"cluster-samples-operator-6ddb6c8b94-n75h8\" (UID: \"1342ef67-5f63-47fb-a337-64c693e11228\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n75h8" Oct 30 19:47:07.560525 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.560506 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6wqfc" Oct 30 19:47:07.562386 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.562348 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-kflff" Oct 30 19:47:07.570812 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.570791 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-bnp7s" Oct 30 19:47:07.573977 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.573956 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-qdj4h" Oct 30 19:47:07.586926 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.586906 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95bqt" Oct 30 19:47:07.633486 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.631309 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-k8mq4" Oct 30 19:47:07.636500 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.636473 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xlgtn" Oct 30 19:47:07.653225 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.653170 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lk574" Oct 30 19:47:07.869129 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.869105 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-r7djq"] Oct 30 19:47:07.870933 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:47:07.870906 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2962e3da_8439_4316_9c3a_1679842609b7.slice/crio-b7d40f50df1defef862a02a01308a7daf4efac3422a86e9e68e684f5ca5b89db WatchSource:0}: Error finding container b7d40f50df1defef862a02a01308a7daf4efac3422a86e9e68e684f5ca5b89db: Status 404 returned error can't find the container with id b7d40f50df1defef862a02a01308a7daf4efac3422a86e9e68e684f5ca5b89db Oct 30 19:47:07.882093 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.882064 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-c7n9t"] Oct 30 19:47:07.882547 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:47:07.882527 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod908c2874_510c_43c4_a07c_c1771b924e8d.slice/crio-cce1b88df062b6b5cdbaeb19917460725ff99b2b7148c9944d05b9717d0c329a WatchSource:0}: Error finding container cce1b88df062b6b5cdbaeb19917460725ff99b2b7148c9944d05b9717d0c329a: Status 404 returned error can't find the container with id cce1b88df062b6b5cdbaeb19917460725ff99b2b7148c9944d05b9717d0c329a Oct 30 19:47:07.920267 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.920002 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-6wqfc"] Oct 30 19:47:07.929256 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.928756 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/afb6fa1c-9722-42c4-a914-4941bcbcaaad-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-tv4xh\" (UID: \"afb6fa1c-9722-42c4-a914-4941bcbcaaad\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-tv4xh" Oct 30 19:47:07.929256 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.928802 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-registry-tls\") pod \"image-registry-5bc64fc876-25g7d\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:07.929256 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:07.929076 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:47:07.929256 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:07.929091 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bc64fc876-25g7d: secret "image-registry-tls" not found Oct 30 19:47:07.929256 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:07.929137 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-registry-tls podName:9a8ef63e-bcc4-4b71-b345-3fcbc48f4396 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:08.929119521 +0000 UTC m=+31.052330816 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-registry-tls") pod "image-registry-5bc64fc876-25g7d" (UID: "9a8ef63e-bcc4-4b71-b345-3fcbc48f4396") : secret "image-registry-tls" not found Oct 30 19:47:07.929256 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:07.929202 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 19:47:07.929256 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:07.929234 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afb6fa1c-9722-42c4-a914-4941bcbcaaad-networking-console-plugin-cert podName:afb6fa1c-9722-42c4-a914-4941bcbcaaad nodeName:}" failed. No retries permitted until 2026-10-30 19:47:08.929224835 +0000 UTC m=+31.052436132 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/afb6fa1c-9722-42c4-a914-4941bcbcaaad-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-tv4xh" (UID: "afb6fa1c-9722-42c4-a914-4941bcbcaaad") : secret "networking-console-plugin-cert" not found Oct 30 19:47:07.970959 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.970939 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kflff"] Oct 30 19:47:07.979379 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.977421 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95bqt"] Oct 30 19:47:07.980606 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:47:07.980576 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda421de2b_dd41_4862_9849_0a4f0256af1b.slice/crio-26404c8e513a2d41ca262479da0df364db0e615a446757d4c5bae02dee230aad WatchSource:0}: Error finding container 26404c8e513a2d41ca262479da0df364db0e615a446757d4c5bae02dee230aad: Status 404 returned error can't find the container with id 26404c8e513a2d41ca262479da0df364db0e615a446757d4c5bae02dee230aad Oct 30 19:47:07.991227 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:07.991206 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-bnp7s"] Oct 30 19:47:07.992673 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:47:07.992650 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd78606e4_ccb9_421a_8ae5_8fbecbcd3831.slice/crio-8e1a9d3715c1d30e491ef88d0e2eb3879930812c250d7702632943bdc0298a4c WatchSource:0}: Error finding container 8e1a9d3715c1d30e491ef88d0e2eb3879930812c250d7702632943bdc0298a4c: Status 404 returned error can't find the container with id 8e1a9d3715c1d30e491ef88d0e2eb3879930812c250d7702632943bdc0298a4c Oct 30 19:47:08.024048 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:08.023953 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-k8mq4"] Oct 30 19:47:08.027507 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:08.027281 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-qdj4h"] Oct 30 19:47:08.030025 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:08.029998 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1440092-9901-4eff-8a9d-6174b2dfdada-registry-tls\") pod \"image-registry-6796cf4b57-b9d8k\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:08.030113 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:08.030063 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41534930-917f-44ec-bccd-86d6c6fc7c70-service-ca-bundle\") pod \"router-default-79458f7dbc-vbb28\" (UID: \"41534930-917f-44ec-bccd-86d6c6fc7c70\") " pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:08.030113 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:08.030106 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/20f593c8-594e-4a64-9c5e-881311b269a3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qdv4s\" (UID: \"20f593c8-594e-4a64-9c5e-881311b269a3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qdv4s" Oct 30 19:47:08.030211 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:08.030149 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1342ef67-5f63-47fb-a337-64c693e11228-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n75h8\" (UID: \"1342ef67-5f63-47fb-a337-64c693e11228\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n75h8" Oct 30 19:47:08.030247 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:08.030234 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8e3ed9d-5156-4698-82c3-df72fa728f94-metrics-tls\") pod \"dns-default-ml5lp\" (UID: \"b8e3ed9d-5156-4698-82c3-df72fa728f94\") " pod="openshift-dns/dns-default-ml5lp" Oct 30 19:47:08.030289 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:08.030259 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41534930-917f-44ec-bccd-86d6c6fc7c70-metrics-certs\") pod \"router-default-79458f7dbc-vbb28\" (UID: \"41534930-917f-44ec-bccd-86d6c6fc7c70\") " pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:08.030401 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:08.030386 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 19:47:08.030446 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:08.030442 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41534930-917f-44ec-bccd-86d6c6fc7c70-metrics-certs podName:41534930-917f-44ec-bccd-86d6c6fc7c70 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:09.030424449 +0000 UTC m=+31.153635745 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/41534930-917f-44ec-bccd-86d6c6fc7c70-metrics-certs") pod "router-default-79458f7dbc-vbb28" (UID: "41534930-917f-44ec-bccd-86d6c6fc7c70") : secret "router-metrics-certs-default" not found Oct 30 19:47:08.030520 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:08.030509 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:47:08.030555 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:08.030523 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6796cf4b57-b9d8k: secret "image-registry-tls" not found Oct 30 19:47:08.030585 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:08.030556 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1440092-9901-4eff-8a9d-6174b2dfdada-registry-tls podName:b1440092-9901-4eff-8a9d-6174b2dfdada nodeName:}" failed. No retries permitted until 2026-10-30 19:47:09.030544681 +0000 UTC m=+31.153755980 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b1440092-9901-4eff-8a9d-6174b2dfdada-registry-tls") pod "image-registry-6796cf4b57-b9d8k" (UID: "b1440092-9901-4eff-8a9d-6174b2dfdada") : secret "image-registry-tls" not found Oct 30 19:47:08.030652 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:08.030636 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/41534930-917f-44ec-bccd-86d6c6fc7c70-service-ca-bundle podName:41534930-917f-44ec-bccd-86d6c6fc7c70 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:09.030626306 +0000 UTC m=+31.153837603 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/41534930-917f-44ec-bccd-86d6c6fc7c70-service-ca-bundle") pod "router-default-79458f7dbc-vbb28" (UID: "41534930-917f-44ec-bccd-86d6c6fc7c70") : configmap references non-existent config key: service-ca.crt Oct 30 19:47:08.030716 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:08.030692 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 19:47:08.030764 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:08.030723 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20f593c8-594e-4a64-9c5e-881311b269a3-cluster-monitoring-operator-tls podName:20f593c8-594e-4a64-9c5e-881311b269a3 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:09.030713075 +0000 UTC m=+31.153924385 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/20f593c8-594e-4a64-9c5e-881311b269a3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-qdv4s" (UID: "20f593c8-594e-4a64-9c5e-881311b269a3") : secret "cluster-monitoring-operator-tls" not found Oct 30 19:47:08.030823 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:08.030773 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 19:47:08.030823 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:08.030802 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1342ef67-5f63-47fb-a337-64c693e11228-samples-operator-tls podName:1342ef67-5f63-47fb-a337-64c693e11228 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:09.030793551 +0000 UTC m=+31.154004847 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1342ef67-5f63-47fb-a337-64c693e11228-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-n75h8" (UID: "1342ef67-5f63-47fb-a337-64c693e11228") : secret "samples-operator-tls" not found Oct 30 19:47:08.030915 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:08.030848 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:47:08.030915 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:08.030873 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8e3ed9d-5156-4698-82c3-df72fa728f94-metrics-tls podName:b8e3ed9d-5156-4698-82c3-df72fa728f94 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:09.03086481 +0000 UTC m=+31.154076108 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b8e3ed9d-5156-4698-82c3-df72fa728f94-metrics-tls") pod "dns-default-ml5lp" (UID: "b8e3ed9d-5156-4698-82c3-df72fa728f94") : secret "dns-default-metrics-tls" not found Oct 30 19:47:08.058060 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:08.058030 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xlgtn"] Oct 30 19:47:08.063162 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:08.062280 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lk574"] Oct 30 19:47:08.064365 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:47:08.064339 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ad8f4b0_50ba_4191_8df4_25930717acb0.slice/crio-66d6cb40d975bbe02e71cdfd2ac7eb157ed03b647be72ff5cb25853e5cff69d4 WatchSource:0}: Error finding container 66d6cb40d975bbe02e71cdfd2ac7eb157ed03b647be72ff5cb25853e5cff69d4: Status 404 returned error can't find the container with id 66d6cb40d975bbe02e71cdfd2ac7eb157ed03b647be72ff5cb25853e5cff69d4 Oct 30 19:47:08.347944 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:08.347915 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvw9t" Oct 30 19:47:08.348351 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:08.347990 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hz6jn" Oct 30 19:47:08.348351 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:08.348098 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7f4jl" Oct 30 19:47:08.366502 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:08.366474 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 19:47:08.366636 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:08.366602 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-mbspd\"" Oct 30 19:47:08.366700 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:08.366680 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 19:47:08.366801 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:08.366789 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-wnngx\"" Oct 30 19:47:08.366899 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:08.366851 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2gzs4\"" Oct 30 19:47:08.610568 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:08.610417 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-k8mq4" event={"ID":"cba5fe49-b5b5-413e-9085-e35762430fa9","Type":"ContainerStarted","Data":"8af5c415310b263652215a1df5401ccec65bddaf7269f2c52eceec4a3928d171"} Oct 30 19:47:08.612118 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:08.612086 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xlgtn" event={"ID":"9b6dd02b-d4bd-4959-9a06-8941d6baf1a5","Type":"ContainerStarted","Data":"b7584d9dd4d573c6b3ee6d13272564a38c277e2ddd0433616835bdc34de8b7d5"} Oct 30 19:47:08.613267 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:08.613243 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-qdj4h" event={"ID":"da850825-59ab-480f-9047-afd7d2bdc19f","Type":"ContainerStarted","Data":"81aec110cec87d8e18ee9df4b80cca28dbd9db6491128e2ab0102eb2a60f0b7b"} Oct 30 19:47:08.615935 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:08.615914 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-c7n9t" event={"ID":"908c2874-510c-43c4-a07c-c1771b924e8d","Type":"ContainerStarted","Data":"3c7995f1ee5c3ad0b202c9dfd64147c0dacf2babdc478d770feea483de76a095"} Oct 30 19:47:08.616024 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:08.615986 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-c7n9t" event={"ID":"908c2874-510c-43c4-a07c-c1771b924e8d","Type":"ContainerStarted","Data":"cce1b88df062b6b5cdbaeb19917460725ff99b2b7148c9944d05b9717d0c329a"} Oct 30 19:47:08.619053 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:08.619011 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lk574" event={"ID":"5ad8f4b0-50ba-4191-8df4-25930717acb0","Type":"ContainerStarted","Data":"66d6cb40d975bbe02e71cdfd2ac7eb157ed03b647be72ff5cb25853e5cff69d4"} Oct 30 19:47:08.620907 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:08.620850 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95bqt" event={"ID":"a421de2b-dd41-4862-9849-0a4f0256af1b","Type":"ContainerStarted","Data":"26404c8e513a2d41ca262479da0df364db0e615a446757d4c5bae02dee230aad"} Oct 30 19:47:08.623384 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:08.623362 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-kflff" event={"ID":"aab2acf3-4af5-4d2e-8336-05b01a2dca1e","Type":"ContainerStarted","Data":"472b288385fa56ef772da63e5187433b93894fcd1203444dfb83bf4f7e3015e0"} Oct 30 19:47:08.623457 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:08.623402 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-kflff" event={"ID":"aab2acf3-4af5-4d2e-8336-05b01a2dca1e","Type":"ContainerStarted","Data":"4aac46774b6ffb3edab20dbd2bd19deea306cdd7943d68fa2691fca13aad8517"} Oct 30 19:47:08.627354 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:08.627293 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-bnp7s" event={"ID":"d78606e4-ccb9-421a-8ae5-8fbecbcd3831","Type":"ContainerStarted","Data":"8e1a9d3715c1d30e491ef88d0e2eb3879930812c250d7702632943bdc0298a4c"} Oct 30 19:47:08.628788 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:08.628747 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6wqfc" event={"ID":"74b7c739-483f-4756-b97b-e65d37387b17","Type":"ContainerStarted","Data":"eefa490e00b5d52d55a6bf699644887c2b2adf74b421f09f276267016e42ecee"} Oct 30 19:47:08.631078 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:08.630633 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-r7djq" event={"ID":"2962e3da-8439-4316-9c3a-1679842609b7","Type":"ContainerStarted","Data":"c8191530f6e2288c924c6536afa48b217e4a3b54fc81e12dbe7f8779f7580103"} Oct 30 19:47:08.631078 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:08.630668 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-r7djq" event={"ID":"2962e3da-8439-4316-9c3a-1679842609b7","Type":"ContainerStarted","Data":"b7d40f50df1defef862a02a01308a7daf4efac3422a86e9e68e684f5ca5b89db"} Oct 30 19:47:08.665114 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:08.665061 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-c7n9t" podStartSLOduration=310.665045967 podStartE2EDuration="5m10.665045967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:08.663394121 +0000 UTC m=+30.786605439" watchObservedRunningTime="2026-10-30 19:47:08.665045967 +0000 UTC m=+30.788257286" Oct 30 19:47:08.710030 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:08.709977 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-kflff" podStartSLOduration=310.70996153 podStartE2EDuration="5m10.70996153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:08.709191457 +0000 UTC m=+30.832402777" watchObservedRunningTime="2026-10-30 19:47:08.70996153 +0000 UTC m=+30.833172848" Oct 30 19:47:08.939823 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:08.939021 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/afb6fa1c-9722-42c4-a914-4941bcbcaaad-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-tv4xh\" (UID: \"afb6fa1c-9722-42c4-a914-4941bcbcaaad\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-tv4xh" Oct 30 19:47:08.939823 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:08.939070 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-registry-tls\") pod \"image-registry-5bc64fc876-25g7d\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:08.939823 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:08.939348 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 19:47:08.939823 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:08.939413 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afb6fa1c-9722-42c4-a914-4941bcbcaaad-networking-console-plugin-cert podName:afb6fa1c-9722-42c4-a914-4941bcbcaaad nodeName:}" failed. No retries permitted until 2026-10-30 19:47:10.939395522 +0000 UTC m=+33.062606821 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/afb6fa1c-9722-42c4-a914-4941bcbcaaad-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-tv4xh" (UID: "afb6fa1c-9722-42c4-a914-4941bcbcaaad") : secret "networking-console-plugin-cert" not found Oct 30 19:47:08.940309 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:08.940212 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:47:08.940309 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:08.940233 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bc64fc876-25g7d: secret "image-registry-tls" not found Oct 30 19:47:08.940309 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:08.940289 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-registry-tls podName:9a8ef63e-bcc4-4b71-b345-3fcbc48f4396 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:10.940272181 +0000 UTC m=+33.063483498 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-registry-tls") pod "image-registry-5bc64fc876-25g7d" (UID: "9a8ef63e-bcc4-4b71-b345-3fcbc48f4396") : secret "image-registry-tls" not found Oct 30 19:47:09.040439 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:09.040407 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/20f593c8-594e-4a64-9c5e-881311b269a3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qdv4s\" (UID: \"20f593c8-594e-4a64-9c5e-881311b269a3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qdv4s" Oct 30 19:47:09.040634 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:09.040457 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1342ef67-5f63-47fb-a337-64c693e11228-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n75h8\" (UID: \"1342ef67-5f63-47fb-a337-64c693e11228\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n75h8" Oct 30 19:47:09.040634 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:09.040526 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8e3ed9d-5156-4698-82c3-df72fa728f94-metrics-tls\") pod \"dns-default-ml5lp\" (UID: \"b8e3ed9d-5156-4698-82c3-df72fa728f94\") " pod="openshift-dns/dns-default-ml5lp" Oct 30 19:47:09.040634 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:09.040553 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41534930-917f-44ec-bccd-86d6c6fc7c70-metrics-certs\") pod \"router-default-79458f7dbc-vbb28\" (UID: \"41534930-917f-44ec-bccd-86d6c6fc7c70\") " pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:09.040818 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:09.040732 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 19:47:09.040818 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:09.040794 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1342ef67-5f63-47fb-a337-64c693e11228-samples-operator-tls podName:1342ef67-5f63-47fb-a337-64c693e11228 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:11.040775886 +0000 UTC m=+33.163987203 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1342ef67-5f63-47fb-a337-64c693e11228-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-n75h8" (UID: "1342ef67-5f63-47fb-a337-64c693e11228") : secret "samples-operator-tls" not found Oct 30 19:47:09.041246 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:09.041201 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 19:47:09.041344 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:09.041252 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20f593c8-594e-4a64-9c5e-881311b269a3-cluster-monitoring-operator-tls podName:20f593c8-594e-4a64-9c5e-881311b269a3 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:11.041239027 +0000 UTC m=+33.164450326 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/20f593c8-594e-4a64-9c5e-881311b269a3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-qdv4s" (UID: "20f593c8-594e-4a64-9c5e-881311b269a3") : secret "cluster-monitoring-operator-tls" not found Oct 30 19:47:09.041344 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:09.041313 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:47:09.041344 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:09.041346 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8e3ed9d-5156-4698-82c3-df72fa728f94-metrics-tls podName:b8e3ed9d-5156-4698-82c3-df72fa728f94 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:11.041334574 +0000 UTC m=+33.164545881 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b8e3ed9d-5156-4698-82c3-df72fa728f94-metrics-tls") pod "dns-default-ml5lp" (UID: "b8e3ed9d-5156-4698-82c3-df72fa728f94") : secret "dns-default-metrics-tls" not found Oct 30 19:47:09.041510 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:09.041401 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 19:47:09.041510 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:09.041434 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41534930-917f-44ec-bccd-86d6c6fc7c70-metrics-certs podName:41534930-917f-44ec-bccd-86d6c6fc7c70 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:11.041423411 +0000 UTC m=+33.164634719 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/41534930-917f-44ec-bccd-86d6c6fc7c70-metrics-certs") pod "router-default-79458f7dbc-vbb28" (UID: "41534930-917f-44ec-bccd-86d6c6fc7c70") : secret "router-metrics-certs-default" not found Oct 30 19:47:09.041760 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:09.041742 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:47:09.041760 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:09.041761 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6796cf4b57-b9d8k: secret "image-registry-tls" not found Oct 30 19:47:09.041885 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:09.041800 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1440092-9901-4eff-8a9d-6174b2dfdada-registry-tls podName:b1440092-9901-4eff-8a9d-6174b2dfdada nodeName:}" failed. No retries permitted until 2026-10-30 19:47:11.041786938 +0000 UTC m=+33.164998249 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b1440092-9901-4eff-8a9d-6174b2dfdada-registry-tls") pod "image-registry-6796cf4b57-b9d8k" (UID: "b1440092-9901-4eff-8a9d-6174b2dfdada") : secret "image-registry-tls" not found Oct 30 19:47:09.041995 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:09.040623 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1440092-9901-4eff-8a9d-6174b2dfdada-registry-tls\") pod \"image-registry-6796cf4b57-b9d8k\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:09.042050 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:09.042008 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41534930-917f-44ec-bccd-86d6c6fc7c70-service-ca-bundle\") pod \"router-default-79458f7dbc-vbb28\" (UID: \"41534930-917f-44ec-bccd-86d6c6fc7c70\") " pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:09.042140 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:09.042119 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/41534930-917f-44ec-bccd-86d6c6fc7c70-service-ca-bundle podName:41534930-917f-44ec-bccd-86d6c6fc7c70 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:11.042108165 +0000 UTC m=+33.165319463 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/41534930-917f-44ec-bccd-86d6c6fc7c70-service-ca-bundle") pod "router-default-79458f7dbc-vbb28" (UID: "41534930-917f-44ec-bccd-86d6c6fc7c70") : configmap references non-existent config key: service-ca.crt Oct 30 19:47:10.959561 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:10.959473 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/afb6fa1c-9722-42c4-a914-4941bcbcaaad-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-tv4xh\" (UID: \"afb6fa1c-9722-42c4-a914-4941bcbcaaad\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-tv4xh" Oct 30 19:47:10.959561 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:10.959528 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-registry-tls\") pod \"image-registry-5bc64fc876-25g7d\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:10.960056 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:10.959765 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:47:10.960056 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:10.959782 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bc64fc876-25g7d: secret "image-registry-tls" not found Oct 30 19:47:10.960056 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:10.959843 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-registry-tls podName:9a8ef63e-bcc4-4b71-b345-3fcbc48f4396 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:14.959822872 +0000 UTC m=+37.083034184 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-registry-tls") pod "image-registry-5bc64fc876-25g7d" (UID: "9a8ef63e-bcc4-4b71-b345-3fcbc48f4396") : secret "image-registry-tls" not found Oct 30 19:47:10.960283 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:10.960265 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 19:47:10.960344 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:10.960325 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afb6fa1c-9722-42c4-a914-4941bcbcaaad-networking-console-plugin-cert podName:afb6fa1c-9722-42c4-a914-4941bcbcaaad nodeName:}" failed. No retries permitted until 2026-10-30 19:47:14.960310674 +0000 UTC m=+37.083521983 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/afb6fa1c-9722-42c4-a914-4941bcbcaaad-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-tv4xh" (UID: "afb6fa1c-9722-42c4-a914-4941bcbcaaad") : secret "networking-console-plugin-cert" not found Oct 30 19:47:11.060954 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:11.060781 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41534930-917f-44ec-bccd-86d6c6fc7c70-service-ca-bundle\") pod \"router-default-79458f7dbc-vbb28\" (UID: \"41534930-917f-44ec-bccd-86d6c6fc7c70\") " pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:11.061132 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:11.061020 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 19:47:11.061271 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:11.061217 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/41534930-917f-44ec-bccd-86d6c6fc7c70-service-ca-bundle podName:41534930-917f-44ec-bccd-86d6c6fc7c70 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:15.060935645 +0000 UTC m=+37.184146943 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/41534930-917f-44ec-bccd-86d6c6fc7c70-service-ca-bundle") pod "router-default-79458f7dbc-vbb28" (UID: "41534930-917f-44ec-bccd-86d6c6fc7c70") : configmap references non-existent config key: service-ca.crt Oct 30 19:47:11.061271 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:11.061246 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20f593c8-594e-4a64-9c5e-881311b269a3-cluster-monitoring-operator-tls podName:20f593c8-594e-4a64-9c5e-881311b269a3 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:15.061234546 +0000 UTC m=+37.184445845 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/20f593c8-594e-4a64-9c5e-881311b269a3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-qdv4s" (UID: "20f593c8-594e-4a64-9c5e-881311b269a3") : secret "cluster-monitoring-operator-tls" not found Oct 30 19:47:11.061876 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:11.060841 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/20f593c8-594e-4a64-9c5e-881311b269a3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qdv4s\" (UID: \"20f593c8-594e-4a64-9c5e-881311b269a3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qdv4s" Oct 30 19:47:11.061876 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:11.061486 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1342ef67-5f63-47fb-a337-64c693e11228-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n75h8\" (UID: \"1342ef67-5f63-47fb-a337-64c693e11228\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n75h8" Oct 30 19:47:11.061876 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:11.061585 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8e3ed9d-5156-4698-82c3-df72fa728f94-metrics-tls\") pod \"dns-default-ml5lp\" (UID: \"b8e3ed9d-5156-4698-82c3-df72fa728f94\") " pod="openshift-dns/dns-default-ml5lp" Oct 30 19:47:11.061876 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:11.061632 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41534930-917f-44ec-bccd-86d6c6fc7c70-metrics-certs\") pod \"router-default-79458f7dbc-vbb28\" (UID: \"41534930-917f-44ec-bccd-86d6c6fc7c70\") " pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:11.061876 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:11.061646 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 19:47:11.061876 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:11.061700 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1440092-9901-4eff-8a9d-6174b2dfdada-registry-tls\") pod \"image-registry-6796cf4b57-b9d8k\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:11.061876 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:11.061737 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:47:11.061876 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:11.061769 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8e3ed9d-5156-4698-82c3-df72fa728f94-metrics-tls podName:b8e3ed9d-5156-4698-82c3-df72fa728f94 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:15.06175611 +0000 UTC m=+37.184967406 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b8e3ed9d-5156-4698-82c3-df72fa728f94-metrics-tls") pod "dns-default-ml5lp" (UID: "b8e3ed9d-5156-4698-82c3-df72fa728f94") : secret "dns-default-metrics-tls" not found Oct 30 19:47:11.061876 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:11.061809 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:47:11.061876 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:11.061821 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6796cf4b57-b9d8k: secret "image-registry-tls" not found Oct 30 19:47:11.061876 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:11.061822 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 19:47:11.061876 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:11.061855 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41534930-917f-44ec-bccd-86d6c6fc7c70-metrics-certs podName:41534930-917f-44ec-bccd-86d6c6fc7c70 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:15.061845897 +0000 UTC m=+37.185057196 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/41534930-917f-44ec-bccd-86d6c6fc7c70-metrics-certs") pod "router-default-79458f7dbc-vbb28" (UID: "41534930-917f-44ec-bccd-86d6c6fc7c70") : secret "router-metrics-certs-default" not found Oct 30 19:47:11.061876 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:11.061870 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1440092-9901-4eff-8a9d-6174b2dfdada-registry-tls podName:b1440092-9901-4eff-8a9d-6174b2dfdada nodeName:}" failed. No retries permitted until 2026-10-30 19:47:15.061861654 +0000 UTC m=+37.185072949 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b1440092-9901-4eff-8a9d-6174b2dfdada-registry-tls") pod "image-registry-6796cf4b57-b9d8k" (UID: "b1440092-9901-4eff-8a9d-6174b2dfdada") : secret "image-registry-tls" not found Oct 30 19:47:11.061876 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:11.061883 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1342ef67-5f63-47fb-a337-64c693e11228-samples-operator-tls podName:1342ef67-5f63-47fb-a337-64c693e11228 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:15.061875895 +0000 UTC m=+37.185087190 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1342ef67-5f63-47fb-a337-64c693e11228-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-n75h8" (UID: "1342ef67-5f63-47fb-a337-64c693e11228") : secret "samples-operator-tls" not found Oct 30 19:47:11.163443 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:11.162571 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57ddd2a0-f081-4f90-95a6-0f6f8df2f433-metrics-certs\") pod \"network-metrics-daemon-7f4jl\" (UID: \"57ddd2a0-f081-4f90-95a6-0f6f8df2f433\") " pod="openshift-multus/network-metrics-daemon-7f4jl" Oct 30 19:47:11.163443 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:11.162737 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73bb0a1f-597a-4078-8a01-20d0ed24715b-original-pull-secret\") pod \"global-pull-secret-syncer-hz6jn\" (UID: \"73bb0a1f-597a-4078-8a01-20d0ed24715b\") " pod="kube-system/global-pull-secret-syncer-hz6jn" Oct 30 19:47:11.163443 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:11.163350 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 19:47:11.163443 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:11.163412 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57ddd2a0-f081-4f90-95a6-0f6f8df2f433-metrics-certs podName:57ddd2a0-f081-4f90-95a6-0f6f8df2f433 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:43.163394133 +0000 UTC m=+65.286605442 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57ddd2a0-f081-4f90-95a6-0f6f8df2f433-metrics-certs") pod "network-metrics-daemon-7f4jl" (UID: "57ddd2a0-f081-4f90-95a6-0f6f8df2f433") : secret "metrics-daemon-secret" not found Oct 30 19:47:11.176042 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:11.175989 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73bb0a1f-597a-4078-8a01-20d0ed24715b-original-pull-secret\") pod \"global-pull-secret-syncer-hz6jn\" (UID: \"73bb0a1f-597a-4078-8a01-20d0ed24715b\") " pod="kube-system/global-pull-secret-syncer-hz6jn" Oct 30 19:47:11.263624 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:11.263568 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lph46\" (UniqueName: \"kubernetes.io/projected/116a1241-c9a3-4664-8217-60c2b6aa8b76-kube-api-access-lph46\") pod \"network-check-target-kvw9t\" (UID: \"116a1241-c9a3-4664-8217-60c2b6aa8b76\") " pod="openshift-network-diagnostics/network-check-target-kvw9t" Oct 30 19:47:11.273912 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:11.273885 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lph46\" (UniqueName: \"kubernetes.io/projected/116a1241-c9a3-4664-8217-60c2b6aa8b76-kube-api-access-lph46\") pod \"network-check-target-kvw9t\" (UID: \"116a1241-c9a3-4664-8217-60c2b6aa8b76\") " pod="openshift-network-diagnostics/network-check-target-kvw9t" Oct 30 19:47:11.385860 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:11.385816 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvw9t" Oct 30 19:47:11.410287 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:11.410091 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hz6jn" Oct 30 19:47:11.655559 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:11.654624 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xlgtn" event={"ID":"9b6dd02b-d4bd-4959-9a06-8941d6baf1a5","Type":"ContainerStarted","Data":"4e26531b22d74561d30726d1e4c5bbcfc0afe69f538319cc856e376733bac28e"} Oct 30 19:47:11.704428 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:11.704360 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-r7djq" podStartSLOduration=313.704341833 podStartE2EDuration="5m13.704341833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:08.756359693 +0000 UTC m=+30.879571013" watchObservedRunningTime="2026-10-30 19:47:11.704341833 +0000 UTC m=+33.827553149" Oct 30 19:47:11.705184 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:11.705136 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xlgtn" podStartSLOduration=292.265138092 podStartE2EDuration="4m54.705125475s" totalImagesPullingTime="2.439987383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:11.703700656 +0000 UTC m=+33.826911974" watchObservedRunningTime="2026-10-30 19:47:11.705125475 +0000 UTC m=+33.828336792" Oct 30 19:47:11.719730 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:11.719701 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kvw9t"] Oct 30 19:47:11.721332 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:47:11.721281 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod116a1241_c9a3_4664_8217_60c2b6aa8b76.slice/crio-c89f2a9b45855b5d37585e8bf88f3445809637e073d5f3aefb19d6c70c442609 WatchSource:0}: Error finding container c89f2a9b45855b5d37585e8bf88f3445809637e073d5f3aefb19d6c70c442609: Status 404 returned error can't find the container with id c89f2a9b45855b5d37585e8bf88f3445809637e073d5f3aefb19d6c70c442609 Oct 30 19:47:11.744885 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:11.744277 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hz6jn"] Oct 30 19:47:11.751553 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:47:11.751523 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73bb0a1f_597a_4078_8a01_20d0ed24715b.slice/crio-58e66fb0fbdcab2d2c02e4599cb05f12072986ff4279d9b50db1927ab9bf18cb WatchSource:0}: Error finding container 58e66fb0fbdcab2d2c02e4599cb05f12072986ff4279d9b50db1927ab9bf18cb: Status 404 returned error can't find the container with id 58e66fb0fbdcab2d2c02e4599cb05f12072986ff4279d9b50db1927ab9bf18cb Oct 30 19:47:12.659045 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:12.658957 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hz6jn" event={"ID":"73bb0a1f-597a-4078-8a01-20d0ed24715b","Type":"ContainerStarted","Data":"58e66fb0fbdcab2d2c02e4599cb05f12072986ff4279d9b50db1927ab9bf18cb"} Oct 30 19:47:12.660448 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:12.660403 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kvw9t" event={"ID":"116a1241-c9a3-4664-8217-60c2b6aa8b76","Type":"ContainerStarted","Data":"c89f2a9b45855b5d37585e8bf88f3445809637e073d5f3aefb19d6c70c442609"} Oct 30 19:47:12.662230 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:12.662207 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qdj4h_da850825-59ab-480f-9047-afd7d2bdc19f/console-operator/0.log" Oct 30 19:47:12.662333 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:12.662248 2578 generic.go:356] "Generic (PLEG): container finished" podID="da850825-59ab-480f-9047-afd7d2bdc19f" containerID="03f24d4ecd17d01e6e72f676e783ce9de042384a3d3dad84adb81d159a14837a" exitCode=255 Oct 30 19:47:12.662403 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:12.662335 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-qdj4h" event={"ID":"da850825-59ab-480f-9047-afd7d2bdc19f","Type":"ContainerDied","Data":"03f24d4ecd17d01e6e72f676e783ce9de042384a3d3dad84adb81d159a14837a"} Oct 30 19:47:12.662588 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:12.662574 2578 scope.go:122] "RemoveContainer" containerID="03f24d4ecd17d01e6e72f676e783ce9de042384a3d3dad84adb81d159a14837a" Oct 30 19:47:13.668952 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:13.668915 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-bnp7s" event={"ID":"d78606e4-ccb9-421a-8ae5-8fbecbcd3831","Type":"ContainerStarted","Data":"0311b6a67457029147850bb5d461ad63c35e9cdc77c6212ede73fc2499803caf"} Oct 30 19:47:13.671900 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:13.671865 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6wqfc" event={"ID":"74b7c739-483f-4756-b97b-e65d37387b17","Type":"ContainerStarted","Data":"1f2235f3507dd80283f2bd971b8b3cd676d6501ed14e0ed9485dc488cdf82209"} Oct 30 19:47:13.679042 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:13.677251 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qdj4h_da850825-59ab-480f-9047-afd7d2bdc19f/console-operator/0.log" Oct 30 19:47:13.679042 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:13.677330 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-qdj4h" event={"ID":"da850825-59ab-480f-9047-afd7d2bdc19f","Type":"ContainerStarted","Data":"77f9a154bdfbbdcca31431c7016e021fc1250443b58b9925789af279a4b64a0f"} Oct 30 19:47:13.679257 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:13.679234 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-qdj4h" Oct 30 19:47:13.679357 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:13.679333 2578 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-qdj4h container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.11:8443/readyz\": dial tcp 10.132.0.11:8443: connect: connection refused" start-of-body= Oct 30 19:47:13.679413 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:13.679373 2578 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-qdj4h" podUID="da850825-59ab-480f-9047-afd7d2bdc19f" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.11:8443/readyz\": dial tcp 10.132.0.11:8443: connect: connection refused" Oct 30 19:47:13.682912 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:13.682888 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lk574" event={"ID":"5ad8f4b0-50ba-4191-8df4-25930717acb0","Type":"ContainerStarted","Data":"bdb4095b2551dfcffad79d1cdefe98f022a81b319edca7a9bbff9d40505b6982"} Oct 30 19:47:13.685432 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:13.685055 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95bqt" event={"ID":"a421de2b-dd41-4862-9849-0a4f0256af1b","Type":"ContainerStarted","Data":"068a6c75ad9f87aab05aee31cfa9d24e47fdcb138ed92b01e8b68b4f1c0c8359"} Oct 30 19:47:13.719049 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:13.718992 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-bnp7s" podStartSLOduration=266.308503365 podStartE2EDuration="4m31.718975596s" totalImagesPullingTime="5.410472231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:13.718098878 +0000 UTC m=+35.841310195" watchObservedRunningTime="2026-10-30 19:47:13.718975596 +0000 UTC m=+35.842186916" Oct 30 19:47:13.766862 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:13.766028 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95bqt" podStartSLOduration=266.343348218 podStartE2EDuration="4m31.766010862s" totalImagesPullingTime="5.422662644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:13.765711717 +0000 UTC m=+35.888923035" watchObservedRunningTime="2026-10-30 19:47:13.766010862 +0000 UTC m=+35.889222184" Oct 30 19:47:13.815087 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:13.814327 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lk574" podStartSLOduration=1.445074137 podStartE2EDuration="6.814307497s" totalImagesPullingTime="5.36923336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:13.812252983 +0000 UTC m=+35.935464301" watchObservedRunningTime="2026-10-30 19:47:13.814307497 +0000 UTC m=+35.937518815" Oct 30 19:47:13.910734 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:13.910680 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6wqfc" podStartSLOduration=288.428387022 podStartE2EDuration="4m53.910660127s" totalImagesPullingTime="5.482273105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:13.860006759 +0000 UTC m=+35.983218075" watchObservedRunningTime="2026-10-30 19:47:13.910660127 +0000 UTC m=+36.033871446" Oct 30 19:47:13.912507 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:13.911035 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7cd56b8dbc-qdj4h" podStartSLOduration=294.619610084 podStartE2EDuration="4m58.911027611s" totalImagesPullingTime="4.291417527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:13.90998579 +0000 UTC m=+36.033197112" watchObservedRunningTime="2026-10-30 19:47:13.911027611 +0000 UTC m=+36.034238928" Oct 30 19:47:14.679684 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:14.679540 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-b7sz6"] Oct 30 19:47:14.696854 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:14.696824 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qdj4h_da850825-59ab-480f-9047-afd7d2bdc19f/console-operator/1.log" Oct 30 19:47:14.697994 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:14.697666 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qdj4h_da850825-59ab-480f-9047-afd7d2bdc19f/console-operator/0.log" Oct 30 19:47:14.697994 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:14.697708 2578 generic.go:356] "Generic (PLEG): container finished" podID="da850825-59ab-480f-9047-afd7d2bdc19f" containerID="77f9a154bdfbbdcca31431c7016e021fc1250443b58b9925789af279a4b64a0f" exitCode=255 Oct 30 19:47:14.698909 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:14.698876 2578 scope.go:122] "RemoveContainer" containerID="77f9a154bdfbbdcca31431c7016e021fc1250443b58b9925789af279a4b64a0f" Oct 30 19:47:14.702756 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:14.702648 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-k8mq4" event={"ID":"cba5fe49-b5b5-413e-9085-e35762430fa9","Type":"ContainerStarted","Data":"ee1c887c3fd3b8c1e7f7a3bcdbc05e20a4592eb252dcd76b02c2a02529ed3532"} Oct 30 19:47:14.702756 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:14.702728 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-qdj4h" event={"ID":"da850825-59ab-480f-9047-afd7d2bdc19f","Type":"ContainerDied","Data":"77f9a154bdfbbdcca31431c7016e021fc1250443b58b9925789af279a4b64a0f"} Oct 30 19:47:14.703005 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:14.702763 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vdklw"] Oct 30 19:47:14.703005 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:14.702967 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-qdj4h_openshift-console-operator(da850825-59ab-480f-9047-afd7d2bdc19f)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-qdj4h" podUID="da850825-59ab-480f-9047-afd7d2bdc19f" Oct 30 19:47:14.703313 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:14.703296 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b7sz6" Oct 30 19:47:14.703393 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:14.703348 2578 scope.go:122] "RemoveContainer" containerID="03f24d4ecd17d01e6e72f676e783ce9de042384a3d3dad84adb81d159a14837a" Oct 30 19:47:14.718651 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:14.718080 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-b7sz6"] Oct 30 19:47:14.718651 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:14.718201 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vdklw" Oct 30 19:47:14.725850 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:14.725826 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 19:47:14.731067 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:14.731041 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 19:47:14.745360 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:14.745326 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vdklw"] Oct 30 19:47:14.750354 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:14.750295 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-k8mq4" podStartSLOduration=266.994024686 podStartE2EDuration="4m32.750277473s" totalImagesPullingTime="5.756252787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:14.747457 +0000 UTC m=+36.870668321" watchObservedRunningTime="2026-10-30 19:47:14.750277473 +0000 UTC m=+36.873488793" Oct 30 19:47:14.795471 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:14.795439 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-tvr85\"" Oct 30 19:47:14.823588 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:14.823558 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8txdr"] Oct 30 19:47:14.843536 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:14.843508 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8txdr" Oct 30 19:47:14.849080 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:14.849047 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8txdr"] Oct 30 19:47:14.862905 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:14.862861 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 19:47:14.866407 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:14.866386 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 19:47:14.866985 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:14.866925 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q7g4j\"" Oct 30 19:47:14.897354 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:14.896546 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-d7ptc-wwx7j"] Oct 30 19:47:14.900168 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:14.900139 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvcvq\" (UniqueName: \"kubernetes.io/projected/0547f118-5c20-403a-beca-c6a5a671a1ff-kube-api-access-vvcvq\") pod \"migrator-66df5d496b-b7sz6\" (UID: \"0547f118-5c20-403a-beca-c6a5a671a1ff\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b7sz6" Oct 30 19:47:14.900354 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:14.900332 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tng88\" (UniqueName: \"kubernetes.io/projected/7a570505-5193-4564-b776-0f83df60d4e9-kube-api-access-tng88\") pod \"migrator-66df5d496b-vdklw\" (UID: \"7a570505-5193-4564-b776-0f83df60d4e9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vdklw" Oct 30 19:47:14.916129 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:14.916106 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-d7ptc-wwx7j" Oct 30 19:47:14.920122 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:14.920086 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-d7ptc-wwx7j"] Oct 30 19:47:14.935144 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:14.935073 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-9xs5t\"" Oct 30 19:47:15.001581 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.001546 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/afb6fa1c-9722-42c4-a914-4941bcbcaaad-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-tv4xh\" (UID: \"afb6fa1c-9722-42c4-a914-4941bcbcaaad\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-tv4xh" Oct 30 19:47:15.001581 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.001592 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-registry-tls\") pod \"image-registry-5bc64fc876-25g7d\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:15.001797 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.001648 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a160cec7-bc1f-4202-8382-e84318122fd9-crio-socket\") pod \"insights-runtime-extractor-8txdr\" (UID: \"a160cec7-bc1f-4202-8382-e84318122fd9\") " pod="openshift-insights/insights-runtime-extractor-8txdr" Oct 30 19:47:15.001797 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:15.001792 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:47:15.001904 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:15.001809 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bc64fc876-25g7d: secret "image-registry-tls" not found Oct 30 19:47:15.001904 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:15.001844 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 19:47:15.001904 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:15.001859 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-registry-tls podName:9a8ef63e-bcc4-4b71-b345-3fcbc48f4396 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:23.001841056 +0000 UTC m=+45.125052392 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-registry-tls") pod "image-registry-5bc64fc876-25g7d" (UID: "9a8ef63e-bcc4-4b71-b345-3fcbc48f4396") : secret "image-registry-tls" not found Oct 30 19:47:15.002039 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:15.001905 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afb6fa1c-9722-42c4-a914-4941bcbcaaad-networking-console-plugin-cert podName:afb6fa1c-9722-42c4-a914-4941bcbcaaad nodeName:}" failed. No retries permitted until 2026-10-30 19:47:23.001887449 +0000 UTC m=+45.125098759 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/afb6fa1c-9722-42c4-a914-4941bcbcaaad-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-tv4xh" (UID: "afb6fa1c-9722-42c4-a914-4941bcbcaaad") : secret "networking-console-plugin-cert" not found Oct 30 19:47:15.002156 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.002141 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvcvq\" (UniqueName: \"kubernetes.io/projected/0547f118-5c20-403a-beca-c6a5a671a1ff-kube-api-access-vvcvq\") pod \"migrator-66df5d496b-b7sz6\" (UID: \"0547f118-5c20-403a-beca-c6a5a671a1ff\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b7sz6" Oct 30 19:47:15.002229 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.002188 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a160cec7-bc1f-4202-8382-e84318122fd9-data-volume\") pod \"insights-runtime-extractor-8txdr\" (UID: \"a160cec7-bc1f-4202-8382-e84318122fd9\") " pod="openshift-insights/insights-runtime-extractor-8txdr" Oct 30 19:47:15.002229 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.002215 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blstw\" (UniqueName: \"kubernetes.io/projected/a160cec7-bc1f-4202-8382-e84318122fd9-kube-api-access-blstw\") pod \"insights-runtime-extractor-8txdr\" (UID: \"a160cec7-bc1f-4202-8382-e84318122fd9\") " pod="openshift-insights/insights-runtime-extractor-8txdr" Oct 30 19:47:15.002317 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.002246 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a160cec7-bc1f-4202-8382-e84318122fd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8txdr\" (UID: \"a160cec7-bc1f-4202-8382-e84318122fd9\") " pod="openshift-insights/insights-runtime-extractor-8txdr" Oct 30 19:47:15.002317 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.002299 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a160cec7-bc1f-4202-8382-e84318122fd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8txdr\" (UID: \"a160cec7-bc1f-4202-8382-e84318122fd9\") " pod="openshift-insights/insights-runtime-extractor-8txdr" Oct 30 19:47:15.002424 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.002352 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tng88\" (UniqueName: \"kubernetes.io/projected/7a570505-5193-4564-b776-0f83df60d4e9-kube-api-access-tng88\") pod \"migrator-66df5d496b-vdklw\" (UID: \"7a570505-5193-4564-b776-0f83df60d4e9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vdklw" Oct 30 19:47:15.032973 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.032913 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tng88\" (UniqueName: \"kubernetes.io/projected/7a570505-5193-4564-b776-0f83df60d4e9-kube-api-access-tng88\") pod \"migrator-66df5d496b-vdklw\" (UID: \"7a570505-5193-4564-b776-0f83df60d4e9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vdklw" Oct 30 19:47:15.039999 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.039978 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvcvq\" (UniqueName: \"kubernetes.io/projected/0547f118-5c20-403a-beca-c6a5a671a1ff-kube-api-access-vvcvq\") pod \"migrator-66df5d496b-b7sz6\" (UID: \"0547f118-5c20-403a-beca-c6a5a671a1ff\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b7sz6" Oct 30 19:47:15.048591 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.048572 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b7sz6" Oct 30 19:47:15.086383 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.086350 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vdklw" Oct 30 19:47:15.103252 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.103228 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a160cec7-bc1f-4202-8382-e84318122fd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8txdr\" (UID: \"a160cec7-bc1f-4202-8382-e84318122fd9\") " pod="openshift-insights/insights-runtime-extractor-8txdr" Oct 30 19:47:15.103381 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.103295 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8e3ed9d-5156-4698-82c3-df72fa728f94-metrics-tls\") pod \"dns-default-ml5lp\" (UID: \"b8e3ed9d-5156-4698-82c3-df72fa728f94\") " pod="openshift-dns/dns-default-ml5lp" Oct 30 19:47:15.103381 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.103324 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41534930-917f-44ec-bccd-86d6c6fc7c70-metrics-certs\") pod \"router-default-79458f7dbc-vbb28\" (UID: \"41534930-917f-44ec-bccd-86d6c6fc7c70\") " pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:15.103381 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:15.103376 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:47:15.103515 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.103388 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68nwv\" (UniqueName: \"kubernetes.io/projected/005bece8-8d37-4a58-954c-0ae3dab9dadc-kube-api-access-68nwv\") pod \"periodic-gathering-d7ptc-wwx7j\" (UID: \"005bece8-8d37-4a58-954c-0ae3dab9dadc\") " pod="openshift-insights/periodic-gathering-d7ptc-wwx7j" Oct 30 19:47:15.103515 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.103418 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1440092-9901-4eff-8a9d-6174b2dfdada-registry-tls\") pod \"image-registry-6796cf4b57-b9d8k\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:15.103515 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:15.103439 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a160cec7-bc1f-4202-8382-e84318122fd9-insights-runtime-extractor-tls podName:a160cec7-bc1f-4202-8382-e84318122fd9 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:15.603421683 +0000 UTC m=+37.726632992 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a160cec7-bc1f-4202-8382-e84318122fd9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8txdr" (UID: "a160cec7-bc1f-4202-8382-e84318122fd9") : secret "insights-runtime-extractor-tls" not found Oct 30 19:47:15.103515 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.103468 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a160cec7-bc1f-4202-8382-e84318122fd9-crio-socket\") pod \"insights-runtime-extractor-8txdr\" (UID: \"a160cec7-bc1f-4202-8382-e84318122fd9\") " pod="openshift-insights/insights-runtime-extractor-8txdr" Oct 30 19:47:15.103515 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:15.103483 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:47:15.103515 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:15.103495 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6796cf4b57-b9d8k: secret "image-registry-tls" not found Oct 30 19:47:15.103515 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.103507 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/005bece8-8d37-4a58-954c-0ae3dab9dadc-service-ca-bundle\") pod \"periodic-gathering-d7ptc-wwx7j\" (UID: \"005bece8-8d37-4a58-954c-0ae3dab9dadc\") " pod="openshift-insights/periodic-gathering-d7ptc-wwx7j" Oct 30 19:47:15.103835 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:15.103528 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1440092-9901-4eff-8a9d-6174b2dfdada-registry-tls podName:b1440092-9901-4eff-8a9d-6174b2dfdada nodeName:}" failed. No retries permitted until 2026-10-30 19:47:23.103518083 +0000 UTC m=+45.226729395 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b1440092-9901-4eff-8a9d-6174b2dfdada-registry-tls") pod "image-registry-6796cf4b57-b9d8k" (UID: "b1440092-9901-4eff-8a9d-6174b2dfdada") : secret "image-registry-tls" not found Oct 30 19:47:15.103835 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.103591 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a160cec7-bc1f-4202-8382-e84318122fd9-crio-socket\") pod \"insights-runtime-extractor-8txdr\" (UID: \"a160cec7-bc1f-4202-8382-e84318122fd9\") " pod="openshift-insights/insights-runtime-extractor-8txdr" Oct 30 19:47:15.103835 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:15.103689 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:47:15.103835 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:15.103759 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 19:47:15.103835 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:15.103797 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41534930-917f-44ec-bccd-86d6c6fc7c70-metrics-certs podName:41534930-917f-44ec-bccd-86d6c6fc7c70 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:23.103783056 +0000 UTC m=+45.226994352 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/41534930-917f-44ec-bccd-86d6c6fc7c70-metrics-certs") pod "router-default-79458f7dbc-vbb28" (UID: "41534930-917f-44ec-bccd-86d6c6fc7c70") : secret "router-metrics-certs-default" not found Oct 30 19:47:15.104058 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.103839 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41534930-917f-44ec-bccd-86d6c6fc7c70-service-ca-bundle\") pod \"router-default-79458f7dbc-vbb28\" (UID: \"41534930-917f-44ec-bccd-86d6c6fc7c70\") " pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:15.104058 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:15.103859 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8e3ed9d-5156-4698-82c3-df72fa728f94-metrics-tls podName:b8e3ed9d-5156-4698-82c3-df72fa728f94 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:23.103848363 +0000 UTC m=+45.227059665 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b8e3ed9d-5156-4698-82c3-df72fa728f94-metrics-tls") pod "dns-default-ml5lp" (UID: "b8e3ed9d-5156-4698-82c3-df72fa728f94") : secret "dns-default-metrics-tls" not found Oct 30 19:47:15.104058 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.103894 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/005bece8-8d37-4a58-954c-0ae3dab9dadc-archives-path\") pod \"periodic-gathering-d7ptc-wwx7j\" (UID: \"005bece8-8d37-4a58-954c-0ae3dab9dadc\") " pod="openshift-insights/periodic-gathering-d7ptc-wwx7j" Oct 30 19:47:15.104058 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:15.103905 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/41534930-917f-44ec-bccd-86d6c6fc7c70-service-ca-bundle podName:41534930-917f-44ec-bccd-86d6c6fc7c70 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:23.103895051 +0000 UTC m=+45.227106348 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/41534930-917f-44ec-bccd-86d6c6fc7c70-service-ca-bundle") pod "router-default-79458f7dbc-vbb28" (UID: "41534930-917f-44ec-bccd-86d6c6fc7c70") : configmap references non-existent config key: service-ca.crt Oct 30 19:47:15.104058 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.103980 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/20f593c8-594e-4a64-9c5e-881311b269a3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qdv4s\" (UID: \"20f593c8-594e-4a64-9c5e-881311b269a3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qdv4s" Oct 30 19:47:15.104058 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.104027 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1342ef67-5f63-47fb-a337-64c693e11228-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n75h8\" (UID: \"1342ef67-5f63-47fb-a337-64c693e11228\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n75h8" Oct 30 19:47:15.104058 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.104054 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/005bece8-8d37-4a58-954c-0ae3dab9dadc-trusted-ca-bundle\") pod \"periodic-gathering-d7ptc-wwx7j\" (UID: \"005bece8-8d37-4a58-954c-0ae3dab9dadc\") " pod="openshift-insights/periodic-gathering-d7ptc-wwx7j" Oct 30 19:47:15.104362 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.104108 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a160cec7-bc1f-4202-8382-e84318122fd9-data-volume\") pod \"insights-runtime-extractor-8txdr\" (UID: \"a160cec7-bc1f-4202-8382-e84318122fd9\") " pod="openshift-insights/insights-runtime-extractor-8txdr" Oct 30 19:47:15.104362 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.104151 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blstw\" (UniqueName: \"kubernetes.io/projected/a160cec7-bc1f-4202-8382-e84318122fd9-kube-api-access-blstw\") pod \"insights-runtime-extractor-8txdr\" (UID: \"a160cec7-bc1f-4202-8382-e84318122fd9\") " pod="openshift-insights/insights-runtime-extractor-8txdr" Oct 30 19:47:15.104362 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.104182 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a160cec7-bc1f-4202-8382-e84318122fd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8txdr\" (UID: \"a160cec7-bc1f-4202-8382-e84318122fd9\") " pod="openshift-insights/insights-runtime-extractor-8txdr" Oct 30 19:47:15.104362 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:15.104308 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 19:47:15.104539 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:15.104372 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a160cec7-bc1f-4202-8382-e84318122fd9-kube-rbac-proxy-cm podName:a160cec7-bc1f-4202-8382-e84318122fd9 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:15.604356796 +0000 UTC m=+37.727568117 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a160cec7-bc1f-4202-8382-e84318122fd9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8txdr" (UID: "a160cec7-bc1f-4202-8382-e84318122fd9") : configmap "kube-rbac-proxy" not found Oct 30 19:47:15.104539 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:15.104435 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 19:47:15.104539 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:15.104467 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20f593c8-594e-4a64-9c5e-881311b269a3-cluster-monitoring-operator-tls podName:20f593c8-594e-4a64-9c5e-881311b269a3 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:23.104457425 +0000 UTC m=+45.227668733 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/20f593c8-594e-4a64-9c5e-881311b269a3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-qdv4s" (UID: "20f593c8-594e-4a64-9c5e-881311b269a3") : secret "cluster-monitoring-operator-tls" not found Oct 30 19:47:15.104539 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:15.104520 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 19:47:15.104748 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:15.104551 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1342ef67-5f63-47fb-a337-64c693e11228-samples-operator-tls podName:1342ef67-5f63-47fb-a337-64c693e11228 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:23.104541961 +0000 UTC m=+45.227753269 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1342ef67-5f63-47fb-a337-64c693e11228-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-n75h8" (UID: "1342ef67-5f63-47fb-a337-64c693e11228") : secret "samples-operator-tls" not found Oct 30 19:47:15.104904 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.104882 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a160cec7-bc1f-4202-8382-e84318122fd9-data-volume\") pod \"insights-runtime-extractor-8txdr\" (UID: \"a160cec7-bc1f-4202-8382-e84318122fd9\") " pod="openshift-insights/insights-runtime-extractor-8txdr" Oct 30 19:47:15.131834 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.131785 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blstw\" (UniqueName: \"kubernetes.io/projected/a160cec7-bc1f-4202-8382-e84318122fd9-kube-api-access-blstw\") pod \"insights-runtime-extractor-8txdr\" (UID: \"a160cec7-bc1f-4202-8382-e84318122fd9\") " pod="openshift-insights/insights-runtime-extractor-8txdr" Oct 30 19:47:15.206394 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.205043 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68nwv\" (UniqueName: \"kubernetes.io/projected/005bece8-8d37-4a58-954c-0ae3dab9dadc-kube-api-access-68nwv\") pod \"periodic-gathering-d7ptc-wwx7j\" (UID: \"005bece8-8d37-4a58-954c-0ae3dab9dadc\") " pod="openshift-insights/periodic-gathering-d7ptc-wwx7j" Oct 30 19:47:15.206394 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.205086 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/005bece8-8d37-4a58-954c-0ae3dab9dadc-service-ca-bundle\") pod \"periodic-gathering-d7ptc-wwx7j\" (UID: \"005bece8-8d37-4a58-954c-0ae3dab9dadc\") " pod="openshift-insights/periodic-gathering-d7ptc-wwx7j" Oct 30 19:47:15.206394 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.205113 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/005bece8-8d37-4a58-954c-0ae3dab9dadc-archives-path\") pod \"periodic-gathering-d7ptc-wwx7j\" (UID: \"005bece8-8d37-4a58-954c-0ae3dab9dadc\") " pod="openshift-insights/periodic-gathering-d7ptc-wwx7j" Oct 30 19:47:15.206394 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.205155 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/005bece8-8d37-4a58-954c-0ae3dab9dadc-trusted-ca-bundle\") pod \"periodic-gathering-d7ptc-wwx7j\" (UID: \"005bece8-8d37-4a58-954c-0ae3dab9dadc\") " pod="openshift-insights/periodic-gathering-d7ptc-wwx7j" Oct 30 19:47:15.206394 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.206017 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/005bece8-8d37-4a58-954c-0ae3dab9dadc-trusted-ca-bundle\") pod \"periodic-gathering-d7ptc-wwx7j\" (UID: \"005bece8-8d37-4a58-954c-0ae3dab9dadc\") " pod="openshift-insights/periodic-gathering-d7ptc-wwx7j" Oct 30 19:47:15.206394 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.206061 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/005bece8-8d37-4a58-954c-0ae3dab9dadc-service-ca-bundle\") pod \"periodic-gathering-d7ptc-wwx7j\" (UID: \"005bece8-8d37-4a58-954c-0ae3dab9dadc\") " pod="openshift-insights/periodic-gathering-d7ptc-wwx7j" Oct 30 19:47:15.206394 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.206338 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/005bece8-8d37-4a58-954c-0ae3dab9dadc-archives-path\") pod \"periodic-gathering-d7ptc-wwx7j\" (UID: \"005bece8-8d37-4a58-954c-0ae3dab9dadc\") " pod="openshift-insights/periodic-gathering-d7ptc-wwx7j" Oct 30 19:47:15.233137 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.233116 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68nwv\" (UniqueName: \"kubernetes.io/projected/005bece8-8d37-4a58-954c-0ae3dab9dadc-kube-api-access-68nwv\") pod \"periodic-gathering-d7ptc-wwx7j\" (UID: \"005bece8-8d37-4a58-954c-0ae3dab9dadc\") " pod="openshift-insights/periodic-gathering-d7ptc-wwx7j" Oct 30 19:47:15.252754 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.252720 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-d7ptc-wwx7j" Oct 30 19:47:15.377072 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.377043 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-b7sz6"] Oct 30 19:47:15.380566 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:47:15.380538 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0547f118_5c20_403a_beca_c6a5a671a1ff.slice/crio-f5dc44d97b888ba5d594ce06033437408ecdf0f37751254b72ba1d4c0ff40160 WatchSource:0}: Error finding container f5dc44d97b888ba5d594ce06033437408ecdf0f37751254b72ba1d4c0ff40160: Status 404 returned error can't find the container with id f5dc44d97b888ba5d594ce06033437408ecdf0f37751254b72ba1d4c0ff40160 Oct 30 19:47:15.415245 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.415215 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-pfwg8"] Oct 30 19:47:15.429411 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.429390 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-pfwg8" Oct 30 19:47:15.429989 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.429967 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vdklw"] Oct 30 19:47:15.435810 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:47:15.435788 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a570505_5193_4564_b776_0f83df60d4e9.slice/crio-9457f9e3f05603a6042c4a9b4675ec328fa288e629d276292069bcd730da6f3f WatchSource:0}: Error finding container 9457f9e3f05603a6042c4a9b4675ec328fa288e629d276292069bcd730da6f3f: Status 404 returned error can't find the container with id 9457f9e3f05603a6042c4a9b4675ec328fa288e629d276292069bcd730da6f3f Oct 30 19:47:15.439791 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.439775 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-pfwg8"] Oct 30 19:47:15.447927 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.447908 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 19:47:15.448070 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.448057 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-xzffq\"" Oct 30 19:47:15.448856 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.448837 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 19:47:15.448856 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.448845 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 19:47:15.448992 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.448897 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 19:47:15.509313 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.509285 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d71659c8-f08f-4221-a160-56ac78bd8473-signing-cabundle\") pod \"service-ca-5fffc97bdf-pfwg8\" (UID: \"d71659c8-f08f-4221-a160-56ac78bd8473\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pfwg8" Oct 30 19:47:15.509472 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.509427 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d71659c8-f08f-4221-a160-56ac78bd8473-signing-key\") pod \"service-ca-5fffc97bdf-pfwg8\" (UID: \"d71659c8-f08f-4221-a160-56ac78bd8473\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pfwg8" Oct 30 19:47:15.509538 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.509508 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4h6p\" (UniqueName: \"kubernetes.io/projected/d71659c8-f08f-4221-a160-56ac78bd8473-kube-api-access-s4h6p\") pod \"service-ca-5fffc97bdf-pfwg8\" (UID: \"d71659c8-f08f-4221-a160-56ac78bd8473\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pfwg8" Oct 30 19:47:15.564884 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.564859 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-d7ptc-wwx7j"] Oct 30 19:47:15.566358 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:47:15.566330 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod005bece8_8d37_4a58_954c_0ae3dab9dadc.slice/crio-bf54950fee0ea5f71052f3a0ba8d1af23942d0be378225fc4e5d2cfa2fcc1e69 WatchSource:0}: Error finding container bf54950fee0ea5f71052f3a0ba8d1af23942d0be378225fc4e5d2cfa2fcc1e69: Status 404 returned error can't find the container with id bf54950fee0ea5f71052f3a0ba8d1af23942d0be378225fc4e5d2cfa2fcc1e69 Oct 30 19:47:15.610578 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.610545 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d71659c8-f08f-4221-a160-56ac78bd8473-signing-cabundle\") pod \"service-ca-5fffc97bdf-pfwg8\" (UID: \"d71659c8-f08f-4221-a160-56ac78bd8473\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pfwg8" Oct 30 19:47:15.610688 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.610638 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a160cec7-bc1f-4202-8382-e84318122fd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8txdr\" (UID: \"a160cec7-bc1f-4202-8382-e84318122fd9\") " pod="openshift-insights/insights-runtime-extractor-8txdr" Oct 30 19:47:15.610742 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.610694 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d71659c8-f08f-4221-a160-56ac78bd8473-signing-key\") pod \"service-ca-5fffc97bdf-pfwg8\" (UID: \"d71659c8-f08f-4221-a160-56ac78bd8473\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pfwg8" Oct 30 19:47:15.610742 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.610721 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a160cec7-bc1f-4202-8382-e84318122fd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8txdr\" (UID: \"a160cec7-bc1f-4202-8382-e84318122fd9\") " pod="openshift-insights/insights-runtime-extractor-8txdr" Oct 30 19:47:15.610848 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.610770 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4h6p\" (UniqueName: \"kubernetes.io/projected/d71659c8-f08f-4221-a160-56ac78bd8473-kube-api-access-s4h6p\") pod \"service-ca-5fffc97bdf-pfwg8\" (UID: \"d71659c8-f08f-4221-a160-56ac78bd8473\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pfwg8" Oct 30 19:47:15.611703 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.611685 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d71659c8-f08f-4221-a160-56ac78bd8473-signing-cabundle\") pod \"service-ca-5fffc97bdf-pfwg8\" (UID: \"d71659c8-f08f-4221-a160-56ac78bd8473\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pfwg8" Oct 30 19:47:15.611783 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:15.611756 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 19:47:15.611828 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:15.611803 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a160cec7-bc1f-4202-8382-e84318122fd9-kube-rbac-proxy-cm podName:a160cec7-bc1f-4202-8382-e84318122fd9 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:16.611789602 +0000 UTC m=+38.735000910 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a160cec7-bc1f-4202-8382-e84318122fd9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8txdr" (UID: "a160cec7-bc1f-4202-8382-e84318122fd9") : configmap "kube-rbac-proxy" not found Oct 30 19:47:15.612202 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:15.612186 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:47:15.612296 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:15.612238 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a160cec7-bc1f-4202-8382-e84318122fd9-insights-runtime-extractor-tls podName:a160cec7-bc1f-4202-8382-e84318122fd9 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:16.612223875 +0000 UTC m=+38.735435184 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a160cec7-bc1f-4202-8382-e84318122fd9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8txdr" (UID: "a160cec7-bc1f-4202-8382-e84318122fd9") : secret "insights-runtime-extractor-tls" not found Oct 30 19:47:15.614377 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.614356 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d71659c8-f08f-4221-a160-56ac78bd8473-signing-key\") pod \"service-ca-5fffc97bdf-pfwg8\" (UID: \"d71659c8-f08f-4221-a160-56ac78bd8473\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pfwg8" Oct 30 19:47:15.639206 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.639185 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4h6p\" (UniqueName: \"kubernetes.io/projected/d71659c8-f08f-4221-a160-56ac78bd8473-kube-api-access-s4h6p\") pod \"service-ca-5fffc97bdf-pfwg8\" (UID: \"d71659c8-f08f-4221-a160-56ac78bd8473\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pfwg8" Oct 30 19:47:15.707008 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.706972 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-d7ptc-wwx7j" event={"ID":"005bece8-8d37-4a58-954c-0ae3dab9dadc","Type":"ContainerStarted","Data":"4237f64d37febae57523262a2c633409efd9684b43cea4ca819f7a6623c23386"} Oct 30 19:47:15.707407 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.707066 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-d7ptc-wwx7j" event={"ID":"005bece8-8d37-4a58-954c-0ae3dab9dadc","Type":"ContainerStarted","Data":"bf54950fee0ea5f71052f3a0ba8d1af23942d0be378225fc4e5d2cfa2fcc1e69"} Oct 30 19:47:15.708636 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.708595 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qdj4h_da850825-59ab-480f-9047-afd7d2bdc19f/console-operator/1.log" Oct 30 19:47:15.708877 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.708862 2578 scope.go:122] "RemoveContainer" containerID="77f9a154bdfbbdcca31431c7016e021fc1250443b58b9925789af279a4b64a0f" Oct 30 19:47:15.709045 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:15.709026 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-qdj4h_openshift-console-operator(da850825-59ab-480f-9047-afd7d2bdc19f)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-qdj4h" podUID="da850825-59ab-480f-9047-afd7d2bdc19f" Oct 30 19:47:15.710466 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.710446 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b7sz6" event={"ID":"0547f118-5c20-403a-beca-c6a5a671a1ff","Type":"ContainerStarted","Data":"f5dc44d97b888ba5d594ce06033437408ecdf0f37751254b72ba1d4c0ff40160"} Oct 30 19:47:15.711896 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.711878 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kvw9t" event={"ID":"116a1241-c9a3-4664-8217-60c2b6aa8b76","Type":"ContainerStarted","Data":"e86e8bb6f7e395c260247d261ca76918167f47db75f1eeff684571682d83b48e"} Oct 30 19:47:15.712009 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.711998 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kvw9t" Oct 30 19:47:15.713372 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.713354 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vdklw" event={"ID":"7a570505-5193-4564-b776-0f83df60d4e9","Type":"ContainerStarted","Data":"9457f9e3f05603a6042c4a9b4675ec328fa288e629d276292069bcd730da6f3f"} Oct 30 19:47:15.755274 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.755225 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-d7ptc-wwx7j" podStartSLOduration=1.755211047 podStartE2EDuration="1.755211047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:15.753931517 +0000 UTC m=+37.877142844" watchObservedRunningTime="2026-10-30 19:47:15.755211047 +0000 UTC m=+37.878422362" Oct 30 19:47:15.762091 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.762022 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-pfwg8" Oct 30 19:47:15.799755 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:15.799694 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kvw9t" podStartSLOduration=33.905302579 podStartE2EDuration="37.799679293s" totalImagesPullingTime="3.894376714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:15.799495467 +0000 UTC m=+37.922706784" watchObservedRunningTime="2026-10-30 19:47:15.799679293 +0000 UTC m=+37.922890610" Oct 30 19:47:16.077062 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:16.076985 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-pfwg8"] Oct 30 19:47:16.081571 ip-10-0-132-173 kubenswrapper[2578]: W1030 19:47:16.081539 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd71659c8_f08f_4221_a160_56ac78bd8473.slice/crio-f558d523550b45bdea63acef2bcb30b91f6faafc45c1bc83160a62b828ae0b63 WatchSource:0}: Error finding container f558d523550b45bdea63acef2bcb30b91f6faafc45c1bc83160a62b828ae0b63: Status 404 returned error can't find the container with id f558d523550b45bdea63acef2bcb30b91f6faafc45c1bc83160a62b828ae0b63 Oct 30 19:47:16.622127 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:16.622087 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a160cec7-bc1f-4202-8382-e84318122fd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8txdr\" (UID: \"a160cec7-bc1f-4202-8382-e84318122fd9\") " pod="openshift-insights/insights-runtime-extractor-8txdr" Oct 30 19:47:16.622359 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:16.622158 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a160cec7-bc1f-4202-8382-e84318122fd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8txdr\" (UID: \"a160cec7-bc1f-4202-8382-e84318122fd9\") " pod="openshift-insights/insights-runtime-extractor-8txdr" Oct 30 19:47:16.622359 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:16.622235 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 19:47:16.622359 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:16.622313 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a160cec7-bc1f-4202-8382-e84318122fd9-kube-rbac-proxy-cm podName:a160cec7-bc1f-4202-8382-e84318122fd9 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:18.622292005 +0000 UTC m=+40.745503325 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a160cec7-bc1f-4202-8382-e84318122fd9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8txdr" (UID: "a160cec7-bc1f-4202-8382-e84318122fd9") : configmap "kube-rbac-proxy" not found Oct 30 19:47:16.622359 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:16.622323 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:47:16.622560 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:16.622388 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a160cec7-bc1f-4202-8382-e84318122fd9-insights-runtime-extractor-tls podName:a160cec7-bc1f-4202-8382-e84318122fd9 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:18.622369549 +0000 UTC m=+40.745580861 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a160cec7-bc1f-4202-8382-e84318122fd9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8txdr" (UID: "a160cec7-bc1f-4202-8382-e84318122fd9") : secret "insights-runtime-extractor-tls" not found Oct 30 19:47:16.719523 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:16.719480 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-pfwg8" event={"ID":"d71659c8-f08f-4221-a160-56ac78bd8473","Type":"ContainerStarted","Data":"f1cab55a6b6637a57955143af73214c6fa00cc2e9efadfc71338ffef2b101c60"} Oct 30 19:47:16.719917 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:16.719545 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-pfwg8" event={"ID":"d71659c8-f08f-4221-a160-56ac78bd8473","Type":"ContainerStarted","Data":"f558d523550b45bdea63acef2bcb30b91f6faafc45c1bc83160a62b828ae0b63"} Oct 30 19:47:17.575627 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:17.574707 2578 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-qdj4h" Oct 30 19:47:17.575627 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:17.575132 2578 scope.go:122] "RemoveContainer" containerID="77f9a154bdfbbdcca31431c7016e021fc1250443b58b9925789af279a4b64a0f" Oct 30 19:47:17.575627 ip-10-0-132-173 kubenswrapper[2578]: E1030 19:47:17.575335 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-qdj4h_openshift-console-operator(da850825-59ab-480f-9047-afd7d2bdc19f)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-qdj4h" podUID="da850825-59ab-480f-9047-afd7d2bdc19f" Oct 30 19:47:17.725699 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:17.725649 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vdklw" event={"ID":"7a570505-5193-4564-b776-0f83df60d4e9","Type":"ContainerStarted","Data":"d6d807f6008faed657bef70c6f732c6570b9b9f62ab5ab3e953c9dfc8cb79a65"} Oct 30 19:47:17.727580 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:17.727551 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b7sz6" event={"ID":"0547f118-5c20-403a-beca-c6a5a671a1ff","Type":"ContainerStarted","Data":"12834289b01739c8f6aaf87cc6f1ed801e8247290e0fa55ad9447cabf3d8f5ff"} Oct 30 19:47:17.789873 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:17.789818 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-pfwg8" podStartSLOduration=2.789800176 podStartE2EDuration="2.789800176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:16.768480563 +0000 UTC m=+38.891691883" watchObservedRunningTime="2026-10-30 19:47:17.789800176 +0000 UTC m=+39.913011496" Oct 30 19:47:17.790095 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:17.790066 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vdklw" podStartSLOduration=2.114518282 podStartE2EDuration="3.790057673s" totalImagesPullingTime="1.675539391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:17.789555774 +0000 UTC m=+39.912767104" watchObservedRunningTime="2026-10-30 19:47:17.790057673 +0000 UTC m=+39.913269030" Oct 30 19:47:17.847391 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:17.847294 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b7sz6" podStartSLOduration=2.113250747 podStartE2EDuration="3.847277467s" totalImagesPullingTime="1.73402672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:17.846812938 +0000 UTC m=+39.970024254" watchObservedRunningTime="2026-10-30 19:47:17.847277467 +0000 UTC m=+39.970488786" Oct 30 19:47:18.136221 ip-10-0-132-173 kubenswrapper[2578]: I1030 19:47:18.136159 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l9fpp_08b5a0ca-cfe1-4366-a404-5ebbaeb7c149/dns-node-resolver/0.log" Oct 30 19:47:18.527551 ip-10-0-132-173 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 19:47:18.540996 ip-10-0-132-173 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 19:47:18.541156 ip-10-0-132-173 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 19:47:18.541205 ip-10-0-132-173 systemd[1]: kubelet.service: Consumed 3.021s CPU time. Oct 30 19:47:18.550074 ip-10-0-132-173 systemd[1]: Starting Kubernetes Kubelet... Oct 30 19:47:18.644661 ip-10-0-132-173 kubenswrapper[5088]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:47:18.644661 ip-10-0-132-173 kubenswrapper[5088]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 19:47:18.644661 ip-10-0-132-173 kubenswrapper[5088]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:47:18.645136 ip-10-0-132-173 kubenswrapper[5088]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:47:18.647269 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647143 5088 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:47:18.647269 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647222 5088 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:47:18.647269 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647227 5088 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:47:18.647269 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647230 5088 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:47:18.647269 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647232 5088 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:47:18.647269 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647235 5088 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:47:18.647269 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647238 5088 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:47:18.647269 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647241 5088 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:47:18.647269 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647243 5088 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:47:18.647269 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647246 5088 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:47:18.647269 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647249 5088 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:47:18.647269 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647252 5088 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:47:18.647269 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647255 5088 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:47:18.647269 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647258 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:47:18.647269 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647260 5088 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:47:18.647269 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647263 5088 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:47:18.647269 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647265 5088 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:47:18.647269 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647268 5088 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:47:18.647269 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647276 5088 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:47:18.647269 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647279 5088 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:47:18.648030 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647283 5088 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:47:18.648030 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647286 5088 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:47:18.648030 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647289 5088 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:47:18.648030 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647292 5088 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:47:18.648030 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647294 5088 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:47:18.648030 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647297 5088 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:47:18.648030 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647299 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:47:18.648030 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647302 5088 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:47:18.648030 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647304 5088 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:47:18.648030 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647307 5088 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:47:18.648030 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647309 5088 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:47:18.648030 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647313 5088 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:47:18.648030 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647318 5088 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:47:18.648030 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647321 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:47:18.648030 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647324 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:47:18.648030 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647327 5088 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:47:18.648030 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647330 5088 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:47:18.648030 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647333 5088 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:47:18.648472 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647335 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:47:18.648472 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647338 5088 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:47:18.648472 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647340 5088 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:47:18.648472 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647343 5088 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:47:18.648472 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647345 5088 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:47:18.648472 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647348 5088 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:47:18.648472 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647350 5088 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:47:18.648472 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647352 5088 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:47:18.648472 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647355 5088 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:47:18.648472 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647358 5088 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:47:18.648472 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647361 5088 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:47:18.648472 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647364 5088 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:47:18.648472 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647367 5088 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:47:18.648472 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647369 5088 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:47:18.648472 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647372 5088 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:47:18.648472 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647375 5088 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:47:18.648472 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647378 5088 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:47:18.648472 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647380 5088 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:47:18.648472 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647384 5088 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:47:18.648472 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647388 5088 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:47:18.649042 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647391 5088 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:47:18.649042 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647393 5088 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:47:18.649042 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647397 5088 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:47:18.649042 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647400 5088 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:47:18.649042 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647402 5088 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:47:18.649042 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647405 5088 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:47:18.649042 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647407 5088 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:47:18.649042 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647410 5088 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:47:18.649042 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647412 5088 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:47:18.649042 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647415 5088 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:47:18.649042 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647417 5088 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:47:18.649042 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647420 5088 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:47:18.649042 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647422 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:47:18.649042 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647426 5088 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:47:18.649042 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647428 5088 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:47:18.649042 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647431 5088 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:47:18.649042 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647434 5088 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:47:18.649042 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647436 5088 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:47:18.649042 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647439 5088 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:47:18.649042 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647442 5088 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:47:18.649548 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647444 5088 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:47:18.649548 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647447 5088 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:47:18.649548 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647449 5088 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:47:18.649548 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647452 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:47:18.649548 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647455 5088 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:47:18.649548 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647458 5088 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:47:18.649548 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647461 5088 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:47:18.649548 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647463 5088 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:47:18.649548 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647466 5088 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:47:18.649548 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647468 5088 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:47:18.649548 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647471 5088 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:47:18.649548 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647473 5088 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:47:18.649548 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647476 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:47:18.649548 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647478 5088 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:47:18.649548 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647482 5088 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:47:18.649548 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647484 5088 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:47:18.649548 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647487 5088 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:47:18.649548 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647489 5088 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:47:18.649548 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647492 5088 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:47:18.649548 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647494 5088 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:47:18.650061 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647497 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:47:18.650061 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647499 5088 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:47:18.650061 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647502 5088 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:47:18.650061 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647504 5088 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:47:18.650061 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647506 5088 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:47:18.650061 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647509 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:47:18.650061 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647511 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:47:18.650061 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.647514 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:47:18.650061 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648377 5088 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:47:18.650061 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648386 5088 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:47:18.650061 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648389 5088 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:47:18.650061 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648393 5088 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:47:18.650061 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648395 5088 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:47:18.650061 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648398 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:47:18.650061 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648401 5088 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:47:18.650061 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648403 5088 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:47:18.650061 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648406 5088 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:47:18.650061 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648410 5088 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:47:18.650061 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648412 5088 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:47:18.650061 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648415 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:47:18.650688 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648419 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:47:18.650688 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648422 5088 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:47:18.650688 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648428 5088 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:47:18.650688 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648434 5088 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:47:18.650688 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648438 5088 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:47:18.650688 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648443 5088 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:47:18.650688 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648446 5088 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:47:18.650688 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648449 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:47:18.650688 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648452 5088 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:47:18.650688 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648454 5088 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:47:18.650688 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648457 5088 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:47:18.650688 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648460 5088 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:47:18.650688 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648462 5088 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:47:18.650688 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648466 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:47:18.650688 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648468 5088 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:47:18.650688 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648471 5088 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:47:18.650688 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648473 5088 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:47:18.650688 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648476 5088 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:47:18.650688 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648479 5088 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:47:18.651163 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648485 5088 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:47:18.651163 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648488 5088 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:47:18.651163 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648490 5088 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:47:18.651163 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648493 5088 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:47:18.651163 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648495 5088 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:47:18.651163 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648499 5088 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:47:18.651163 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648504 5088 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:47:18.651163 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648508 5088 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:47:18.651163 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648513 5088 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:47:18.651163 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648517 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:47:18.651163 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648520 5088 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:47:18.651163 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648524 5088 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:47:18.651163 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648526 5088 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:47:18.651163 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648529 5088 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:47:18.651163 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648531 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:47:18.651163 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648533 5088 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:47:18.651163 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648536 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:47:18.651163 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648538 5088 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:47:18.651163 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648541 5088 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:47:18.651163 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648543 5088 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:47:18.651699 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648546 5088 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:47:18.651699 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648548 5088 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:47:18.651699 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648551 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:47:18.651699 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648553 5088 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:47:18.651699 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648558 5088 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:47:18.651699 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648561 5088 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:47:18.651699 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648563 5088 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:47:18.651699 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648566 5088 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:47:18.651699 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648568 5088 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:47:18.651699 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648571 5088 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:47:18.651699 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648573 5088 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:47:18.651699 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648577 5088 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:47:18.651699 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648582 5088 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:47:18.651699 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648586 5088 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:47:18.651699 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648590 5088 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:47:18.651699 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648594 5088 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:47:18.651699 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648596 5088 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:47:18.651699 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648599 5088 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:47:18.651699 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648602 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:47:18.651699 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648604 5088 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:47:18.652187 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648607 5088 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:47:18.652187 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648623 5088 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:47:18.652187 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648625 5088 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:47:18.652187 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648628 5088 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:47:18.652187 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648631 5088 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:47:18.652187 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648633 5088 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:47:18.652187 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648636 5088 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:47:18.652187 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648639 5088 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:47:18.652187 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648641 5088 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:47:18.652187 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648644 5088 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:47:18.652187 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648646 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:47:18.652187 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648649 5088 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:47:18.652187 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648651 5088 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:47:18.652187 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648653 5088 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:47:18.652187 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648657 5088 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:47:18.652187 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648661 5088 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:47:18.652187 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648667 5088 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:47:18.652187 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648672 5088 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:47:18.652187 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648675 5088 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:47:18.652668 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648677 5088 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:47:18.652668 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648680 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:47:18.652668 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648682 5088 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:47:18.652668 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648685 5088 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:47:18.652668 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648688 5088 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:47:18.652668 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648690 5088 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:47:18.652668 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648693 5088 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:47:18.652668 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648695 5088 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:47:18.652668 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648698 5088 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:47:18.652668 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648700 5088 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:47:18.652668 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648704 5088 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:47:18.652668 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648706 5088 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:47:18.652668 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648709 5088 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:47:18.652668 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648711 5088 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:47:18.652668 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648714 5088 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:47:18.652668 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.648716 5088 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:47:18.652668 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649289 5088 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:47:18.652668 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649299 5088 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:47:18.652668 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649303 5088 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:47:18.652668 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649306 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:47:18.653207 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649310 5088 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:47:18.653207 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649314 5088 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:47:18.653207 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649317 5088 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:47:18.653207 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649320 5088 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:47:18.653207 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649323 5088 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:47:18.653207 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649325 5088 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:47:18.653207 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649328 5088 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:47:18.653207 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649330 5088 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:47:18.653207 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649333 5088 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:47:18.653207 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649338 5088 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:47:18.653207 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649341 5088 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:47:18.653207 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649343 5088 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:47:18.653207 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649346 5088 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:47:18.653207 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649349 5088 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:47:18.653207 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649351 5088 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:47:18.653207 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649354 5088 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:47:18.653207 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649356 5088 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:47:18.653207 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649359 5088 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:47:18.653207 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649361 5088 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:47:18.653207 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649364 5088 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:47:18.654081 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649367 5088 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:47:18.654081 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649369 5088 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:47:18.654081 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649372 5088 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:47:18.654081 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649374 5088 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:47:18.654081 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649377 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:47:18.654081 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649380 5088 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:47:18.654081 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649382 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:47:18.654081 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649384 5088 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:47:18.654081 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649387 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:47:18.654081 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649391 5088 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:47:18.654081 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649393 5088 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:47:18.654081 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649396 5088 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:47:18.654081 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649398 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:47:18.654081 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649401 5088 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:47:18.654081 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649403 5088 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:47:18.654081 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649406 5088 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:47:18.654081 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649408 5088 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:47:18.654081 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649411 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:47:18.654081 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649413 5088 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:47:18.654907 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649416 5088 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:47:18.654907 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649419 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:47:18.654907 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649423 5088 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:47:18.654907 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649425 5088 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:47:18.654907 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649428 5088 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:47:18.654907 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649430 5088 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:47:18.654907 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649433 5088 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:47:18.654907 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649435 5088 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:47:18.654907 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649438 5088 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:47:18.654907 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649441 5088 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:47:18.654907 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649443 5088 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:47:18.654907 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649446 5088 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:47:18.654907 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649448 5088 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:47:18.654907 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649450 5088 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:47:18.654907 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649453 5088 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:47:18.654907 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649457 5088 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:47:18.654907 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649461 5088 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:47:18.654907 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649464 5088 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:47:18.654907 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649467 5088 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:47:18.654907 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649470 5088 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:47:18.655772 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649473 5088 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:47:18.655772 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649476 5088 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:47:18.655772 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649478 5088 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:47:18.655772 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649482 5088 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:47:18.655772 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649484 5088 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:47:18.655772 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649487 5088 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:47:18.655772 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649489 5088 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:47:18.655772 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649491 5088 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:47:18.655772 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649494 5088 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:47:18.655772 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649497 5088 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:47:18.655772 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649499 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:47:18.655772 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649502 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:47:18.655772 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649506 5088 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:47:18.655772 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649508 5088 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:47:18.655772 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649512 5088 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:47:18.655772 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649514 5088 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:47:18.655772 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649517 5088 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:47:18.655772 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649519 5088 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:47:18.655772 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649522 5088 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:47:18.655772 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649524 5088 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:47:18.656640 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649527 5088 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:47:18.656640 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649530 5088 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:47:18.656640 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649532 5088 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:47:18.656640 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649535 5088 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:47:18.656640 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649537 5088 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:47:18.656640 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649540 5088 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:47:18.656640 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649542 5088 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:47:18.656640 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649545 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:47:18.656640 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649547 5088 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:47:18.656640 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649549 5088 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:47:18.656640 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649552 5088 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:47:18.656640 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649555 5088 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:47:18.656640 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649557 5088 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:47:18.656640 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649560 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:47:18.656640 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649563 5088 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:47:18.656640 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649565 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:47:18.656640 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649568 5088 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:47:18.656640 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649571 5088 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:47:18.656640 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649574 5088 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:47:18.657475 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649576 5088 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:47:18.657475 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649578 5088 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:47:18.657475 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649582 5088 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:47:18.657475 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.649584 5088 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:47:18.657475 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649690 5088 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 19:47:18.657475 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649700 5088 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 19:47:18.657475 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649706 5088 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 19:47:18.657475 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649712 5088 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 19:47:18.657475 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649716 5088 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 19:47:18.657475 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649719 5088 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 19:47:18.657475 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649724 5088 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 19:47:18.657475 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649728 5088 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 19:47:18.657475 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649731 5088 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 19:47:18.657475 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649734 5088 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 19:47:18.657475 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649737 5088 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 19:47:18.657475 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649741 5088 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 19:47:18.657475 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649744 5088 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 19:47:18.657475 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649747 5088 flags.go:64] FLAG: --cgroup-root="" Oct 30 19:47:18.657475 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649750 5088 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 19:47:18.657475 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649754 5088 flags.go:64] FLAG: --client-ca-file="" Oct 30 19:47:18.657475 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649757 5088 flags.go:64] FLAG: --cloud-provider="external" Oct 30 19:47:18.657475 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649760 5088 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 19:47:18.657475 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649765 5088 flags.go:64] FLAG: --cluster-domain="" Oct 30 19:47:18.658500 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649768 5088 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 19:47:18.658500 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649771 5088 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 19:47:18.658500 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649774 5088 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 19:47:18.658500 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649777 5088 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 19:47:18.658500 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649781 5088 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 19:47:18.658500 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649784 5088 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 19:47:18.658500 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649787 5088 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 19:47:18.658500 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649790 5088 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 19:47:18.658500 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649794 5088 flags.go:64] FLAG: --contention-profiling="false" Oct 30 19:47:18.658500 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649797 5088 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 19:47:18.658500 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649800 5088 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 19:47:18.658500 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649803 5088 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 19:47:18.658500 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649806 5088 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 19:47:18.658500 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649810 5088 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 19:47:18.658500 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649814 5088 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 19:47:18.658500 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649817 5088 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 19:47:18.658500 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649820 5088 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 19:47:18.658500 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649823 5088 flags.go:64] FLAG: --enable-server="true" Oct 30 19:47:18.658500 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649826 5088 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 19:47:18.658500 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649830 5088 flags.go:64] FLAG: --event-burst="100" Oct 30 19:47:18.658500 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649833 5088 flags.go:64] FLAG: --event-qps="50" Oct 30 19:47:18.658500 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649836 5088 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 19:47:18.658500 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649839 5088 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 19:47:18.658500 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649842 5088 flags.go:64] FLAG: --eviction-hard="" Oct 30 19:47:18.659550 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649846 5088 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 19:47:18.659550 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649849 5088 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 19:47:18.659550 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649852 5088 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 19:47:18.659550 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649855 5088 flags.go:64] FLAG: --eviction-soft="" Oct 30 19:47:18.659550 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649858 5088 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 19:47:18.659550 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649861 5088 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 19:47:18.659550 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649864 5088 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 19:47:18.659550 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649867 5088 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 19:47:18.659550 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649870 5088 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 19:47:18.659550 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649873 5088 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 19:47:18.659550 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649876 5088 flags.go:64] FLAG: --feature-gates="" Oct 30 19:47:18.659550 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649879 5088 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 19:47:18.659550 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649882 5088 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 19:47:18.659550 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649885 5088 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 19:47:18.659550 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649888 5088 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 19:47:18.659550 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649891 5088 flags.go:64] FLAG: --healthz-port="10248" Oct 30 19:47:18.659550 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649895 5088 flags.go:64] FLAG: --help="false" Oct 30 19:47:18.659550 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649898 5088 flags.go:64] FLAG: --hostname-override="ip-10-0-132-173.ec2.internal" Oct 30 19:47:18.659550 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649901 5088 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 19:47:18.659550 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649904 5088 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 19:47:18.659550 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649906 5088 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 19:47:18.659550 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649910 5088 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 19:47:18.659550 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649914 5088 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 19:47:18.660529 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649917 5088 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 19:47:18.660529 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649920 5088 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 19:47:18.660529 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649922 5088 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 19:47:18.660529 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649925 5088 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 19:47:18.660529 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649928 5088 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 19:47:18.660529 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649931 5088 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 19:47:18.660529 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649934 5088 flags.go:64] FLAG: --kube-reserved="" Oct 30 19:47:18.660529 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649937 5088 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 19:47:18.660529 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649939 5088 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 19:47:18.660529 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649942 5088 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 19:47:18.660529 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649945 5088 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 19:47:18.660529 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649948 5088 flags.go:64] FLAG: --lock-file="" Oct 30 19:47:18.660529 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649950 5088 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 19:47:18.660529 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649953 5088 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 19:47:18.660529 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649956 5088 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 19:47:18.660529 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649961 5088 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 19:47:18.660529 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649964 5088 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 19:47:18.660529 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649967 5088 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 19:47:18.660529 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649972 5088 flags.go:64] FLAG: --logging-format="text" Oct 30 19:47:18.660529 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649974 5088 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 19:47:18.660529 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649978 5088 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 19:47:18.660529 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649980 5088 flags.go:64] FLAG: --manifest-url="" Oct 30 19:47:18.660529 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649983 5088 flags.go:64] FLAG: --manifest-url-header="" Oct 30 19:47:18.660529 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649988 5088 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 19:47:18.660529 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649991 5088 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 19:47:18.661752 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649994 5088 flags.go:64] FLAG: --max-pods="110" Oct 30 19:47:18.661752 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.649998 5088 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 19:47:18.661752 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650001 5088 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 19:47:18.661752 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650003 5088 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 19:47:18.661752 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650006 5088 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 19:47:18.661752 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650009 5088 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 19:47:18.661752 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650013 5088 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 19:47:18.661752 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650016 5088 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 19:47:18.661752 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650023 5088 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 19:47:18.661752 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650026 5088 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 19:47:18.661752 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650029 5088 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 19:47:18.661752 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650032 5088 flags.go:64] FLAG: --pod-cidr="" Oct 30 19:47:18.661752 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650035 5088 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 19:47:18.661752 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650038 5088 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 19:47:18.661752 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650041 5088 flags.go:64] FLAG: --pods-per-core="0" Oct 30 19:47:18.661752 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650044 5088 flags.go:64] FLAG: --port="10250" Oct 30 19:47:18.661752 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650047 5088 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 19:47:18.661752 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650050 5088 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06712a67f28d34232" Oct 30 19:47:18.661752 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650053 5088 flags.go:64] FLAG: --qos-reserved="" Oct 30 19:47:18.661752 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650056 5088 flags.go:64] FLAG: --read-only-port="10255" Oct 30 19:47:18.661752 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650058 5088 flags.go:64] FLAG: --register-node="true" Oct 30 19:47:18.661752 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650061 5088 flags.go:64] FLAG: --register-with-taints="" Oct 30 19:47:18.661752 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650065 5088 flags.go:64] FLAG: --registry-burst="10" Oct 30 19:47:18.661752 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650068 5088 flags.go:64] FLAG: --registry-qps="5" Oct 30 19:47:18.661752 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650071 5088 flags.go:64] FLAG: --reserved-cpus="" Oct 30 19:47:18.661752 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650077 5088 flags.go:64] FLAG: --reserved-memory="" Oct 30 19:47:18.662881 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650084 5088 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 19:47:18.662881 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650088 5088 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 19:47:18.662881 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650093 5088 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 19:47:18.662881 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650097 5088 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 19:47:18.662881 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650102 5088 flags.go:64] FLAG: --runonce="false" Oct 30 19:47:18.662881 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650106 5088 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 19:47:18.662881 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650110 5088 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 19:47:18.662881 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650115 5088 flags.go:64] FLAG: --seccomp-default="false" Oct 30 19:47:18.662881 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650123 5088 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 19:47:18.662881 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650128 5088 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 19:47:18.662881 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650133 5088 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 19:47:18.662881 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650137 5088 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 19:47:18.662881 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650143 5088 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 19:47:18.662881 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650147 5088 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 19:47:18.662881 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650152 5088 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 19:47:18.662881 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650156 5088 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 19:47:18.662881 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650161 5088 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 19:47:18.662881 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650165 5088 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 19:47:18.662881 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650169 5088 flags.go:64] FLAG: --system-cgroups="" Oct 30 19:47:18.662881 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650174 5088 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 19:47:18.662881 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650182 5088 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 19:47:18.662881 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650186 5088 flags.go:64] FLAG: --tls-cert-file="" Oct 30 19:47:18.662881 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650190 5088 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 19:47:18.662881 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650196 5088 flags.go:64] FLAG: --tls-min-version="" Oct 30 19:47:18.662881 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650200 5088 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 19:47:18.663789 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650204 5088 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 19:47:18.663789 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650208 5088 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 19:47:18.663789 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650212 5088 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 19:47:18.663789 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650217 5088 flags.go:64] FLAG: --v="2" Oct 30 19:47:18.663789 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650223 5088 flags.go:64] FLAG: --version="false" Oct 30 19:47:18.663789 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650229 5088 flags.go:64] FLAG: --vmodule="" Oct 30 19:47:18.663789 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650237 5088 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 19:47:18.663789 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650242 5088 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 19:47:18.663789 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650413 5088 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:47:18.663789 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650419 5088 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:47:18.663789 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650422 5088 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:47:18.663789 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650425 5088 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:47:18.663789 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650428 5088 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:47:18.663789 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650430 5088 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:47:18.663789 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650434 5088 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:47:18.663789 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650437 5088 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:47:18.663789 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650440 5088 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:47:18.663789 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650443 5088 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:47:18.663789 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650446 5088 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:47:18.663789 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650450 5088 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:47:18.663789 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650453 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:47:18.663789 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650455 5088 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:47:18.664592 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650458 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:47:18.664592 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650461 5088 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:47:18.664592 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650463 5088 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:47:18.664592 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650466 5088 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:47:18.664592 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650468 5088 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:47:18.664592 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650472 5088 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:47:18.664592 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650474 5088 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:47:18.664592 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650477 5088 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:47:18.664592 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650479 5088 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:47:18.664592 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650483 5088 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:47:18.664592 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650487 5088 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:47:18.664592 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650489 5088 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:47:18.664592 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650492 5088 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:47:18.664592 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650495 5088 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:47:18.664592 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650497 5088 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:47:18.664592 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650500 5088 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:47:18.664592 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650504 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:47:18.664592 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650506 5088 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:47:18.664592 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650509 5088 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:47:18.664592 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650512 5088 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:47:18.665377 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650515 5088 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:47:18.665377 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650517 5088 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:47:18.665377 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650520 5088 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:47:18.665377 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650522 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:47:18.665377 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650525 5088 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:47:18.665377 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650528 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:47:18.665377 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650530 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:47:18.665377 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650538 5088 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:47:18.665377 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650541 5088 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:47:18.665377 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650544 5088 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:47:18.665377 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650547 5088 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:47:18.665377 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650550 5088 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:47:18.665377 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650552 5088 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:47:18.665377 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650555 5088 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:47:18.665377 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650557 5088 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:47:18.665377 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650560 5088 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:47:18.665377 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650562 5088 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:47:18.665377 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650565 5088 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:47:18.665377 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650567 5088 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:47:18.665914 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650570 5088 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:47:18.665914 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650572 5088 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:47:18.665914 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650575 5088 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:47:18.665914 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650577 5088 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:47:18.665914 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650580 5088 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:47:18.665914 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650582 5088 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:47:18.665914 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650585 5088 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:47:18.665914 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650587 5088 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:47:18.665914 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650590 5088 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:47:18.665914 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650593 5088 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:47:18.665914 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650596 5088 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:47:18.665914 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650598 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:47:18.665914 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650601 5088 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:47:18.665914 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650603 5088 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:47:18.665914 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650606 5088 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:47:18.665914 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650622 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:47:18.665914 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650625 5088 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:47:18.665914 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650628 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:47:18.665914 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650630 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:47:18.666388 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650633 5088 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:47:18.666388 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650635 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:47:18.666388 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650644 5088 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:47:18.666388 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650647 5088 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:47:18.666388 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650650 5088 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:47:18.666388 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650653 5088 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:47:18.666388 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650655 5088 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:47:18.666388 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650658 5088 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:47:18.666388 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650661 5088 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:47:18.666388 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650663 5088 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:47:18.666388 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650666 5088 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:47:18.666388 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650669 5088 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:47:18.666388 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650671 5088 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:47:18.666388 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650674 5088 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:47:18.666388 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650676 5088 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:47:18.666388 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650679 5088 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:47:18.666388 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650681 5088 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:47:18.666388 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650684 5088 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:47:18.666388 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650688 5088 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:47:18.666870 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650692 5088 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:47:18.666870 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650695 5088 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:47:18.666870 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650698 5088 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:47:18.666870 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650703 5088 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:47:18.666870 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650705 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:47:18.666870 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650708 5088 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:47:18.666870 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650711 5088 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:47:18.666870 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650713 5088 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:47:18.666870 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650716 5088 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:47:18.666870 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650718 5088 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:47:18.666870 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650721 5088 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:47:18.666870 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650723 5088 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:47:18.666870 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650726 5088 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:47:18.666870 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650728 5088 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:47:18.666870 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650731 5088 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:47:18.666870 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.650759 5088 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:47:18.667536 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650808 5088 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:47:18.667536 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650813 5088 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:47:18.667536 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650816 5088 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:47:18.667536 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650818 5088 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:47:18.667536 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650821 5088 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:47:18.667536 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650823 5088 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:47:18.667536 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650826 5088 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:47:18.667536 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650828 5088 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:47:18.667536 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650831 5088 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:47:18.667536 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650834 5088 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:47:18.667536 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650836 5088 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:47:18.667536 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650839 5088 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:47:18.667536 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650841 5088 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:47:18.667536 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650844 5088 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:47:18.667536 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650846 5088 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:47:18.667536 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650850 5088 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:47:18.667536 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650854 5088 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:47:18.667536 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650857 5088 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:47:18.667536 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650860 5088 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:47:18.668186 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650862 5088 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:47:18.668186 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650866 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:47:18.668186 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650869 5088 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:47:18.668186 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650871 5088 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:47:18.668186 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650874 5088 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:47:18.668186 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650876 5088 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:47:18.668186 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650879 5088 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:47:18.668186 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650881 5088 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:47:18.668186 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650884 5088 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:47:18.668186 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650886 5088 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:47:18.668186 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650889 5088 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:47:18.668186 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650892 5088 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:47:18.668186 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650894 5088 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:47:18.668186 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650898 5088 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:47:18.668186 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650906 5088 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:47:18.668186 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650909 5088 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:47:18.668186 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650911 5088 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:47:18.668186 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650914 5088 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:47:18.668186 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650917 5088 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:47:18.668186 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650919 5088 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:47:18.668724 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650922 5088 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:47:18.668724 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650924 5088 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:47:18.668724 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650927 5088 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:47:18.668724 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650929 5088 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:47:18.668724 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650932 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:47:18.668724 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650934 5088 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:47:18.668724 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650937 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:47:18.668724 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650940 5088 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:47:18.668724 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650942 5088 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:47:18.668724 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650945 5088 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:47:18.668724 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650947 5088 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:47:18.668724 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650950 5088 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:47:18.668724 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650953 5088 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:47:18.668724 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650956 5088 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:47:18.668724 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650959 5088 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:47:18.668724 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650963 5088 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:47:18.668724 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650966 5088 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:47:18.668724 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650969 5088 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:47:18.668724 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650972 5088 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:47:18.668724 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650975 5088 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:47:18.669249 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650977 5088 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:47:18.669249 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650980 5088 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:47:18.669249 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650983 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:47:18.669249 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650985 5088 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:47:18.669249 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650988 5088 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:47:18.669249 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650992 5088 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:47:18.669249 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.650994 5088 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:47:18.669249 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.651004 5088 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:47:18.669249 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.651006 5088 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:47:18.669249 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.651009 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:47:18.669249 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.651012 5088 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:47:18.669249 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.651014 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:47:18.669249 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.651017 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:47:18.669249 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.651019 5088 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:47:18.669249 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.651022 5088 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:47:18.669249 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.651024 5088 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:47:18.669249 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.651027 5088 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:47:18.669249 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.651029 5088 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:47:18.669249 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.651032 5088 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:47:18.669716 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.651035 5088 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:47:18.669716 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.651037 5088 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:47:18.669716 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.651039 5088 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:47:18.669716 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.651042 5088 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:47:18.669716 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.651044 5088 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:47:18.669716 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.651047 5088 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:47:18.669716 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.651051 5088 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:47:18.669716 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.651053 5088 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:47:18.669716 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.651056 5088 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:47:18.669716 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.651058 5088 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:47:18.669716 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.651061 5088 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:47:18.669716 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.651063 5088 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:47:18.669716 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.651066 5088 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:47:18.669716 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.651068 5088 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:47:18.669716 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.651071 5088 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:47:18.669716 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.651074 5088 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:47:18.669716 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.651076 5088 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:47:18.669716 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.651079 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:47:18.669716 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.651082 5088 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:47:18.670166 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.651085 5088 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:47:18.670166 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.651087 5088 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:47:18.670166 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.651095 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:47:18.670166 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.651098 5088 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:47:18.670166 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.651100 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:47:18.670166 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.651103 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:47:18.670166 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.651109 5088 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:47:18.670166 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.651112 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:47:18.670166 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.651115 5088 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:47:18.670166 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.656296 5088 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 19:47:18.670166 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.656315 5088 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 19:47:18.670166 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.656341 5088 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 19:47:18.670166 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.656350 5088 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 19:47:18.670166 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656438 5088 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:47:18.670166 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656445 5088 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:47:18.670166 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656451 5088 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:47:18.670166 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656456 5088 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:47:18.670166 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656460 5088 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:47:18.670166 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656466 5088 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:47:18.670675 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656472 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:47:18.670675 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656477 5088 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:47:18.670675 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656482 5088 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:47:18.670675 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656486 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:47:18.670675 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656491 5088 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:47:18.670675 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656495 5088 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:47:18.670675 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656499 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:47:18.670675 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656504 5088 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:47:18.670675 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656508 5088 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:47:18.670675 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656512 5088 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:47:18.670675 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656516 5088 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:47:18.670675 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656521 5088 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:47:18.670675 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656525 5088 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:47:18.670675 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656529 5088 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:47:18.670675 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656533 5088 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:47:18.670675 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656537 5088 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:47:18.670675 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656541 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:47:18.670675 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656546 5088 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:47:18.670675 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656551 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:47:18.670675 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656556 5088 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:47:18.671142 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656560 5088 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:47:18.671142 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656565 5088 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:47:18.671142 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656568 5088 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:47:18.671142 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656580 5088 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:47:18.671142 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656585 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:47:18.671142 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656589 5088 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:47:18.671142 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656594 5088 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:47:18.671142 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656598 5088 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:47:18.671142 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656602 5088 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:47:18.671142 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656606 5088 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:47:18.671142 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656629 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:47:18.671142 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656634 5088 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:47:18.671142 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656637 5088 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:47:18.671142 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656641 5088 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:47:18.671142 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656645 5088 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:47:18.671142 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656649 5088 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:47:18.671142 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656655 5088 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:47:18.671142 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656662 5088 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:47:18.671142 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656668 5088 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:47:18.671586 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656674 5088 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:47:18.671586 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656678 5088 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:47:18.671586 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656683 5088 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:47:18.671586 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656688 5088 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:47:18.671586 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656693 5088 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:47:18.671586 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656698 5088 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:47:18.671586 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656702 5088 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:47:18.671586 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656706 5088 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:47:18.671586 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656710 5088 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:47:18.671586 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656715 5088 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:47:18.671586 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656719 5088 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:47:18.671586 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656723 5088 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:47:18.671586 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656727 5088 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:47:18.671586 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656731 5088 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:47:18.671586 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656736 5088 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:47:18.671586 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656740 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:47:18.671586 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656744 5088 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:47:18.671586 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656756 5088 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:47:18.671586 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656761 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:47:18.671586 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656765 5088 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:47:18.672063 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656770 5088 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:47:18.672063 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656774 5088 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:47:18.672063 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656778 5088 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:47:18.672063 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656782 5088 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:47:18.672063 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656786 5088 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:47:18.672063 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656790 5088 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:47:18.672063 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656794 5088 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:47:18.672063 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656799 5088 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:47:18.672063 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656803 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:47:18.672063 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656807 5088 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:47:18.672063 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656820 5088 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:47:18.672063 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656826 5088 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:47:18.672063 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656831 5088 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:47:18.672063 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656835 5088 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:47:18.672063 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656839 5088 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:47:18.672063 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656844 5088 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:47:18.672063 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656848 5088 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:47:18.672063 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656852 5088 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:47:18.672063 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656856 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:47:18.672687 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656861 5088 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:47:18.672687 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656865 5088 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:47:18.672687 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656869 5088 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:47:18.672687 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656874 5088 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:47:18.672687 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656878 5088 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:47:18.672687 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656882 5088 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:47:18.672687 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656886 5088 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:47:18.672687 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656890 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:47:18.672687 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656895 5088 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:47:18.672687 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656899 5088 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:47:18.672687 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656903 5088 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:47:18.672687 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656915 5088 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:47:18.672687 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656919 5088 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:47:18.672687 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656924 5088 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:47:18.672687 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656928 5088 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:47:18.672687 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656932 5088 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:47:18.672687 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656935 5088 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:47:18.672687 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656940 5088 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:47:18.672687 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656944 5088 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:47:18.672687 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656948 5088 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:47:18.673226 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656952 5088 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:47:18.673226 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.656956 5088 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:47:18.673226 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.656999 5088 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:47:18.673226 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657270 5088 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:47:18.673226 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657278 5088 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:47:18.673226 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657283 5088 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:47:18.673226 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657288 5088 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:47:18.673226 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657293 5088 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:47:18.673226 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657297 5088 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:47:18.673226 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657301 5088 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:47:18.673226 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657306 5088 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:47:18.673226 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657310 5088 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:47:18.673226 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657314 5088 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:47:18.673226 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657318 5088 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:47:18.673226 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657324 5088 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:47:18.673830 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657331 5088 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:47:18.673830 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657336 5088 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:47:18.673830 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657340 5088 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:47:18.673830 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657345 5088 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:47:18.673830 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657349 5088 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:47:18.673830 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657354 5088 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:47:18.673830 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657358 5088 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:47:18.673830 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657362 5088 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:47:18.673830 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657366 5088 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:47:18.673830 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657380 5088 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:47:18.673830 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657384 5088 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:47:18.673830 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657388 5088 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:47:18.673830 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657392 5088 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:47:18.673830 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657396 5088 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:47:18.673830 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657401 5088 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:47:18.673830 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657405 5088 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:47:18.673830 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657409 5088 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:47:18.673830 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657413 5088 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:47:18.673830 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657417 5088 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:47:18.673830 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657424 5088 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:47:18.674454 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657428 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:47:18.674454 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657432 5088 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:47:18.674454 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657436 5088 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:47:18.674454 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657441 5088 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:47:18.674454 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657445 5088 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:47:18.674454 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657452 5088 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:47:18.674454 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657458 5088 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:47:18.674454 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657462 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:47:18.674454 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657467 5088 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:47:18.674454 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657471 5088 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:47:18.674454 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657475 5088 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:47:18.674454 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657479 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:47:18.674454 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657483 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:47:18.674454 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657487 5088 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:47:18.674454 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657490 5088 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:47:18.674454 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657494 5088 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:47:18.674454 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657498 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:47:18.674454 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657503 5088 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:47:18.674454 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657507 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:47:18.675004 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657511 5088 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:47:18.675004 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657516 5088 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:47:18.675004 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657520 5088 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:47:18.675004 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657531 5088 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:47:18.675004 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657536 5088 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:47:18.675004 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657540 5088 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:47:18.675004 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657544 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:47:18.675004 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657548 5088 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:47:18.675004 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657552 5088 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:47:18.675004 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657556 5088 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:47:18.675004 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657561 5088 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:47:18.675004 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657565 5088 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:47:18.675004 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657569 5088 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:47:18.675004 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657573 5088 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:47:18.675004 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657578 5088 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:47:18.675004 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657582 5088 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:47:18.675004 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657586 5088 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:47:18.675004 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657591 5088 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:47:18.675004 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657595 5088 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:47:18.675004 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657599 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:47:18.675490 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657604 5088 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:47:18.675490 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657622 5088 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:47:18.675490 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657627 5088 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:47:18.675490 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657631 5088 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:47:18.675490 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657643 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:47:18.675490 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657647 5088 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:47:18.675490 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657651 5088 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:47:18.675490 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657655 5088 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:47:18.675490 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657659 5088 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:47:18.675490 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657663 5088 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:47:18.675490 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657667 5088 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:47:18.675490 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657671 5088 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:47:18.675490 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657675 5088 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:47:18.675490 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657679 5088 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:47:18.675490 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657683 5088 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:47:18.675490 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657688 5088 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:47:18.675490 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657699 5088 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:47:18.675490 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657704 5088 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:47:18.675490 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657709 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:47:18.675490 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657713 5088 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:47:18.676050 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657717 5088 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:47:18.676050 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657721 5088 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:47:18.676050 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657725 5088 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:47:18.676050 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657729 5088 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:47:18.676050 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657734 5088 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:47:18.676050 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657738 5088 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:47:18.676050 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657742 5088 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:47:18.676050 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657747 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:47:18.676050 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657751 5088 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:47:18.676050 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657755 5088 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:47:18.676050 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657759 5088 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:47:18.676050 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657764 5088 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:47:18.676050 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657768 5088 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:47:18.676050 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657773 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:47:18.676050 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657777 5088 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:47:18.676050 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.657816 5088 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:47:18.676425 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657881 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:47:18.676425 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657888 5088 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:47:18.676425 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657892 5088 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:47:18.676425 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657896 5088 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:47:18.676425 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657901 5088 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:47:18.676425 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657905 5088 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:47:18.676425 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657909 5088 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:47:18.676425 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657913 5088 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:47:18.676425 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657916 5088 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:47:18.676425 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657921 5088 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:47:18.676425 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657926 5088 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:47:18.676425 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657930 5088 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:47:18.676425 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657934 5088 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:47:18.676425 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657946 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:47:18.676425 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657950 5088 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:47:18.676425 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657954 5088 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:47:18.676425 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657958 5088 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:47:18.676425 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657962 5088 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:47:18.676425 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657966 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:47:18.676425 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657970 5088 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:47:18.676956 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657974 5088 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:47:18.676956 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657979 5088 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:47:18.676956 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657983 5088 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:47:18.676956 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657987 5088 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:47:18.676956 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657992 5088 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:47:18.676956 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.657996 5088 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:47:18.676956 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658000 5088 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:47:18.676956 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658004 5088 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:47:18.676956 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658008 5088 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:47:18.676956 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658013 5088 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:47:18.676956 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658017 5088 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:47:18.676956 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658021 5088 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:47:18.676956 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658025 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:47:18.676956 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658029 5088 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:47:18.676956 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658034 5088 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:47:18.676956 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658038 5088 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:47:18.676956 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658042 5088 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:47:18.676956 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658046 5088 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:47:18.676956 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658050 5088 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:47:18.676956 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658054 5088 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:47:18.677425 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658058 5088 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:47:18.677425 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658063 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:47:18.677425 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658067 5088 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:47:18.677425 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658071 5088 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:47:18.677425 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658075 5088 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:47:18.677425 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658079 5088 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:47:18.677425 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658091 5088 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:47:18.677425 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658095 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:47:18.677425 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658100 5088 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:47:18.677425 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658103 5088 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:47:18.677425 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658108 5088 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:47:18.677425 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658112 5088 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:47:18.677425 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658116 5088 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:47:18.677425 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658120 5088 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:47:18.677425 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658124 5088 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:47:18.677425 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658128 5088 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:47:18.677425 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658132 5088 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:47:18.677425 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658136 5088 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:47:18.677425 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658140 5088 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:47:18.677927 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658144 5088 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:47:18.677927 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658152 5088 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:47:18.677927 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658157 5088 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:47:18.677927 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658161 5088 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:47:18.677927 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658165 5088 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:47:18.677927 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658169 5088 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:47:18.677927 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658173 5088 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:47:18.677927 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658177 5088 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:47:18.677927 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658182 5088 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:47:18.677927 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658186 5088 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:47:18.677927 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658190 5088 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:47:18.677927 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658194 5088 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:47:18.677927 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658198 5088 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:47:18.677927 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658202 5088 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:47:18.677927 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658206 5088 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:47:18.677927 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658210 5088 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:47:18.677927 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658217 5088 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:47:18.677927 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658221 5088 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:47:18.677927 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658225 5088 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:47:18.677927 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658229 5088 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:47:18.678399 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658243 5088 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:47:18.678399 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658248 5088 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:47:18.678399 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658252 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:47:18.678399 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658256 5088 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:47:18.678399 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658260 5088 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:47:18.678399 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658264 5088 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:47:18.678399 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658270 5088 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:47:18.678399 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658277 5088 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:47:18.678399 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658283 5088 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:47:18.678399 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658288 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:47:18.678399 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658293 5088 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:47:18.678399 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658298 5088 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:47:18.678399 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658302 5088 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:47:18.678399 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658306 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:47:18.678399 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658310 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:47:18.678399 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658314 5088 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:47:18.678399 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658318 5088 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:47:18.678399 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658322 5088 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:47:18.678399 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658327 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:47:18.678920 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658331 5088 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:47:18.678920 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658335 5088 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:47:18.678920 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658339 5088 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:47:18.678920 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658343 5088 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:47:18.678920 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658347 5088 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:47:18.678920 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658351 5088 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:47:18.678920 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658357 5088 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:47:18.678920 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:18.658361 5088 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:47:18.678920 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.658443 5088 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 19:47:18.678920 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.659988 5088 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 19:47:18.678920 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.660053 5088 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 19:47:18.678920 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.660329 5088 server.go:1014] "Starting client certificate rotation" Oct 30 19:47:18.678920 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.660466 5088 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 19:47:18.678920 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.660514 5088 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:41:38 +0000 UTC" deadline="2028-06-12 02:44:29.360913491 +0000 UTC" Oct 30 19:47:18.678920 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.660581 5088 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14166h57m10.700334707s" Oct 30 19:47:18.678920 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.661217 5088 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:47:18.679581 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.663062 5088 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:47:18.679581 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.665640 5088 log.go:25] "Validated CRI v1 runtime API" Oct 30 19:47:18.679581 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.667362 5088 log.go:25] "Validated CRI v1 image API" Oct 30 19:47:18.679581 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.668083 5088 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 19:47:18.679581 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.670005 5088 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7a701ec6-e7d6-469d-afe7-54d9727c599a:/dev/nvme0n1p4 968cc63f-7bdd-4a21-ac44-654993fbf10c:/dev/nvme0n1p3] Oct 30 19:47:18.679866 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.670024 5088 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/23213dbd30843595194cc488e41405260a4e82391bf97b3fec6e67059dd7cdc8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/23213dbd30843595194cc488e41405260a4e82391bf97b3fec6e67059dd7cdc8/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/26404c8e513a2d41ca262479da0df364db0e615a446757d4c5bae02dee230aad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/26404c8e513a2d41ca262479da0df364db0e615a446757d4c5bae02dee230aad/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4aac46774b6ffb3edab20dbd2bd19deea306cdd7943d68fa2691fca13aad8517/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4aac46774b6ffb3edab20dbd2bd19deea306cdd7943d68fa2691fca13aad8517/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/58e66fb0fbdcab2d2c02e4599cb05f12072986ff4279d9b50db1927ab9bf18cb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/58e66fb0fbdcab2d2c02e4599cb05f12072986ff4279d9b50db1927ab9bf18cb/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/60d9e8cdd3474de1d46b40c7de021a8ecb3649b259ff50160012f1e139c2eea0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/60d9e8cdd3474de1d46b40c7de021a8ecb3649b259ff50160012f1e139c2eea0/userdata/shm major:0 minor:69 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/634672d421bdf247c95c9ab51e410db3d67827bf8df3fc64e67196ec304d2060/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/634672d421bdf247c95c9ab51e410db3d67827bf8df3fc64e67196ec304d2060/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/66d6cb40d975bbe02e71cdfd2ac7eb157ed03b647be72ff5cb25853e5cff69d4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/66d6cb40d975bbe02e71cdfd2ac7eb157ed03b647be72ff5cb25853e5cff69d4/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/81aec110cec87d8e18ee9df4b80cca28dbd9db6491128e2ab0102eb2a60f0b7b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/81aec110cec87d8e18ee9df4b80cca28dbd9db6491128e2ab0102eb2a60f0b7b/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/841d60c54cb7391e4399d2d0c208ca47a923a3e8420002d84e1741ed7cabb499/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/841d60c54cb7391e4399d2d0c208ca47a923a3e8420002d84e1741ed7cabb499/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8ab94cd9a89ec8b7e8f48c21784a2d597264034894e149a4d80accb109d73fb2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8ab94cd9a89ec8b7e8f48c21784a2d597264034894e149a4d80accb109d73fb2/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8af5c415310b263652215a1df5401ccec65bddaf7269f2c52eceec4a3928d171/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8af5c415310b263652215a1df5401ccec65bddaf7269f2c52eceec4a3928d171/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8e1a9d3715c1d30e491ef88d0e2eb3879930812c250d7702632943bdc0298a4c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8e1a9d3715c1d30e491ef88d0e2eb3879930812c250d7702632943bdc0298a4c/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/90448fd4921147af9aa353186a47c1bc847a290dac2c8f8fa71ee2aea8aba4fc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/90448fd4921147af9aa353186a47c1bc847a290dac2c8f8fa71ee2aea8aba4fc/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/90b0f7385b5e7b323cec3734b4d8f301a608c62518a3e22a7dc8dd1a7e0dc10a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/90b0f7385b5e7b323cec3734b4d8f301a608c62518a3e22a7dc8dd1a7e0dc10a/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9457f9e3f05603a6042c4a9b4675ec328fa288e629d276292069bcd730da6f3f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9457f9e3f05603a6042c4a9b4675ec328fa288e629d276292069bcd730da6f3f/userdata/shm major:0 minor:328 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b7584d9dd4d573c6b3ee6d13272564a38c277e2ddd0433616835bdc34de8b7d5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b7584d9dd4d573c6b3ee6d13272564a38c277e2ddd0433616835bdc34de8b7d5/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b7d40f50df1defef862a02a01308a7daf4efac3422a86e9e68e684f5ca5b89db/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b7d40f50df1defef862a02a01308a7daf4efac3422a86e9e68e684f5ca5b89db/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bf54950fee0ea5f71052f3a0ba8d1af23942d0be378225fc4e5d2cfa2fcc1e69/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bf54950fee0ea5f71052f3a0ba8d1af23942d0be378225fc4e5d2cfa2fcc1e69/userdata/shm major:0 minor:340 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c89f2a9b45855b5d37585e8bf88f3445809637e073d5f3aefb19d6c70c442609/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c89f2a9b45855b5d37585e8bf88f3445809637e073d5f3aefb19d6c70c442609/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cce1b88df062b6b5cdbaeb19917460725ff99b2b7148c9944d05b9717d0c329a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cce1b88df062b6b5cdbaeb19917460725ff99b2b7148c9944d05b9717d0c329a/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cf7943e1f360d110b88532af1506657a5375a296969f623eb0255a1d8af8971b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cf7943e1f360d110b88532af1506657a5375a296969f623eb0255a1d8af8971b/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e35f258ac7b615b3bfeecd1d223abab000a5302bf771b3ee5ec4da4f4ff1a869/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e35f258ac7b615b3bfeecd1d223abab000a5302bf771b3ee5ec4da4f4ff1a869/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e368ecd6c6d37ba2ecd82cbb81f3f55544ff8fc82499f271489e82786ae6d5eb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e368ecd6c6d37ba2ecd82cbb81f3f55544ff8fc82499f271489e82786ae6d5eb/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eefa490e00b5d52d55a6bf699644887c2b2adf74b421f09f276267016e42ecee/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eefa490e00b5d52d55a6bf699644887c2b2adf74b421f09f276267016e42ecee/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f558d523550b45bdea63acef2bcb30b91f6faafc45c1bc83160a62b828ae0b63/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f558d523550b45bdea63acef2bcb30b91f6faafc45c1bc83160a62b828ae0b63/userdata/shm major:0 minor:354 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f5dc44d97b888ba5d594ce06033437408ecdf0f37751254b72ba1d4c0ff40160/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f5dc44d97b888ba5d594ce06033437408ecdf0f37751254b72ba1d4c0ff40160/userdata/shm major:0 minor:326 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/005bece8-8d37-4a58-954c-0ae3dab9dadc/volumes/kubernetes.io~projected/kube-api-access-68nwv:{mountpoint:/var/lib/kubelet/pods/005bece8-8d37-4a58-954c-0ae3dab9dadc/volumes/kubernetes.io~projected/kube-api-access-68nwv major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0547f118-5c20-403a-beca-c6a5a671a1ff/volumes/kubernetes.io~projected/kube-api-access-vvcvq:{mountpoint:/var/lib/kubelet/pods/0547f118-5c20-403a-beca-c6a5a671a1ff/volumes/kubernetes.io~projected/kube-api-access-vvcvq major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08b5a0ca-cfe1-4366-a404-5ebbaeb7c149/volumes/kubernetes.io~projected/kube-api-access-49lpb:{mountpoint:/var/lib/kubelet/pods/08b5a0ca-cfe1-4366-a404-5ebbaeb7c149/volumes/kubernetes.io~projected/kube-api-access-49lpb major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/116a1241-c9a3-4664-8217-60c2b6aa8b76/volumes/kubernetes.io~projected/kube-api-access-lph46:{mountpoint:/var/lib/kubelet/pods/116a1241-c9a3-4664-8217-60c2b6aa8b76/volumes/kubernetes.io~projected/kube-api-access-lph46 major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1342ef67-5f63-47fb-a337-64c693e11228/volumes/kubernetes.io~projected/kube-api-access-vj8ws:{mountpoint:/var/lib/kubelet/pods/1342ef67-5f63-47fb-a337-64c693e11228/volumes/kubernetes.io~projected/kube-api-access-vj8ws major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13962162-194c-4e37-ac6b-b3bbc810cedb/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/13962162-194c-4e37-ac6b-b3bbc810cedb/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1afac97f-a617-4113-b8e8-c668caed54f9/volumes/kubernetes.io~projected/kube-api-access-nr5dn:{mountpoint:/var/lib/kubelet/pods/1afac97f-a617-4113-b8e8-c668caed54f9/volumes/kubernetes.io~projected/kube-api-access-nr5dn major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/20f593c8-594e-4a64-9c5e-881311b269a3/volumes/kubernetes.io~projected/kube-api-access-6rwww:{mountpoint:/var/lib/kubelet/pods/20f593c8-594e-4a64-9c5e-881311b269a3/volumes/kubernetes.io~projected/kube-api-access-6rwww major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2962e3da-8439-4316-9c3a-1679842609b7/volumes/kubernetes.io~projected/kube-api-access-rkbkr:{mountpoint:/var/lib/kubelet/pods/2962e3da-8439-4316-9c3a-1679842609b7/volumes/kubernetes.io~projected/kube-api-access-rkbkr major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e03cb52-fd85-4e35-87b4-38833cbc7d74/volumes/kubernetes.io~projected/kube-api-access-94r98:{mountpoint:/var/lib/kubelet/pods/2e03cb52-fd85-4e35-87b4-38833cbc7d74/volumes/kubernetes.io~projected/kube-api-access-94r98 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41534930-917f-44ec-bccd-86d6c6fc7c70/volumes/kubernetes.io~projected/kube-api-access-x59q2:{mountpoint:/var/lib/kubelet/pods/41534930-917f-44ec-bccd-86d6c6fc7c70/volumes/kubernetes.io~projected/kube-api-access-x59q2 major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41534930-917f-44ec-bccd-86d6c6fc7c70/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/41534930-917f-44ec-bccd-86d6c6fc7c70/volumes/kubernetes.io~secret/default-certificate major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41534930-917f-44ec-bccd-86d6c6fc7c70/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/41534930-917f-44ec-bccd-86d6c6fc7c70/volumes/kubernetes.io~secret/stats-auth major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/57ddd2a0-f081-4f90-95a6-0f6f8df2f433/volumes/kubernetes.io~projected/kube-api-access-wb8sb:{mountpoint:/var/lib/kubelet/pods/57ddd2a0-f081-4f90-95a6-0f6f8df2f433/volumes/kubernetes.io~projected/kube-api-access-wb8sb major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ad8f4b0-50ba-4191-8df4-25930717acb0/volumes/kubernetes.io~projected/kube-api-access-lxq6z:{mountpoint:/var/lib/kubelet/pods/5ad8f4b0-50ba-4191-8df4-25930717acb0/volumes/kubernetes.io~projected/kube-api-access-lxq6z major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ad8f4b0-50ba-4191-8df4-25930717acb0/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/5ad8f4b0-50ba-4191-8df4-25930717acb0/volumes/kubernetes.io~secret/cert major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66999038-807b-406b-9242-c68c04e521f3/volumes/kubernetes.io~projected/kube-api-access-t5rr5:{mountpoint:/var/lib/kubelet/pods/66999038-807b-406b-9242-c68c04e521f3/volumes/kubernetes.io~projected/kube-api-access-t5rr5 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/688d079a-932d-4f63-97da-5cd6222a03e1/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/688d079a-932d-4f63-97da-5cd6222a03e1/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/688d079a-932d-4f63-97da-5cd6222a03e1/volumes/kubernetes.io~projected/kube-api-access-tfbsr:{mountpoint:/var/lib/kubelet/pods/688d079a-932d-4f63-97da-5cd6222a03e1/volumes/kubernetes.io~projected/kube-api-access-tfbsr major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/688d079a-932d-4f63-97da-5cd6222a03e1/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/688d079a-932d-4f63-97da-5cd6222a03e1/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73bb0a1f-597a-4078-8a01-20d0ed24715b/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/73bb0a1f-597a-4078-8a01-20d0ed24715b/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74b7c739-483f-4756-b97b-e65d37387b17/volumes/kubernetes.io~projected/kube-api-access-gbnlm:{mountpoint:/var/lib/kubelet/pods/74b7c739-483f-4756-b97b-e65d37387b17/volumes/kubernetes.io~projected/kube-api-access-gbnlm major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74b7c739-483f-4756-b97b-e65d37387b17/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/74b7c739-483f-4756-b97b-e65d37387b17/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7a570505-5193-4564-b776-0f83df60d4e9/volumes/kubernetes.io~projected/kube-api-access-tng88:{mountpoint:/var/lib/kubelet/pods/7a570505-5193-4564-b776-0f83df60d4e9/volumes/kubernetes.io~projected/kube-api-access-tng88 major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/908c2874-510c-43c4-a07c-c1771b924e8d/volumes/kubernetes.io~projected/kube-api-access-58q6j:{mountpoint:/var/lib/kubelet/pods/908c2874-510c-43c4-a07c-c1771b924e8d/volumes/kubernetes.io~projected/kube-api-access-58q6j major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396/volumes/kubernetes.io~projected/kube-api-access-rzg2k:{mountpoint:/var/lib/kubelet/pods/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396/volumes/kubernetes.io~projected/kube-api-access-rzg2k major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b6dd02b-d4bd-4959-9a06-8941d6baf1a5/volumes/kubernetes.io~projected/kube-api-access-sv4p7:{mountpoint:/var/lib/kubelet/pods/9b6dd02b-d4bd-4959-9a06-8941d6baf1a5/volumes/kubernetes.io~projected/kube-api-access-sv4p7 major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a160cec7-bc1f-4202-8382-e84318122fd9/volumes/kubernetes.io~projected/kube-api-access-blstw:{mountpoint:/var/lib/kubelet/pods/a160cec7-bc1f-4202-8382-e84318122fd9/volumes/kubernetes.io~projected/kube-api-access-blstw major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a421de2b-dd41-4862-9849-0a4f0256af1b/volumes/kubernetes.io~projected/kube-api-access-rrg8k:{mountpoint:/var/lib/kubelet/pods/a421de2b-dd41-4862-9849-0a4f0256af1b/volumes/kubernetes.io~projected/kube-api-access-rrg8k major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a421de2b-dd41-4862-9849-0a4f0256af1b/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/a421de2b-dd41-4862-9849-0a4f0256af1b/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aab2acf3-4af5-4d2e-8336-05b01a2dca1e/volumes/kubernetes.io~projected/kube-api-access-lg9ks:{mountpoint:/var/lib/kubelet/pods/aab2acf3-4af5-4d2e-8336-05b01a2dca1e/volumes/kubernetes.io~projected/kube-api-access-lg9ks major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1440092-9901-4eff-8a9d-6174b2dfdada/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/b1440092-9901-4eff-8a9d-6174b2dfdada/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1440092-9901-4eff-8a9d-6174b2dfdada/volumes/kubernetes.io~projected/kube-api-access-8xtvg:{mountpoint:/var/lib/kubelet/pods/b1440092-9901-4eff-8a9d-6174b2dfdada/volumes/kubernetes.io~projected/kube-api-access-8xtvg major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1440092-9901-4eff-8a9d-6174b2dfdada/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/b1440092-9901-4eff-8a9d-6174b2dfdada/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1440092-9901-4eff-8a9d-6174b2dfdada/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/b1440092-9901-4eff-8a9d-6174b2dfdada/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b8e3ed9d-5156-4698-82c3-df72fa728f94/volumes/kubernetes.io~projected/kube-api-access-xtksm:{mountpoint:/var/lib/kubelet/pods/b8e3ed9d-5156-4698-82c3-df72fa728f94/volumes/kubernetes.io~projected/kube-api-access-xtksm major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cba5fe49-b5b5-413e-9085-e35762430fa9/volumes/kubernetes.io~projected/kube-api-access-hpkkb:{mountpoint:/var/lib/kubelet/pods/cba5fe49-b5b5-413e-9085-e35762430fa9/volumes/kubernetes.io~projected/kube-api-access-hpkkb major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d1dca14b-4840-4dd6-a7cd-7d795adb9532/volumes/kubernetes.io~projected/kube-api-access-xtqp8:{mountpoint:/var/lib/kubelet/pods/d1dca14b-4840-4dd6-a7cd-7d795adb9532/volumes/kubernetes.io~projected/kube-api-access-xtqp8 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d71659c8-f08f-4221-a160-56ac78bd8473/volumes/kubernetes.io~projected/kube-api-access-s4h6p:{mountpoint:/var/lib/kubelet/pods/d71659c8-f08f-4221-a160-56ac78bd8473/volumes/kubernetes.io~projected/kube-api-access-s4h6p major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d71659c8-f08f-4221-a160-56ac78bd8473/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/d71659c8-f08f-4221-a160-56ac78bd8473/volumes/kubernetes.io~secret/signing-key major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d78606e4-ccb9-421a-8ae5-8fbecbcd3831/volumes/kubernetes.io~projected/kube-api-access-ljrnk:{mountpoint:/var/lib/kubelet/pods/d78606e4-ccb9-421a-8ae5-8fbecbcd3831/volumes/kubernetes.io~projected/kube-api-access-ljrnk major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d78606e4-ccb9-421a-8ae5-8fbecbcd3831/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/d78606e4-ccb9-421a-8ae5-8fbecbcd3831/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d808d253-3f30-48e8-918c-bd3880c2af8f/volumes/kubernetes.io~projected/kube-api-access-mkmdz:{mountpoint:/var/lib/kubelet/pods/d808d253-3f30-48e8-918c-bd3880c2af8f/volumes/kubernetes.io~projected/kube-api-access-mkmdz major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da850825-59ab-480f-9047-afd7d2bdc19f/volumes/kubernetes.io~projected/kube-api-access-v26lp:{mountpoint:/var/lib/kubelet/pods/da850825-59ab-480f-9047-afd7d2bdc19f/volumes/kubernetes.io~projected/kube-api-access-v26lp major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da850825-59ab-480f-9047-afd7d2bdc19f/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/da850825-59ab-480f-9047-afd7d2bdc19f/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617/volumes/kubernetes.io~projected/kube-api-access-cm7rk:{mountpoint:/var/lib/kubelet/pods/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617/volumes/kubernetes.io~projected/kube-api-access-cm7rk major:0 minor:63 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/5c600bc5ea575313457f939ddb21905d60df6132c0a3fd84ee9fa0f969e5fb62/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/6b5c87a2f19a90018cbc69ef7131ec7822b1ec0069640300021371c84cef1e6d/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/2a0f60e0ea69fe1bf6bf0f87ecae90b1f7dd434572a70837ed9fb877d4ccf67c/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/df57eb3f19fbe56f1e38de6069563f0522572d203711106aad8f21aa91777c29/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/ef59703120bb54c0856f7698f204a4a20610ba5ecabbb8450858d9ae1a7885ad/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/545b5457254165a6c3c146f66ede5c80c9d42ac839331e6fa7c35f863e020bc4/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/09bb422b28608b134197d47a16311c839b7afa431175fad188c02bc213116445/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/30cf619a19c763facb20f4d71ef4e4229e0ec17128e62f2e6e7b619aac582f7a/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/7d68a6fe5836291e9701b63225b96317caa39d55163a743b824221a00eb3108e/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/28920e7c6e2be0d55642a70cbccaa222864ba2cacf19a7290bfa15c201a45a51/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/95cbecddbd22820636e63f4f910ef00cc048d7dc17d7e133bc7ffaceed0db171/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/2e7d47b190abab117db080c01d1b2d642ef4467dcb70888e9fcc9123ff89f3e9/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/026a7ca8f5039f40561280fcc5d46ce1786e36d9a7c3bd048b2c2fc8caba7cbd/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/9ea52499ea95dc4a5a3501123e35bb70361f671ff71fb49157000effdc011493/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/7fe9bdaf7506a293dd71f718242af3d7d18e24d18a6068b7d1b844e938b31aa7/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/d4e5a780823682a4aef85c8cbe2840a6ceccdd34a8182c14c3c61af716508a30/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/d95bd2a6422c1ed7e47e0eb946aa77b578f86707637b4d737ee68c73874f4d6a/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/f84c0f78d9e45ab5f5f4482ebc63324331fe2de759cb103d7ee44851caad33a1/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/05807dd6597ad8b710657216dfe8fe4437426c13cf0e274ca753e3e1733655fa/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/308e595e4dc12848576ef16c4b7e141a24e312d9fb9b4c9d21948058c49e4929/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/dc8401d341175c33d41bce7d54a7ebf1342e0876f26768b232bc298991f9b434/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/f7b64151210b34a3b2fec18246ce4879ea3c53b8f376490fd8f45f7f61ee87a3/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/640b6957baee4eb052d175c0a4a1b5cb03f96cee807942575b684bd26f5943db/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/96e716bc572f83953990b00894a291a9796e8f05fc6554b8fb2b6f8b8e76239a/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/6f39e8544cf8d967d7a590b1b447650ee5cd5a43b768b91bd1121a6574318ac5/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/b1dd81c8361640f441d7925225ef354f7009372badeb733002296cf0cddbfdf7/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/f839a3e31442e0bbf3d4dbdb29c93049f7a7f6dc7a90aa60cc95c89e93547a08/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/59b8c4e85c4e559c73d149edfe9fe9f8e20674342a17590aebf9fcf8ac7f97e0/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/14e12361474e16d28024be1fb04f9adb5dd501054d0d340e3cf1d56f4bf37437/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/8d4dbc8f3c9d13cc86ee8ceb75834658240049788b0ed8d8fbd5128e6af91035/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/fcad99f31c32f8abf1b8594afd55b072ecabc325f1709cd5c1c2d51b20701c5d/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/8a1a019247ddb18c9584ec6079b9262f9d97bea6322ab5c0fe51fa1c9e4bd159/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/00101718a537ffedc0d7918ca3b841ad8d87771f229e6f57a3ad06483c9f06f9/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/bea79678e10e662c5a8dc8e4870197f833a3483485700329235c76d066009ddc/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/b10e1d3057f20c4f76af61b17fe2686c3bdc6a7d4b0773f3373155fa65d27a59/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/9c7c626bb2937fc44ed46df674582ea157feb8ee1fbb9f91492659059836e7f6/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/a5484b6215da069fc58d597895e3f67f4622e3e2a63748d3adcd5b6d6c9bf493/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/2a278f8ad22693cd577e49ae173c152788ac70d3de551b3fd4ea93489f60c454/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/0617bd97c8219b247fc35c34e2e971f9a05d409577fc68450c6693b959034145/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/b9754866132e7545d2444e6ad3bf7e2e344a6af70d3ee6fa029519dcd0b428c5/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/dce36ae83208f68f2235ed1420a70e2e120fd386f7dabdaa363af6b021c04064/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/d2327d7030f0038f445c8f1d84e544035f075e32d1892f7e862b71bf5ca026e5/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/219af1b5a8021d6ea11fd7a6b84e79c866b8cdc5d9709fca9537e2b9bd57bb2f/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/d02f688ccea9104b4fe1a5af0f05cd8d2d275fb0cb506dfa33224127595d0725/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/80d3fe5ae975f81bdaabdf9a2375158ab5f42356989580419f9e9f06a7ea0608/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-349:{mountpoint:/var/lib/containers/storage/overlay/cc4dd40d48d677ca331daf278a1dcac9b287d5f3ca2b3d9cdd32522fd369dcc0/merged major:0 minor:349 fsType:overlay blockSize:0} overlay_0-351:{mountpoint:/var/lib/containers/storage/overlay/1fbc2933dc67b30e04cc2be532a436b962060ee5cfe022a6da15a6820d51152b/merged major:0 minor:351 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/0cda4339f8d77d93d8ff9b428a30db1b20a3554263806998008f7f704e268298/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/b4b4796fbeb362175c6f887c225e6d56fae847c63a72e2501eb85355a9451706/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/1d2e321d9671bfe8680e8befcd4b9d29e5831e590eb60bbbcd4ce905e5a8c6d6/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/0126094659cc4e9192480e0641f4b8ba9563d48518c831ed0dd33de3985e1028/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/ca0a83891d69f32b5b5a94ed0c9fdbb051498ad417c7015f538bdf4ad39a0987/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/7ab8d224f2c625cf65a74aa330c2a1a729d6ecf34a2533c20154d280a312ef1c/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/01cf9fa1b1975a9744340b630f0d7398bd0a3248c7c63ece9ed9923c01ee7f32/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/0fcc288fce7a517022adbb833b1fdb30a0ef5b7aac38a444d0f87876813645e9/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/3d2af0f2fce39c6c240a7e5d8bbe7aad35c6a8a02f94289e98a4d85871aaf376/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/132b54b217ac1aef39cd0894156e892d1378be1252b72a56bc2b567d7c466f5d/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/9e103878b062586767ab1e4f62e3443d94d26e84dc5707d2e7b648aee79e11f0/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/4ffd0e065147e464448ee9f055824bd4732063b039f619e56075ad0e483d3c25/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/f9c1bcf002bd772375136d430d26fdb00aa0494ae0af3f9858341aca0756ef37/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/dc2c72400b39071c8991232a8a0ff889cae950be8a8c55ef93d02133b120a52d/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/779e856efa12baa2013c1a59ecda98bfc746540dcf2bfad4c41d76895695dda0/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 19:47:18.680236 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.677492 5088 manager.go:217] Machine: {Timestamp:2026-10-30 19:47:18.677038885 +0000 UTC m=+0.089752655 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3199924 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25aaa95da08431c7fc2d9571a19a09 SystemUUID:ec25aaa9-5da0-8431-c7fc-2d9571a19a09 BootID:fb82e4d9-09ae-4b6d-8184-120a39cd7047 Filesystems:[{Device:/var/lib/kubelet/pods/d1dca14b-4840-4dd6-a7cd-7d795adb9532/volumes/kubernetes.io~projected/kube-api-access-xtqp8 DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/74b7c739-483f-4756-b97b-e65d37387b17/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d78606e4-ccb9-421a-8ae5-8fbecbcd3831/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b1440092-9901-4eff-8a9d-6174b2dfdada/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/116a1241-c9a3-4664-8217-60c2b6aa8b76/volumes/kubernetes.io~projected/kube-api-access-lph46 DeviceMajor:0 DeviceMinor:293 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/74b7c739-483f-4756-b97b-e65d37387b17/volumes/kubernetes.io~projected/kube-api-access-gbnlm DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/57ddd2a0-f081-4f90-95a6-0f6f8df2f433/volumes/kubernetes.io~projected/kube-api-access-wb8sb DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8ab94cd9a89ec8b7e8f48c21784a2d597264034894e149a4d80accb109d73fb2/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b8e3ed9d-5156-4698-82c3-df72fa728f94/volumes/kubernetes.io~projected/kube-api-access-xtksm DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9b6dd02b-d4bd-4959-9a06-8941d6baf1a5/volumes/kubernetes.io~projected/kube-api-access-sv4p7 DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/688d079a-932d-4f63-97da-5cd6222a03e1/volumes/kubernetes.io~projected/kube-api-access-tfbsr DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5ad8f4b0-50ba-4191-8df4-25930717acb0/volumes/kubernetes.io~projected/kube-api-access-lxq6z DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0547f118-5c20-403a-beca-c6a5a671a1ff/volumes/kubernetes.io~projected/kube-api-access-vvcvq DeviceMajor:0 DeviceMinor:325 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f5dc44d97b888ba5d594ce06033437408ecdf0f37751254b72ba1d4c0ff40160/userdata/shm DeviceMajor:0 DeviceMinor:326 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1afac97f-a617-4113-b8e8-c668caed54f9/volumes/kubernetes.io~projected/kube-api-access-nr5dn DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/908c2874-510c-43c4-a07c-c1771b924e8d/volumes/kubernetes.io~projected/kube-api-access-58q6j DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/41534930-917f-44ec-bccd-86d6c6fc7c70/volumes/kubernetes.io~projected/kube-api-access-x59q2 DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1342ef67-5f63-47fb-a337-64c693e11228/volumes/kubernetes.io~projected/kube-api-access-vj8ws DeviceMajor:0 DeviceMinor:227 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aab2acf3-4af5-4d2e-8336-05b01a2dca1e/volumes/kubernetes.io~projected/kube-api-access-lg9ks DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b1440092-9901-4eff-8a9d-6174b2dfdada/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/41534930-917f-44ec-bccd-86d6c6fc7c70/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/20f593c8-594e-4a64-9c5e-881311b269a3/volumes/kubernetes.io~projected/kube-api-access-6rwww DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8e1a9d3715c1d30e491ef88d0e2eb3879930812c250d7702632943bdc0298a4c/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4aac46774b6ffb3edab20dbd2bd19deea306cdd7943d68fa2691fca13aad8517/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/005bece8-8d37-4a58-954c-0ae3dab9dadc/volumes/kubernetes.io~projected/kube-api-access-68nwv DeviceMajor:0 DeviceMinor:339 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-349 DeviceMajor:0 DeviceMinor:349 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2e03cb52-fd85-4e35-87b4-38833cbc7d74/volumes/kubernetes.io~projected/kube-api-access-94r98 DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d808d253-3f30-48e8-918c-bd3880c2af8f/volumes/kubernetes.io~projected/kube-api-access-mkmdz DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cce1b88df062b6b5cdbaeb19917460725ff99b2b7148c9944d05b9717d0c329a/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/81aec110cec87d8e18ee9df4b80cca28dbd9db6491128e2ab0102eb2a60f0b7b/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d71659c8-f08f-4221-a160-56ac78bd8473/volumes/kubernetes.io~projected/kube-api-access-s4h6p DeviceMajor:0 DeviceMinor:353 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cf7943e1f360d110b88532af1506657a5375a296969f623eb0255a1d8af8971b/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/58e66fb0fbdcab2d2c02e4599cb05f12072986ff4279d9b50db1927ab9bf18cb/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/23213dbd30843595194cc488e41405260a4e82391bf97b3fec6e67059dd7cdc8/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/688d079a-932d-4f63-97da-5cd6222a03e1/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/5ad8f4b0-50ba-4191-8df4-25930717acb0/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/73bb0a1f-597a-4078-8a01-20d0ed24715b/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/841d60c54cb7391e4399d2d0c208ca47a923a3e8420002d84e1741ed7cabb499/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396/volumes/kubernetes.io~projected/kube-api-access-rzg2k DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a160cec7-bc1f-4202-8382-e84318122fd9/volumes/kubernetes.io~projected/kube-api-access-blstw DeviceMajor:0 DeviceMinor:338 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d71659c8-f08f-4221-a160-56ac78bd8473/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:348 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/41534930-917f-44ec-bccd-86d6c6fc7c70/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b1440092-9901-4eff-8a9d-6174b2dfdada/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9457f9e3f05603a6042c4a9b4675ec328fa288e629d276292069bcd730da6f3f/userdata/shm DeviceMajor:0 DeviceMinor:328 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/13962162-194c-4e37-ac6b-b3bbc810cedb/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e368ecd6c6d37ba2ecd82cbb81f3f55544ff8fc82499f271489e82786ae6d5eb/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a421de2b-dd41-4862-9849-0a4f0256af1b/volumes/kubernetes.io~projected/kube-api-access-rrg8k DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/26404c8e513a2d41ca262479da0df364db0e615a446757d4c5bae02dee230aad/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/688d079a-932d-4f63-97da-5cd6222a03e1/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e35f258ac7b615b3bfeecd1d223abab000a5302bf771b3ee5ec4da4f4ff1a869/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/90448fd4921147af9aa353186a47c1bc847a290dac2c8f8fa71ee2aea8aba4fc/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8af5c415310b263652215a1df5401ccec65bddaf7269f2c52eceec4a3928d171/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7a570505-5193-4564-b776-0f83df60d4e9/volumes/kubernetes.io~projected/kube-api-access-tng88 DeviceMajor:0 DeviceMinor:312 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-351 DeviceMajor:0 DeviceMinor:351 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2962e3da-8439-4316-9c3a-1679842609b7/volumes/kubernetes.io~projected/kube-api-access-rkbkr DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cba5fe49-b5b5-413e-9085-e35762430fa9/volumes/kubernetes.io~projected/kube-api-access-hpkkb DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/66d6cb40d975bbe02e71cdfd2ac7eb157ed03b647be72ff5cb25853e5cff69d4/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/08b5a0ca-cfe1-4366-a404-5ebbaeb7c149/volumes/kubernetes.io~projected/kube-api-access-49lpb DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/60d9e8cdd3474de1d46b40c7de021a8ecb3649b259ff50160012f1e139c2eea0/userdata/shm DeviceMajor:0 DeviceMinor:69 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b1440092-9901-4eff-8a9d-6174b2dfdada/volumes/kubernetes.io~projected/kube-api-access-8xtvg DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617/volumes/kubernetes.io~projected/kube-api-access-cm7rk DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a421de2b-dd41-4862-9849-0a4f0256af1b/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b7d40f50df1defef862a02a01308a7daf4efac3422a86e9e68e684f5ca5b89db/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eefa490e00b5d52d55a6bf699644887c2b2adf74b421f09f276267016e42ecee/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b7584d9dd4d573c6b3ee6d13272564a38c277e2ddd0433616835bdc34de8b7d5/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/90b0f7385b5e7b323cec3734b4d8f301a608c62518a3e22a7dc8dd1a7e0dc10a/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/da850825-59ab-480f-9047-afd7d2bdc19f/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c89f2a9b45855b5d37585e8bf88f3445809637e073d5f3aefb19d6c70c442609/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/da850825-59ab-480f-9047-afd7d2bdc19f/volumes/kubernetes.io~projected/kube-api-access-v26lp DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bf54950fee0ea5f71052f3a0ba8d1af23942d0be378225fc4e5d2cfa2fcc1e69/userdata/shm DeviceMajor:0 DeviceMinor:340 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/634672d421bdf247c95c9ab51e410db3d67827bf8df3fc64e67196ec304d2060/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/66999038-807b-406b-9242-c68c04e521f3/volumes/kubernetes.io~projected/kube-api-access-t5rr5 DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d78606e4-ccb9-421a-8ae5-8fbecbcd3831/volumes/kubernetes.io~projected/kube-api-access-ljrnk DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f558d523550b45bdea63acef2bcb30b91f6faafc45c1bc83160a62b828ae0b63/userdata/shm DeviceMajor:0 DeviceMinor:354 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:26404c8e513a2d4 MacAddress:ba:41:d8:26:12:92 Speed:10000 Mtu:8901} {Name:4aac46774b6ffb3 MacAddress:1e:6c:02:9b:75:90 Speed:10000 Mtu:8901} {Name:58e66fb0fbdcab2 MacAddress:26:20:7a:82:3f:93 Speed:10000 Mtu:8901} {Name:66d6cb40d975bbe MacAddress:96:ac:3a:d7:36:55 Speed:10000 Mtu:8901} {Name:81aec110cec87d8 MacAddress:6a:f0:ce:5d:ba:ee Speed:10000 Mtu:8901} {Name:8af5c415310b263 MacAddress:0e:b5:0c:3c:e0:67 Speed:10000 Mtu:8901} {Name:8e1a9d3715c1d30 MacAddress:aa:3c:f6:fe:ff:fd Speed:10000 Mtu:8901} {Name:9457f9e3f05603a MacAddress:f2:39:58:aa:0b:ce Speed:10000 Mtu:8901} {Name:b7584d9dd4d573c MacAddress:2a:2b:d5:81:e1:77 Speed:10000 Mtu:8901} {Name:b7d40f50df1defe MacAddress:d2:92:29:45:6d:d1 Speed:10000 Mtu:8901} {Name:bf54950fee0ea5f MacAddress:5a:ba:8b:88:15:a9 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c3:8c:38:d1 Speed:0 Mtu:9001} {Name:br-int MacAddress:e2:79:2c:b0:0b:cb Speed:0 Mtu:8901} {Name:c89f2a9b45855b5 MacAddress:a6:41:ba:6c:80:6c Speed:10000 Mtu:8901} {Name:cce1b88df062b6b MacAddress:66:48:6b:e6:48:72 Speed:10000 Mtu:8901} {Name:eefa490e00b5d52 MacAddress:92:d8:32:d8:a3:c5 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:c3:8c:38:d1 Speed:0 Mtu:9001} {Name:f558d523550b45b MacAddress:66:0c:07:44:1c:2e Speed:10000 Mtu:8901} {Name:f5dc44d97b888ba MacAddress:5a:bf:4a:d1:f3:7b Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:96:a3:f3:d3:27:2e Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:8a:7e:a0:df:f3:42 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 19:47:18.680236 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.677833 5088 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 19:47:18.680236 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.677874 5088 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 19:47:18.680236 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.678172 5088 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 19:47:18.680236 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.678189 5088 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-173.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 19:47:18.680236 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.678317 5088 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 19:47:18.680236 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.678324 5088 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 19:47:18.680236 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.678337 5088 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:47:18.680236 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.678356 5088 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:47:18.680236 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.678374 5088 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 19:47:18.680236 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.678548 5088 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 19:47:18.680236 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.678685 5088 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 19:47:18.680236 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.678718 5088 kubelet.go:493] "Attempting to sync node with API server" Oct 30 19:47:18.680236 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.678728 5088 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 19:47:18.680236 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.678740 5088 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 19:47:18.680236 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.678753 5088 kubelet.go:405] "Adding apiserver pod source" Oct 30 19:47:18.680236 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.678761 5088 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 19:47:18.680236 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.679631 5088 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 19:47:18.680236 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.679643 5088 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:47:18.680236 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.680176 5088 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 19:47:18.684647 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.684629 5088 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 19:47:18.684647 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.684645 5088 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:47:18.684771 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.684731 5088 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 19:47:18.684932 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.684920 5088 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 19:47:18.684970 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.684948 5088 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 19:47:18.685042 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.685033 5088 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 19:47:18.685072 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.685046 5088 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 19:47:18.685072 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.685052 5088 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 19:47:18.685072 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.685058 5088 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 19:47:18.685072 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.685063 5088 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 19:47:18.685072 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.685068 5088 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 19:47:18.685072 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.685074 5088 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 19:47:18.685259 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.685079 5088 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 19:47:18.685259 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.685085 5088 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 19:47:18.685259 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.685091 5088 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 19:47:18.685259 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.685099 5088 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 19:47:18.685259 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.685108 5088 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 19:47:18.685259 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.685133 5088 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 19:47:18.685259 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.685140 5088 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 19:47:18.688555 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.688522 5088 server.go:1263] "Started kubelet" Oct 30 19:47:18.688661 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.688621 5088 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 19:47:18.692123 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.688719 5088 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 19:47:18.692123 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.688797 5088 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 19:47:18.692123 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.689177 5088 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 19:47:18.692123 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.689988 5088 server.go:317] "Adding debug handlers to kubelet server" Oct 30 19:47:18.692123 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.691575 5088 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 19:47:18.692123 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.691598 5088 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 19:47:18.692123 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.691676 5088 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:41:39 +0000 UTC" deadline="2028-06-04 06:39:52.59625747 +0000 UTC" Oct 30 19:47:18.692123 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.691707 5088 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13978h52m33.904553804s" Oct 30 19:47:18.692123 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:47:18.691739 5088 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-173.ec2.internal\" not found" Oct 30 19:47:18.692123 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.691775 5088 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 19:47:18.692123 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.691784 5088 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 19:47:18.692123 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.691869 5088 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 19:47:18.689052 ip-10-0-132-173 systemd[1]: Started Kubernetes Kubelet. Oct 30 19:47:18.693803 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.693781 5088 factory.go:55] Registering systemd factory Oct 30 19:47:18.693803 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.693807 5088 factory.go:223] Registration of the systemd container factory successfully Oct 30 19:47:18.694964 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.694948 5088 factory.go:153] Registering CRI-O factory Oct 30 19:47:18.695070 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.694966 5088 factory.go:223] Registration of the crio container factory successfully Oct 30 19:47:18.695070 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.695043 5088 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 19:47:18.695070 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.695067 5088 factory.go:103] Registering Raw factory Oct 30 19:47:18.695255 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.695083 5088 manager.go:1199] Started watching for new ooms in manager Oct 30 19:47:18.695558 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.695539 5088 manager.go:319] Starting recovery of all containers Oct 30 19:47:18.698048 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.697722 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13962162-194c-4e37-ac6b-b3bbc810cedb" volumeName="kubernetes.io/secret/13962162-194c-4e37-ac6b-b3bbc810cedb-agent-certs" seLinuxMountContext="" Oct 30 19:47:18.698048 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.697786 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1afac97f-a617-4113-b8e8-c668caed54f9" volumeName="kubernetes.io/configmap/1afac97f-a617-4113-b8e8-c668caed54f9-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 19:47:18.698048 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.697803 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a8ef63e-bcc4-4b71-b345-3fcbc48f4396" volumeName="kubernetes.io/empty-dir/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-ca-trust-extracted" seLinuxMountContext="" Oct 30 19:47:18.698048 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.697820 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a8ef63e-bcc4-4b71-b345-3fcbc48f4396" volumeName="kubernetes.io/secret/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-image-registry-private-configuration" seLinuxMountContext="" Oct 30 19:47:18.698048 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.697839 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73bb0a1f-597a-4078-8a01-20d0ed24715b" volumeName="kubernetes.io/secret/73bb0a1f-597a-4078-8a01-20d0ed24715b-original-pull-secret" seLinuxMountContext="" Oct 30 19:47:18.698048 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.697852 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a421de2b-dd41-4862-9849-0a4f0256af1b" volumeName="kubernetes.io/projected/a421de2b-dd41-4862-9849-0a4f0256af1b-kube-api-access-rrg8k" seLinuxMountContext="" Oct 30 19:47:18.698048 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.697869 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1dca14b-4840-4dd6-a7cd-7d795adb9532" volumeName="kubernetes.io/projected/d1dca14b-4840-4dd6-a7cd-7d795adb9532-kube-api-access-xtqp8" seLinuxMountContext="" Oct 30 19:47:18.698048 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.697899 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d78606e4-ccb9-421a-8ae5-8fbecbcd3831" volumeName="kubernetes.io/secret/d78606e4-ccb9-421a-8ae5-8fbecbcd3831-serving-cert" seLinuxMountContext="" Oct 30 19:47:18.698048 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.697923 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="005bece8-8d37-4a58-954c-0ae3dab9dadc" volumeName="kubernetes.io/configmap/005bece8-8d37-4a58-954c-0ae3dab9dadc-service-ca-bundle" seLinuxMountContext="" Oct 30 19:47:18.698048 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.697941 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e03cb52-fd85-4e35-87b4-38833cbc7d74" volumeName="kubernetes.io/projected/2e03cb52-fd85-4e35-87b4-38833cbc7d74-kube-api-access-94r98" seLinuxMountContext="" Oct 30 19:47:18.698048 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.697953 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41534930-917f-44ec-bccd-86d6c6fc7c70" volumeName="kubernetes.io/secret/41534930-917f-44ec-bccd-86d6c6fc7c70-default-certificate" seLinuxMountContext="" Oct 30 19:47:18.698048 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.697971 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="688d079a-932d-4f63-97da-5cd6222a03e1" volumeName="kubernetes.io/secret/688d079a-932d-4f63-97da-5cd6222a03e1-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 19:47:18.698048 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.697983 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74b7c739-483f-4756-b97b-e65d37387b17" volumeName="kubernetes.io/configmap/74b7c739-483f-4756-b97b-e65d37387b17-config" seLinuxMountContext="" Oct 30 19:47:18.698048 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.697999 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b6dd02b-d4bd-4959-9a06-8941d6baf1a5" volumeName="kubernetes.io/projected/9b6dd02b-d4bd-4959-9a06-8941d6baf1a5-kube-api-access-sv4p7" seLinuxMountContext="" Oct 30 19:47:18.698048 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.698030 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8e3ed9d-5156-4698-82c3-df72fa728f94" volumeName="kubernetes.io/empty-dir/b8e3ed9d-5156-4698-82c3-df72fa728f94-tmp-dir" seLinuxMountContext="" Oct 30 19:47:18.698048 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.698054 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d78606e4-ccb9-421a-8ae5-8fbecbcd3831" volumeName="kubernetes.io/empty-dir/d78606e4-ccb9-421a-8ae5-8fbecbcd3831-snapshots" seLinuxMountContext="" Oct 30 19:47:18.698873 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.698072 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0547f118-5c20-403a-beca-c6a5a671a1ff" volumeName="kubernetes.io/projected/0547f118-5c20-403a-beca-c6a5a671a1ff-kube-api-access-vvcvq" seLinuxMountContext="" Oct 30 19:47:18.698873 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.698095 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20f593c8-594e-4a64-9c5e-881311b269a3" volumeName="kubernetes.io/projected/20f593c8-594e-4a64-9c5e-881311b269a3-kube-api-access-6rwww" seLinuxMountContext="" Oct 30 19:47:18.698873 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.698113 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41534930-917f-44ec-bccd-86d6c6fc7c70" volumeName="kubernetes.io/secret/41534930-917f-44ec-bccd-86d6c6fc7c70-stats-auth" seLinuxMountContext="" Oct 30 19:47:18.698873 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.698125 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57ddd2a0-f081-4f90-95a6-0f6f8df2f433" volumeName="kubernetes.io/projected/57ddd2a0-f081-4f90-95a6-0f6f8df2f433-kube-api-access-wb8sb" seLinuxMountContext="" Oct 30 19:47:18.698873 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.698143 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1440092-9901-4eff-8a9d-6174b2dfdada" volumeName="kubernetes.io/configmap/b1440092-9901-4eff-8a9d-6174b2dfdada-trusted-ca" seLinuxMountContext="" Oct 30 19:47:18.698873 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.698157 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1440092-9901-4eff-8a9d-6174b2dfdada" volumeName="kubernetes.io/empty-dir/b1440092-9901-4eff-8a9d-6174b2dfdada-ca-trust-extracted" seLinuxMountContext="" Oct 30 19:47:18.698873 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.698183 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1440092-9901-4eff-8a9d-6174b2dfdada" volumeName="kubernetes.io/secret/b1440092-9901-4eff-8a9d-6174b2dfdada-image-registry-private-configuration" seLinuxMountContext="" Oct 30 19:47:18.698873 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.698202 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ad8f4b0-50ba-4191-8df4-25930717acb0" volumeName="kubernetes.io/secret/5ad8f4b0-50ba-4191-8df4-25930717acb0-cert" seLinuxMountContext="" Oct 30 19:47:18.698873 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.698216 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66999038-807b-406b-9242-c68c04e521f3" volumeName="kubernetes.io/configmap/66999038-807b-406b-9242-c68c04e521f3-cni-binary-copy" seLinuxMountContext="" Oct 30 19:47:18.698873 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.698234 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="688d079a-932d-4f63-97da-5cd6222a03e1" volumeName="kubernetes.io/projected/688d079a-932d-4f63-97da-5cd6222a03e1-kube-api-access-tfbsr" seLinuxMountContext="" Oct 30 19:47:18.698873 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.698253 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a421de2b-dd41-4862-9849-0a4f0256af1b" volumeName="kubernetes.io/secret/a421de2b-dd41-4862-9849-0a4f0256af1b-serving-cert" seLinuxMountContext="" Oct 30 19:47:18.698873 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.698274 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1440092-9901-4eff-8a9d-6174b2dfdada" volumeName="kubernetes.io/configmap/b1440092-9901-4eff-8a9d-6174b2dfdada-registry-certificates" seLinuxMountContext="" Oct 30 19:47:18.698873 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.698322 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d71659c8-f08f-4221-a160-56ac78bd8473" volumeName="kubernetes.io/projected/d71659c8-f08f-4221-a160-56ac78bd8473-kube-api-access-s4h6p" seLinuxMountContext="" Oct 30 19:47:18.698873 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.698342 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2962e3da-8439-4316-9c3a-1679842609b7" volumeName="kubernetes.io/projected/2962e3da-8439-4316-9c3a-1679842609b7-kube-api-access-rkbkr" seLinuxMountContext="" Oct 30 19:47:18.698873 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.698434 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a8ef63e-bcc4-4b71-b345-3fcbc48f4396" volumeName="kubernetes.io/projected/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-kube-api-access-rzg2k" seLinuxMountContext="" Oct 30 19:47:18.698873 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.698519 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d808d253-3f30-48e8-918c-bd3880c2af8f" volumeName="kubernetes.io/projected/d808d253-3f30-48e8-918c-bd3880c2af8f-kube-api-access-mkmdz" seLinuxMountContext="" Oct 30 19:47:18.698873 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.698541 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74b7c739-483f-4756-b97b-e65d37387b17" volumeName="kubernetes.io/projected/74b7c739-483f-4756-b97b-e65d37387b17-kube-api-access-gbnlm" seLinuxMountContext="" Oct 30 19:47:18.698873 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.698585 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a8ef63e-bcc4-4b71-b345-3fcbc48f4396" volumeName="kubernetes.io/configmap/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-registry-certificates" seLinuxMountContext="" Oct 30 19:47:18.698873 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.698605 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a8ef63e-bcc4-4b71-b345-3fcbc48f4396" volumeName="kubernetes.io/configmap/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-trusted-ca" seLinuxMountContext="" Oct 30 19:47:18.698873 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.698632 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cba5fe49-b5b5-413e-9085-e35762430fa9" volumeName="kubernetes.io/projected/cba5fe49-b5b5-413e-9085-e35762430fa9-kube-api-access-hpkkb" seLinuxMountContext="" Oct 30 19:47:18.698873 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.698660 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d78606e4-ccb9-421a-8ae5-8fbecbcd3831" volumeName="kubernetes.io/projected/d78606e4-ccb9-421a-8ae5-8fbecbcd3831-kube-api-access-ljrnk" seLinuxMountContext="" Oct 30 19:47:18.698873 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.698677 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13962162-194c-4e37-ac6b-b3bbc810cedb" volumeName="kubernetes.io/configmap/13962162-194c-4e37-ac6b-b3bbc810cedb-konnectivity-ca" seLinuxMountContext="" Oct 30 19:47:18.698873 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.698690 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="116a1241-c9a3-4664-8217-60c2b6aa8b76" volumeName="kubernetes.io/projected/116a1241-c9a3-4664-8217-60c2b6aa8b76-kube-api-access-lph46" seLinuxMountContext="" Oct 30 19:47:18.698873 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.698734 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1342ef67-5f63-47fb-a337-64c693e11228" volumeName="kubernetes.io/configmap/1342ef67-5f63-47fb-a337-64c693e11228-samples-operator-config" seLinuxMountContext="" Oct 30 19:47:18.698873 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.698748 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a8ef63e-bcc4-4b71-b345-3fcbc48f4396" volumeName="kubernetes.io/secret/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-installation-pull-secrets" seLinuxMountContext="" Oct 30 19:47:18.698873 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.698761 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da850825-59ab-480f-9047-afd7d2bdc19f" volumeName="kubernetes.io/configmap/da850825-59ab-480f-9047-afd7d2bdc19f-trusted-ca" seLinuxMountContext="" Oct 30 19:47:18.698873 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.698808 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1afac97f-a617-4113-b8e8-c668caed54f9" volumeName="kubernetes.io/configmap/1afac97f-a617-4113-b8e8-c668caed54f9-cni-binary-copy" seLinuxMountContext="" Oct 30 19:47:18.698873 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.698863 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66999038-807b-406b-9242-c68c04e521f3" volumeName="kubernetes.io/configmap/66999038-807b-406b-9242-c68c04e521f3-multus-daemon-config" seLinuxMountContext="" Oct 30 19:47:18.698873 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.698882 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1440092-9901-4eff-8a9d-6174b2dfdada" volumeName="kubernetes.io/secret/b1440092-9901-4eff-8a9d-6174b2dfdada-installation-pull-secrets" seLinuxMountContext="" Oct 30 19:47:18.700260 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.698908 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d71659c8-f08f-4221-a160-56ac78bd8473" volumeName="kubernetes.io/secret/d71659c8-f08f-4221-a160-56ac78bd8473-signing-key" seLinuxMountContext="" Oct 30 19:47:18.700260 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.698924 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e39329dd-fe4e-4ef9-853b-d8e5fe9d0617" volumeName="kubernetes.io/projected/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-kube-api-access-cm7rk" seLinuxMountContext="" Oct 30 19:47:18.700260 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.698961 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="005bece8-8d37-4a58-954c-0ae3dab9dadc" volumeName="kubernetes.io/empty-dir/005bece8-8d37-4a58-954c-0ae3dab9dadc-archives-path" seLinuxMountContext="" Oct 30 19:47:18.700260 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.698989 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08b5a0ca-cfe1-4366-a404-5ebbaeb7c149" volumeName="kubernetes.io/projected/08b5a0ca-cfe1-4366-a404-5ebbaeb7c149-kube-api-access-49lpb" seLinuxMountContext="" Oct 30 19:47:18.700260 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.699020 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1440092-9901-4eff-8a9d-6174b2dfdada" volumeName="kubernetes.io/projected/b1440092-9901-4eff-8a9d-6174b2dfdada-kube-api-access-8xtvg" seLinuxMountContext="" Oct 30 19:47:18.700260 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.699032 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8e3ed9d-5156-4698-82c3-df72fa728f94" volumeName="kubernetes.io/projected/b8e3ed9d-5156-4698-82c3-df72fa728f94-kube-api-access-xtksm" seLinuxMountContext="" Oct 30 19:47:18.700260 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.699068 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e39329dd-fe4e-4ef9-853b-d8e5fe9d0617" volumeName="kubernetes.io/empty-dir/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-etc-tuned" seLinuxMountContext="" Oct 30 19:47:18.700260 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.699089 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1afac97f-a617-4113-b8e8-c668caed54f9" volumeName="kubernetes.io/configmap/1afac97f-a617-4113-b8e8-c668caed54f9-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 19:47:18.700260 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.699109 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1afac97f-a617-4113-b8e8-c668caed54f9" volumeName="kubernetes.io/projected/1afac97f-a617-4113-b8e8-c668caed54f9-kube-api-access-nr5dn" seLinuxMountContext="" Oct 30 19:47:18.700260 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.699140 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a8ef63e-bcc4-4b71-b345-3fcbc48f4396" volumeName="kubernetes.io/projected/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-bound-sa-token" seLinuxMountContext="" Oct 30 19:47:18.700260 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.699154 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a421de2b-dd41-4862-9849-0a4f0256af1b" volumeName="kubernetes.io/configmap/a421de2b-dd41-4862-9849-0a4f0256af1b-config" seLinuxMountContext="" Oct 30 19:47:18.700260 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.699209 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d71659c8-f08f-4221-a160-56ac78bd8473" volumeName="kubernetes.io/configmap/d71659c8-f08f-4221-a160-56ac78bd8473-signing-cabundle" seLinuxMountContext="" Oct 30 19:47:18.700260 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.699236 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da850825-59ab-480f-9047-afd7d2bdc19f" volumeName="kubernetes.io/secret/da850825-59ab-480f-9047-afd7d2bdc19f-serving-cert" seLinuxMountContext="" Oct 30 19:47:18.700260 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.699247 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ad8f4b0-50ba-4191-8df4-25930717acb0" volumeName="kubernetes.io/projected/5ad8f4b0-50ba-4191-8df4-25930717acb0-kube-api-access-lxq6z" seLinuxMountContext="" Oct 30 19:47:18.700260 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.699335 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a570505-5193-4564-b776-0f83df60d4e9" volumeName="kubernetes.io/projected/7a570505-5193-4564-b776-0f83df60d4e9-kube-api-access-tng88" seLinuxMountContext="" Oct 30 19:47:18.700260 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.699351 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a160cec7-bc1f-4202-8382-e84318122fd9" volumeName="kubernetes.io/empty-dir/a160cec7-bc1f-4202-8382-e84318122fd9-data-volume" seLinuxMountContext="" Oct 30 19:47:18.700260 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.699362 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afb6fa1c-9722-42c4-a914-4941bcbcaaad" volumeName="kubernetes.io/configmap/afb6fa1c-9722-42c4-a914-4941bcbcaaad-nginx-conf" seLinuxMountContext="" Oct 30 19:47:18.700260 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.699379 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e39329dd-fe4e-4ef9-853b-d8e5fe9d0617" volumeName="kubernetes.io/empty-dir/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-tmp" seLinuxMountContext="" Oct 30 19:47:18.700260 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.699393 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="005bece8-8d37-4a58-954c-0ae3dab9dadc" volumeName="kubernetes.io/projected/005bece8-8d37-4a58-954c-0ae3dab9dadc-kube-api-access-68nwv" seLinuxMountContext="" Oct 30 19:47:18.700260 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.699410 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="688d079a-932d-4f63-97da-5cd6222a03e1" volumeName="kubernetes.io/configmap/688d079a-932d-4f63-97da-5cd6222a03e1-ovnkube-script-lib" seLinuxMountContext="" Oct 30 19:47:18.700260 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.699423 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74b7c739-483f-4756-b97b-e65d37387b17" volumeName="kubernetes.io/secret/74b7c739-483f-4756-b97b-e65d37387b17-serving-cert" seLinuxMountContext="" Oct 30 19:47:18.700260 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.699436 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da850825-59ab-480f-9047-afd7d2bdc19f" volumeName="kubernetes.io/configmap/da850825-59ab-480f-9047-afd7d2bdc19f-config" seLinuxMountContext="" Oct 30 19:47:18.700260 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.699464 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="005bece8-8d37-4a58-954c-0ae3dab9dadc" volumeName="kubernetes.io/configmap/005bece8-8d37-4a58-954c-0ae3dab9dadc-trusted-ca-bundle" seLinuxMountContext="" Oct 30 19:47:18.700260 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.699477 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e03cb52-fd85-4e35-87b4-38833cbc7d74" volumeName="kubernetes.io/configmap/2e03cb52-fd85-4e35-87b4-38833cbc7d74-serviceca" seLinuxMountContext="" Oct 30 19:47:18.700260 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.699495 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41534930-917f-44ec-bccd-86d6c6fc7c70" volumeName="kubernetes.io/projected/41534930-917f-44ec-bccd-86d6c6fc7c70-kube-api-access-x59q2" seLinuxMountContext="" Oct 30 19:47:18.700260 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.699507 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="688d079a-932d-4f63-97da-5cd6222a03e1" volumeName="kubernetes.io/configmap/688d079a-932d-4f63-97da-5cd6222a03e1-ovnkube-config" seLinuxMountContext="" Oct 30 19:47:18.700260 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.699518 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aab2acf3-4af5-4d2e-8336-05b01a2dca1e" volumeName="kubernetes.io/projected/aab2acf3-4af5-4d2e-8336-05b01a2dca1e-kube-api-access-lg9ks" seLinuxMountContext="" Oct 30 19:47:18.700260 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.699535 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d78606e4-ccb9-421a-8ae5-8fbecbcd3831" volumeName="kubernetes.io/configmap/d78606e4-ccb9-421a-8ae5-8fbecbcd3831-trusted-ca-bundle" seLinuxMountContext="" Oct 30 19:47:18.700260 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.699546 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d78606e4-ccb9-421a-8ae5-8fbecbcd3831" volumeName="kubernetes.io/empty-dir/d78606e4-ccb9-421a-8ae5-8fbecbcd3831-tmp" seLinuxMountContext="" Oct 30 19:47:18.700260 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.699564 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66999038-807b-406b-9242-c68c04e521f3" volumeName="kubernetes.io/projected/66999038-807b-406b-9242-c68c04e521f3-kube-api-access-t5rr5" seLinuxMountContext="" Oct 30 19:47:18.700260 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.699577 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="688d079a-932d-4f63-97da-5cd6222a03e1" volumeName="kubernetes.io/configmap/688d079a-932d-4f63-97da-5cd6222a03e1-env-overrides" seLinuxMountContext="" Oct 30 19:47:18.700260 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.699589 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="908c2874-510c-43c4-a07c-c1771b924e8d" volumeName="kubernetes.io/projected/908c2874-510c-43c4-a07c-c1771b924e8d-kube-api-access-58q6j" seLinuxMountContext="" Oct 30 19:47:18.700260 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.699627 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1440092-9901-4eff-8a9d-6174b2dfdada" volumeName="kubernetes.io/projected/b1440092-9901-4eff-8a9d-6174b2dfdada-bound-sa-token" seLinuxMountContext="" Oct 30 19:47:18.700260 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.699641 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8e3ed9d-5156-4698-82c3-df72fa728f94" volumeName="kubernetes.io/configmap/b8e3ed9d-5156-4698-82c3-df72fa728f94-config-volume" seLinuxMountContext="" Oct 30 19:47:18.700260 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.699657 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d78606e4-ccb9-421a-8ae5-8fbecbcd3831" volumeName="kubernetes.io/configmap/d78606e4-ccb9-421a-8ae5-8fbecbcd3831-service-ca-bundle" seLinuxMountContext="" Oct 30 19:47:18.700260 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.699668 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d808d253-3f30-48e8-918c-bd3880c2af8f" volumeName="kubernetes.io/configmap/d808d253-3f30-48e8-918c-bd3880c2af8f-iptables-alerter-script" seLinuxMountContext="" Oct 30 19:47:18.700260 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.699683 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da850825-59ab-480f-9047-afd7d2bdc19f" volumeName="kubernetes.io/projected/da850825-59ab-480f-9047-afd7d2bdc19f-kube-api-access-v26lp" seLinuxMountContext="" Oct 30 19:47:18.700260 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.699695 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08b5a0ca-cfe1-4366-a404-5ebbaeb7c149" volumeName="kubernetes.io/empty-dir/08b5a0ca-cfe1-4366-a404-5ebbaeb7c149-tmp-dir" seLinuxMountContext="" Oct 30 19:47:18.700260 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.699706 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1342ef67-5f63-47fb-a337-64c693e11228" volumeName="kubernetes.io/projected/1342ef67-5f63-47fb-a337-64c693e11228-kube-api-access-vj8ws" seLinuxMountContext="" Oct 30 19:47:18.700260 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.699724 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20f593c8-594e-4a64-9c5e-881311b269a3" volumeName="kubernetes.io/configmap/20f593c8-594e-4a64-9c5e-881311b269a3-telemetry-config" seLinuxMountContext="" Oct 30 19:47:18.700260 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.699735 5088 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a160cec7-bc1f-4202-8382-e84318122fd9" volumeName="kubernetes.io/projected/a160cec7-bc1f-4202-8382-e84318122fd9-kube-api-access-blstw" seLinuxMountContext="" Oct 30 19:47:18.700260 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.699746 5088 reconstruct.go:97] "Volume reconstruction finished" Oct 30 19:47:18.700260 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.699753 5088 reconciler.go:29] "Reconciler: start to sync state" Oct 30 19:47:18.716376 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.716353 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qdj4h_da850825-59ab-480f-9047-afd7d2bdc19f/console-operator/1.log" Oct 30 19:47:18.716518 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.716504 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qdj4h_da850825-59ab-480f-9047-afd7d2bdc19f/console-operator/1.log" Oct 30 19:47:18.722646 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.722600 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xpjpn_688d079a-932d-4f63-97da-5cd6222a03e1/ovn-acl-logging/0.log" Oct 30 19:47:18.723974 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.723956 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xpjpn_688d079a-932d-4f63-97da-5cd6222a03e1/ovn-acl-logging/0.log" Oct 30 19:47:18.729882 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.729858 5088 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 19:47:18.731225 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.731209 5088 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 19:47:18.731225 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.731227 5088 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 19:47:18.731355 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.731242 5088 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 19:47:18.731355 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:47:18.731272 5088 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 19:47:18.747273 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.747254 5088 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:47:18.747487 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.747341 5088 generic.go:356] "Generic (PLEG): container finished" podID="1afac97f-a617-4113-b8e8-c668caed54f9" containerID="6ee20aae29a5db678983275618de0ae7fe0d74c6ae8803a20db88485e40de6a5" exitCode=0 Oct 30 19:47:18.747562 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.747492 5088 generic.go:356] "Generic (PLEG): container finished" podID="1afac97f-a617-4113-b8e8-c668caed54f9" containerID="cdaea81198fd578e09a5dca061b4ee1b022cc960695a31c154722fbaffd10db3" exitCode=0 Oct 30 19:47:18.747562 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.747507 5088 generic.go:356] "Generic (PLEG): container finished" podID="1afac97f-a617-4113-b8e8-c668caed54f9" containerID="04eb863c3af79eeb76f7bb3fae710547ff4b46691a6e51bfa0d7eb1a4949da29" exitCode=0 Oct 30 19:47:18.747562 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.747520 5088 generic.go:356] "Generic (PLEG): container finished" podID="1afac97f-a617-4113-b8e8-c668caed54f9" containerID="6df040846437c3b32e997c88e7117f1b6d533797da284bc9405555b5da9859c0" exitCode=0 Oct 30 19:47:18.747562 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.747529 5088 generic.go:356] "Generic (PLEG): container finished" podID="1afac97f-a617-4113-b8e8-c668caed54f9" containerID="530b5dd05cf8c5aa916adc78ad54a0f0e8e0cde813e5ea127e032eefbe455a7e" exitCode=0 Oct 30 19:47:18.747562 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.747538 5088 generic.go:356] "Generic (PLEG): container finished" podID="1afac97f-a617-4113-b8e8-c668caed54f9" containerID="bc0bdcb4dbea76c6600166f5f294a2ad5cb60fe73d68f9f9a4a18a896a81f7b6" exitCode=0 Oct 30 19:47:18.747818 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.747578 5088 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 19:47:18.749125 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.749037 5088 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:47:18.751885 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.751869 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qdj4h_da850825-59ab-480f-9047-afd7d2bdc19f/console-operator/1.log" Oct 30 19:47:18.751984 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.751897 5088 generic.go:356] "Generic (PLEG): container finished" podID="da850825-59ab-480f-9047-afd7d2bdc19f" containerID="77f9a154bdfbbdcca31431c7016e021fc1250443b58b9925789af279a4b64a0f" exitCode=255 Oct 30 19:47:18.759825 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.759806 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xpjpn_688d079a-932d-4f63-97da-5cd6222a03e1/ovn-acl-logging/0.log" Oct 30 19:47:18.760221 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.760194 5088 generic.go:356] "Generic (PLEG): container finished" podID="688d079a-932d-4f63-97da-5cd6222a03e1" containerID="5fe9a2e1adf6d1e2dcd320f39f5481d5a67b40aae630fc540688ebcc8c1845e7" exitCode=1 Oct 30 19:47:18.762459 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.762435 5088 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:47:18.763927 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.763909 5088 generic.go:356] "Generic (PLEG): container finished" podID="6ad70715366d9db6ff14ef77cdc50313" containerID="c97983388b9679c12f0c64a9c0187402ee8ff80d0cacdb27704bcb35203c3e3b" exitCode=0 Oct 30 19:47:18.773641 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.773625 5088 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:47:18.793104 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.793057 5088 manager.go:324] Recovery completed Oct 30 19:47:18.815231 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.815204 5088 cpu_manager.go:226] "Starting" policy="none" Oct 30 19:47:18.815231 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.815223 5088 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 19:47:18.815378 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.815244 5088 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 19:47:18.815419 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.815391 5088 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 19:47:18.815859 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.815406 5088 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 19:47:18.815895 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.815861 5088 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 19:47:18.815895 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.815867 5088 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 19:47:18.815895 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.815873 5088 policy_none.go:50] "Start" Oct 30 19:47:18.815895 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.815882 5088 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 19:47:18.815895 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.815894 5088 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 19:47:18.816027 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.816019 5088 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 19:47:18.816027 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.816026 5088 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 19:47:18.816081 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.816031 5088 policy_none.go:44] "Start" Oct 30 19:47:18.820307 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.820283 5088 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 19:47:18.822922 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.822910 5088 manager.go:341] "Starting Device Plugin manager" Oct 30 19:47:18.822971 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:47:18.822938 5088 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 19:47:18.822971 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.822948 5088 server.go:87] "Starting device plugin registration server" Oct 30 19:47:18.823210 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.823200 5088 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 19:47:18.823245 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.823212 5088 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 19:47:18.823325 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.823307 5088 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 19:47:18.823436 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.823397 5088 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 19:47:18.823436 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.823435 5088 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 19:47:18.823586 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.823442 5088 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 19:47:18.823586 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.823492 5088 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:47:18.823424084Z","UUID":"06d87986-cd35-4d71-b7f6-db5341410a31","Handler":null,"Name":"","Endpoint":""} Oct 30 19:47:18.825473 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:47:18.825050 5088 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 19:47:18.826666 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.826647 5088 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 19:47:18.826751 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.826684 5088 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 19:47:18.833381 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.833340 5088 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal"] Oct 30 19:47:18.834100 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.834064 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" event={"ID":"6ad70715366d9db6ff14ef77cdc50313","Type":"ContainerStarted","Data":"1b4fdb1cc924a9f616cf519cedea812417a4457c52b0e475178d3f758a84bfd8"} Oct 30 19:47:18.834185 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.834113 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" event={"ID":"6ad70715366d9db6ff14ef77cdc50313","Type":"ContainerDied","Data":"c97983388b9679c12f0c64a9c0187402ee8ff80d0cacdb27704bcb35203c3e3b"} Oct 30 19:47:18.834185 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.834132 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" event={"ID":"6ad70715366d9db6ff14ef77cdc50313","Type":"ContainerStarted","Data":"23213dbd30843595194cc488e41405260a4e82391bf97b3fec6e67059dd7cdc8"} Oct 30 19:47:18.834185 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.834161 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal" event={"ID":"5b1dc20b8a7bd35e96a3f207cff8defc","Type":"ContainerStarted","Data":"6b4c6d4b4f23c6215e563503a440670518d6d93826c0c6fb7ac29bfbb0dbeb8d"} Oct 30 19:47:18.834185 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.834175 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal" event={"ID":"5b1dc20b8a7bd35e96a3f207cff8defc","Type":"ContainerStarted","Data":"8ab94cd9a89ec8b7e8f48c21784a2d597264034894e149a4d80accb109d73fb2"} Oct 30 19:47:18.834373 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.834198 5088 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" Oct 30 19:47:18.834837 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.834816 5088 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal" Oct 30 19:47:18.863068 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.863045 5088 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:47:18.863158 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:47:18.863102 5088 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-173.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal" Oct 30 19:47:18.864846 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.864829 5088 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:47:18.864906 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:47:18.864865 5088 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" Oct 30 19:47:18.923604 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.923574 5088 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:47:18.924773 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.924753 5088 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:47:18.924870 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.924790 5088 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:47:18.924870 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.924804 5088 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-173.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:47:18.924960 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.924873 5088 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-173.ec2.internal" Oct 30 19:47:18.948575 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.948550 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5b1dc20b8a7bd35e96a3f207cff8defc-config\") pod \"kube-apiserver-proxy-ip-10-0-132-173.ec2.internal\" (UID: \"5b1dc20b8a7bd35e96a3f207cff8defc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal" Oct 30 19:47:18.948685 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.948583 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6ad70715366d9db6ff14ef77cdc50313-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal\" (UID: \"6ad70715366d9db6ff14ef77cdc50313\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" Oct 30 19:47:18.948685 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.948629 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ad70715366d9db6ff14ef77cdc50313-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal\" (UID: \"6ad70715366d9db6ff14ef77cdc50313\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" Oct 30 19:47:18.963942 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.963922 5088 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-132-173.ec2.internal" Oct 30 19:47:18.964009 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:18.963981 5088 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-173.ec2.internal" Oct 30 19:47:19.049790 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.049729 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6ad70715366d9db6ff14ef77cdc50313-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal\" (UID: \"6ad70715366d9db6ff14ef77cdc50313\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" Oct 30 19:47:19.049790 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.049765 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ad70715366d9db6ff14ef77cdc50313-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal\" (UID: \"6ad70715366d9db6ff14ef77cdc50313\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" Oct 30 19:47:19.049790 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.049787 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5b1dc20b8a7bd35e96a3f207cff8defc-config\") pod \"kube-apiserver-proxy-ip-10-0-132-173.ec2.internal\" (UID: \"5b1dc20b8a7bd35e96a3f207cff8defc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal" Oct 30 19:47:19.049998 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.049802 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6ad70715366d9db6ff14ef77cdc50313-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal\" (UID: \"6ad70715366d9db6ff14ef77cdc50313\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" Oct 30 19:47:19.049998 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.049831 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5b1dc20b8a7bd35e96a3f207cff8defc-config\") pod \"kube-apiserver-proxy-ip-10-0-132-173.ec2.internal\" (UID: \"5b1dc20b8a7bd35e96a3f207cff8defc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal" Oct 30 19:47:19.049998 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.049835 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ad70715366d9db6ff14ef77cdc50313-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal\" (UID: \"6ad70715366d9db6ff14ef77cdc50313\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" Oct 30 19:47:19.679756 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.679724 5088 apiserver.go:52] "Watching apiserver" Oct 30 19:47:19.765198 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.765157 5088 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 19:47:19.766200 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.766166 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n75h8","openshift-multus/multus-76qp7","openshift-service-ca/service-ca-5fffc97bdf-pfwg8","openshift-cluster-node-tuning-operator/tuned-rggcq","openshift-multus/network-metrics-daemon-7f4jl","kube-system/global-pull-secret-syncer-hz6jn","kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95bqt","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn","openshift-insights/periodic-gathering-d7ptc-wwx7j","openshift-image-registry/image-registry-6796cf4b57-b9d8k","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xlgtn","openshift-network-diagnostics/network-check-source-6f78d59f99-k8mq4","openshift-service-ca-operator/service-ca-operator-544d749c8c-6wqfc","openshift-ingress/router-default-79458f7dbc-vbb28","openshift-network-console/networking-console-plugin-86c478bb88-tv4xh","openshift-dns/node-resolver-l9fpp","openshift-insights/insights-runtime-extractor-8txdr","openshift-network-diagnostics/network-check-target-kvw9t","openshift-kube-storage-version-migrator/migrator-66df5d496b-vdklw","openshift-monitoring/cluster-monitoring-operator-884576c57-qdv4s","openshift-network-operator/iptables-alerter-882qh","kube-system/kas-connection-checker-685895c7c9-kflff","openshift-dns/dns-default-ml5lp","openshift-image-registry/image-registry-5bc64fc876-25g7d","openshift-insights/insights-operator-7c4986d449-bnp7s","kube-system/kas-connection-checker-685895c7c9-r7djq","openshift-ingress-canary/ingress-canary-lk574","kube-system/konnectivity-agent-djkzp","openshift-image-registry/node-ca-hkvn6","openshift-multus/multus-additional-cni-plugins-2hfrk","kube-system/kas-connection-checker-685895c7c9-c7n9t","openshift-kube-storage-version-migrator/migrator-66df5d496b-b7sz6","openshift-ovn-kubernetes/ovnkube-node-xpjpn","openshift-console-operator/console-operator-7cd56b8dbc-qdj4h"] Oct 30 19:47:19.769778 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.769741 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qdv4s" Oct 30 19:47:19.770116 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.770093 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:19.771303 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.771278 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:19.771666 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.771648 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n75h8" Oct 30 19:47:19.773514 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.773496 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-tv4xh" Oct 30 19:47:19.773930 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.773911 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:19.777280 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.777262 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7f4jl" Oct 30 19:47:19.781020 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.781003 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ml5lp" Oct 30 19:47:19.783154 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.783139 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8txdr" Oct 30 19:47:19.784486 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.784469 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 19:47:19.784583 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.784572 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 19:47:19.787312 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.787295 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 19:47:19.787586 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.787571 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 19:47:19.787900 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.787882 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:47:19.787900 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.787894 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 19:47:19.788662 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.788640 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 19:47:19.788914 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.788894 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 19:47:19.791982 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.791963 5088 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" Oct 30 19:47:19.792792 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.792772 5088 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal" Oct 30 19:47:19.794271 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.794256 5088 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 19:47:19.801602 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.801564 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 19:47:19.801602 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.801564 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 19:47:19.801602 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.801595 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 19:47:19.801602 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.801572 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 19:47:19.801807 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.801577 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 19:47:19.801807 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.801563 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 19:47:19.801807 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.801674 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 19:47:19.801932 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.801918 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 19:47:19.801970 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.801933 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 19:47:19.801999 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.801992 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 19:47:19.802455 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.802443 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 19:47:19.802694 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.802682 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 19:47:19.802832 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.802817 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 19:47:19.803041 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.803022 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 19:47:19.803132 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.803042 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 19:47:19.803132 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.803071 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 19:47:19.803132 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.803085 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 19:47:19.803132 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.803107 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 19:47:19.803132 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.803113 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 19:47:19.803132 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.803133 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 19:47:19.803417 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.803140 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 19:47:19.803417 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.803147 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 19:47:19.803417 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.803116 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 19:47:19.803417 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.803151 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:47:19.803417 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.803183 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 19:47:19.803417 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.803202 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 19:47:19.803417 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.803208 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 19:47:19.803417 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.803219 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 19:47:19.803417 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.803230 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 19:47:19.803417 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.803235 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 19:47:19.803417 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.803253 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 19:47:19.803417 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.803245 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:47:19.803417 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.803263 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:47:19.803417 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.803369 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 19:47:19.803846 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.803450 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 19:47:19.803846 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.803468 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 19:47:19.803846 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.803495 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-n5k26\"" Oct 30 19:47:19.803846 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.803567 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 19:47:19.803846 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.803602 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-wnngx\"" Oct 30 19:47:19.803846 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.803651 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qlg5x\"" Oct 30 19:47:19.803846 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.803714 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 19:47:19.803846 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.803799 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 19:47:19.803846 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.803805 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2gzs4\"" Oct 30 19:47:19.804101 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.803863 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cscqg\"" Oct 30 19:47:19.804101 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.803956 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 19:47:19.804299 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.804279 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 19:47:19.813400 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.813381 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4vv2n\"" Oct 30 19:47:19.813400 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.813395 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 19:47:19.813523 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.813398 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 19:47:19.815331 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.815308 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 19:47:19.815412 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.815352 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 19:47:19.815412 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.815317 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-97cb9\"" Oct 30 19:47:19.815412 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.815359 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 19:47:19.815412 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.815393 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-grfkh\"" Oct 30 19:47:19.815412 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.815409 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 19:47:19.815648 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.815414 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 19:47:19.815648 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.815419 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 19:47:19.815648 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.815528 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 19:47:19.815648 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.815630 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 19:47:19.815889 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.815685 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 19:47:19.815889 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.815691 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 19:47:19.815889 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.815698 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 19:47:19.815889 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.815705 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 19:47:19.815889 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.815714 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:47:19.815889 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.815698 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-d5xr8\"" Oct 30 19:47:19.815889 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.815748 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-mbspd\"" Oct 30 19:47:19.815889 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.815754 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 19:47:19.815889 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.815752 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 19:47:19.815889 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.815767 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:47:19.815889 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.815812 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-9xs5t\"" Oct 30 19:47:19.815889 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.815839 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 19:47:19.815889 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.815887 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 19:47:19.816386 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.816051 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 19:47:19.816386 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.816074 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-l5g68\"" Oct 30 19:47:19.816386 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.816104 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 19:47:19.816386 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.816184 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 19:47:19.816386 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.816263 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 19:47:19.816386 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.816272 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 19:47:19.816386 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.816299 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-bvhff\"" Oct 30 19:47:19.816386 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.816305 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 19:47:19.816386 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.816312 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 19:47:19.816386 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.816266 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 19:47:19.816386 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.816258 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-h4pgj\"" Oct 30 19:47:19.816386 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.816306 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 19:47:19.816386 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.816350 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-h5z6r\"" Oct 30 19:47:19.816386 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.816371 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 19:47:19.816386 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.816377 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 19:47:19.816386 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.816393 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:47:19.816932 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.816410 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 19:47:19.816932 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.816418 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 19:47:19.816932 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.816419 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 19:47:19.816932 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.816434 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-xzffq\"" Oct 30 19:47:19.816932 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.816459 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 19:47:19.816932 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.816550 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 19:47:19.816932 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.816648 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wgtz9\"" Oct 30 19:47:19.816932 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.816737 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-tvr85\"" Oct 30 19:47:19.816932 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.816807 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q7g4j\"" Oct 30 19:47:19.828723 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.828701 5088 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:47:19.828816 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:47:19.828764 5088 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-173.ec2.internal" Oct 30 19:47:19.828816 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.828701 5088 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:47:19.828893 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:47:19.828832 5088 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-173.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-173.ec2.internal" Oct 30 19:47:19.853400 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.853363 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d78606e4-ccb9-421a-8ae5-8fbecbcd3831-service-ca-bundle\") pod \"insights-operator-7c4986d449-bnp7s\" (UID: \"d78606e4-ccb9-421a-8ae5-8fbecbcd3831\") " pod="openshift-insights/insights-operator-7c4986d449-bnp7s" Oct 30 19:47:19.853566 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.853410 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-etc-sysctl-conf\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:47:19.853566 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.853434 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-tmp\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:47:19.853566 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.853542 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-tmp\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:47:19.853814 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.853557 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-host-run-netns\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.853814 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.853600 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/688d079a-932d-4f63-97da-5cd6222a03e1-ovn-node-metrics-cert\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.853814 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.853651 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljrnk\" (UniqueName: \"kubernetes.io/projected/d78606e4-ccb9-421a-8ae5-8fbecbcd3831-kube-api-access-ljrnk\") pod \"insights-operator-7c4986d449-bnp7s\" (UID: \"d78606e4-ccb9-421a-8ae5-8fbecbcd3831\") " pod="openshift-insights/insights-operator-7c4986d449-bnp7s" Oct 30 19:47:19.853814 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.853687 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tng88\" (UniqueName: \"kubernetes.io/projected/7a570505-5193-4564-b776-0f83df60d4e9-kube-api-access-tng88\") pod \"migrator-66df5d496b-vdklw\" (UID: \"7a570505-5193-4564-b776-0f83df60d4e9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vdklw" Oct 30 19:47:19.853814 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.853725 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg9ks\" (UniqueName: \"kubernetes.io/projected/aab2acf3-4af5-4d2e-8336-05b01a2dca1e-kube-api-access-lg9ks\") pod \"kas-connection-checker-685895c7c9-kflff\" (UID: \"aab2acf3-4af5-4d2e-8336-05b01a2dca1e\") " pod="kube-system/kas-connection-checker-685895c7c9-kflff" Oct 30 19:47:19.853814 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.853751 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66999038-807b-406b-9242-c68c04e521f3-cni-binary-copy\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.853814 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.853777 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-registry-tls\") pod \"image-registry-5bc64fc876-25g7d\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:19.853814 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.853804 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/20f593c8-594e-4a64-9c5e-881311b269a3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qdv4s\" (UID: \"20f593c8-594e-4a64-9c5e-881311b269a3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qdv4s" Oct 30 19:47:19.854178 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.853832 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a421de2b-dd41-4862-9849-0a4f0256af1b-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-95bqt\" (UID: \"a421de2b-dd41-4862-9849-0a4f0256af1b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95bqt" Oct 30 19:47:19.854178 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.853879 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b8e3ed9d-5156-4698-82c3-df72fa728f94-config-volume\") pod \"dns-default-ml5lp\" (UID: \"b8e3ed9d-5156-4698-82c3-df72fa728f94\") " pod="openshift-dns/dns-default-ml5lp" Oct 30 19:47:19.854178 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.853916 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8e3ed9d-5156-4698-82c3-df72fa728f94-metrics-tls\") pod \"dns-default-ml5lp\" (UID: \"b8e3ed9d-5156-4698-82c3-df72fa728f94\") " pod="openshift-dns/dns-default-ml5lp" Oct 30 19:47:19.854178 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.853948 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr5dn\" (UniqueName: \"kubernetes.io/projected/1afac97f-a617-4113-b8e8-c668caed54f9-kube-api-access-nr5dn\") pod \"multus-additional-cni-plugins-2hfrk\" (UID: \"1afac97f-a617-4113-b8e8-c668caed54f9\") " pod="openshift-multus/multus-additional-cni-plugins-2hfrk" Oct 30 19:47:19.854178 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.853974 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d1dca14b-4840-4dd6-a7cd-7d795adb9532-kubelet-dir\") pod \"aws-ebs-csi-driver-node-glksn\" (UID: \"d1dca14b-4840-4dd6-a7cd-7d795adb9532\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" Oct 30 19:47:19.854178 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.853999 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b1440092-9901-4eff-8a9d-6174b2dfdada-trusted-ca\") pod \"image-registry-6796cf4b57-b9d8k\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:19.854178 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.854022 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-bound-sa-token\") pod \"image-registry-5bc64fc876-25g7d\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:19.854178 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.854048 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/afb6fa1c-9722-42c4-a914-4941bcbcaaad-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-tv4xh\" (UID: \"afb6fa1c-9722-42c4-a914-4941bcbcaaad\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-tv4xh" Oct 30 19:47:19.854178 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.854073 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/da850825-59ab-480f-9047-afd7d2bdc19f-config\") pod \"console-operator-7cd56b8dbc-qdj4h\" (UID: \"da850825-59ab-480f-9047-afd7d2bdc19f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qdj4h" Oct 30 19:47:19.854178 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.854085 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d78606e4-ccb9-421a-8ae5-8fbecbcd3831-service-ca-bundle\") pod \"insights-operator-7c4986d449-bnp7s\" (UID: \"d78606e4-ccb9-421a-8ae5-8fbecbcd3831\") " pod="openshift-insights/insights-operator-7c4986d449-bnp7s" Oct 30 19:47:19.854649 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.854183 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66999038-807b-406b-9242-c68c04e521f3-cni-binary-copy\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.854649 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.854277 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/da850825-59ab-480f-9047-afd7d2bdc19f-config\") pod \"console-operator-7cd56b8dbc-qdj4h\" (UID: \"da850825-59ab-480f-9047-afd7d2bdc19f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qdj4h" Oct 30 19:47:19.854649 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.854307 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/688d079a-932d-4f63-97da-5cd6222a03e1-ovn-node-metrics-cert\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.854649 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.854318 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvcvq\" (UniqueName: \"kubernetes.io/projected/0547f118-5c20-403a-beca-c6a5a671a1ff-kube-api-access-vvcvq\") pod \"migrator-66df5d496b-b7sz6\" (UID: \"0547f118-5c20-403a-beca-c6a5a671a1ff\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b7sz6" Oct 30 19:47:19.854649 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.854357 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/73bb0a1f-597a-4078-8a01-20d0ed24715b-dbus\") pod \"global-pull-secret-syncer-hz6jn\" (UID: \"73bb0a1f-597a-4078-8a01-20d0ed24715b\") " pod="kube-system/global-pull-secret-syncer-hz6jn" Oct 30 19:47:19.854649 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.854325 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b8e3ed9d-5156-4698-82c3-df72fa728f94-config-volume\") pod \"dns-default-ml5lp\" (UID: \"b8e3ed9d-5156-4698-82c3-df72fa728f94\") " pod="openshift-dns/dns-default-ml5lp" Oct 30 19:47:19.854649 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.854396 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b1440092-9901-4eff-8a9d-6174b2dfdada-installation-pull-secrets\") pod \"image-registry-6796cf4b57-b9d8k\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:19.854649 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.854450 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkbkr\" (UniqueName: \"kubernetes.io/projected/2962e3da-8439-4316-9c3a-1679842609b7-kube-api-access-rkbkr\") pod \"kas-connection-checker-685895c7c9-r7djq\" (UID: \"2962e3da-8439-4316-9c3a-1679842609b7\") " pod="kube-system/kas-connection-checker-685895c7c9-r7djq" Oct 30 19:47:19.854649 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.854511 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-host-cni-netd\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.854649 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.854567 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/005bece8-8d37-4a58-954c-0ae3dab9dadc-service-ca-bundle\") pod \"periodic-gathering-d7ptc-wwx7j\" (UID: \"005bece8-8d37-4a58-954c-0ae3dab9dadc\") " pod="openshift-insights/periodic-gathering-d7ptc-wwx7j" Oct 30 19:47:19.854649 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.854597 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtqp8\" (UniqueName: \"kubernetes.io/projected/d1dca14b-4840-4dd6-a7cd-7d795adb9532-kube-api-access-xtqp8\") pod \"aws-ebs-csi-driver-node-glksn\" (UID: \"d1dca14b-4840-4dd6-a7cd-7d795adb9532\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" Oct 30 19:47:19.855152 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.854668 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5ad8f4b0-50ba-4191-8df4-25930717acb0-cert\") pod \"ingress-canary-lk574\" (UID: \"5ad8f4b0-50ba-4191-8df4-25930717acb0\") " pod="openshift-ingress-canary/ingress-canary-lk574" Oct 30 19:47:19.855152 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.854672 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b1440092-9901-4eff-8a9d-6174b2dfdada-installation-pull-secrets\") pod \"image-registry-6796cf4b57-b9d8k\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:19.855152 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.854700 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/41534930-917f-44ec-bccd-86d6c6fc7c70-default-certificate\") pod \"router-default-79458f7dbc-vbb28\" (UID: \"41534930-917f-44ec-bccd-86d6c6fc7c70\") " pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:19.855152 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.854728 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/688d079a-932d-4f63-97da-5cd6222a03e1-env-overrides\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.855152 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.854755 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d78606e4-ccb9-421a-8ae5-8fbecbcd3831-snapshots\") pod \"insights-operator-7c4986d449-bnp7s\" (UID: \"d78606e4-ccb9-421a-8ae5-8fbecbcd3831\") " pod="openshift-insights/insights-operator-7c4986d449-bnp7s" Oct 30 19:47:19.855152 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.854782 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d78606e4-ccb9-421a-8ae5-8fbecbcd3831-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-bnp7s\" (UID: \"d78606e4-ccb9-421a-8ae5-8fbecbcd3831\") " pod="openshift-insights/insights-operator-7c4986d449-bnp7s" Oct 30 19:47:19.855152 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.854805 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfbsr\" (UniqueName: \"kubernetes.io/projected/688d079a-932d-4f63-97da-5cd6222a03e1-kube-api-access-tfbsr\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.855152 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.854835 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-registry-certificates\") pod \"image-registry-5bc64fc876-25g7d\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:19.855152 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.854862 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-trusted-ca\") pod \"image-registry-5bc64fc876-25g7d\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:19.855152 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.854889 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxq6z\" (UniqueName: \"kubernetes.io/projected/5ad8f4b0-50ba-4191-8df4-25930717acb0-kube-api-access-lxq6z\") pod \"ingress-canary-lk574\" (UID: \"5ad8f4b0-50ba-4191-8df4-25930717acb0\") " pod="openshift-ingress-canary/ingress-canary-lk574" Oct 30 19:47:19.855152 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.854890 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5ad8f4b0-50ba-4191-8df4-25930717acb0-cert\") pod \"ingress-canary-lk574\" (UID: \"5ad8f4b0-50ba-4191-8df4-25930717acb0\") " pod="openshift-ingress-canary/ingress-canary-lk574" Oct 30 19:47:19.855152 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.854926 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68nwv\" (UniqueName: \"kubernetes.io/projected/005bece8-8d37-4a58-954c-0ae3dab9dadc-kube-api-access-68nwv\") pod \"periodic-gathering-d7ptc-wwx7j\" (UID: \"005bece8-8d37-4a58-954c-0ae3dab9dadc\") " pod="openshift-insights/periodic-gathering-d7ptc-wwx7j" Oct 30 19:47:19.855152 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.854955 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/74b7c739-483f-4756-b97b-e65d37387b17-serving-cert\") pod \"service-ca-operator-544d749c8c-6wqfc\" (UID: \"74b7c739-483f-4756-b97b-e65d37387b17\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6wqfc" Oct 30 19:47:19.855152 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.854956 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/41534930-917f-44ec-bccd-86d6c6fc7c70-default-certificate\") pod \"router-default-79458f7dbc-vbb28\" (UID: \"41534930-917f-44ec-bccd-86d6c6fc7c70\") " pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:19.855152 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.854982 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-etc-kubernetes\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:47:19.855152 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.855028 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-host-var-lib-kubelet\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.855152 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.855034 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b1440092-9901-4eff-8a9d-6174b2dfdada-trusted-ca\") pod \"image-registry-6796cf4b57-b9d8k\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:19.855152 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.855044 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/688d079a-932d-4f63-97da-5cd6222a03e1-env-overrides\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.855152 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.855053 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-var-lib-openvswitch\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.855152 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.855078 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-node-log\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.855152 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.855149 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d78606e4-ccb9-421a-8ae5-8fbecbcd3831-snapshots\") pod \"insights-operator-7c4986d449-bnp7s\" (UID: \"d78606e4-ccb9-421a-8ae5-8fbecbcd3831\") " pod="openshift-insights/insights-operator-7c4986d449-bnp7s" Oct 30 19:47:19.856054 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.855182 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-registry-certificates\") pod \"image-registry-5bc64fc876-25g7d\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:19.856054 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.855194 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1afac97f-a617-4113-b8e8-c668caed54f9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2hfrk\" (UID: \"1afac97f-a617-4113-b8e8-c668caed54f9\") " pod="openshift-multus/multus-additional-cni-plugins-2hfrk" Oct 30 19:47:19.856054 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.855202 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/005bece8-8d37-4a58-954c-0ae3dab9dadc-service-ca-bundle\") pod \"periodic-gathering-d7ptc-wwx7j\" (UID: \"005bece8-8d37-4a58-954c-0ae3dab9dadc\") " pod="openshift-insights/periodic-gathering-d7ptc-wwx7j" Oct 30 19:47:19.856054 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.855224 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a421de2b-dd41-4862-9849-0a4f0256af1b-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-95bqt\" (UID: \"a421de2b-dd41-4862-9849-0a4f0256af1b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95bqt" Oct 30 19:47:19.856054 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.855343 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d78606e4-ccb9-421a-8ae5-8fbecbcd3831-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-bnp7s\" (UID: \"d78606e4-ccb9-421a-8ae5-8fbecbcd3831\") " pod="openshift-insights/insights-operator-7c4986d449-bnp7s" Oct 30 19:47:19.856054 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.855375 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a421de2b-dd41-4862-9849-0a4f0256af1b-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-95bqt\" (UID: \"a421de2b-dd41-4862-9849-0a4f0256af1b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95bqt" Oct 30 19:47:19.856054 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.855394 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-system-cni-dir\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.856054 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.855400 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/74b7c739-483f-4756-b97b-e65d37387b17-serving-cert\") pod \"service-ca-operator-544d749c8c-6wqfc\" (UID: \"74b7c739-483f-4756-b97b-e65d37387b17\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6wqfc" Oct 30 19:47:19.856054 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.855424 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv4p7\" (UniqueName: \"kubernetes.io/projected/9b6dd02b-d4bd-4959-9a06-8941d6baf1a5-kube-api-access-sv4p7\") pod \"volume-data-source-validator-58c9f5f46d-xlgtn\" (UID: \"9b6dd02b-d4bd-4959-9a06-8941d6baf1a5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xlgtn" Oct 30 19:47:19.856054 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.855451 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57ddd2a0-f081-4f90-95a6-0f6f8df2f433-metrics-certs\") pod \"network-metrics-daemon-7f4jl\" (UID: \"57ddd2a0-f081-4f90-95a6-0f6f8df2f433\") " pod="openshift-multus/network-metrics-daemon-7f4jl" Oct 30 19:47:19.856054 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.855479 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrg8k\" (UniqueName: \"kubernetes.io/projected/a421de2b-dd41-4862-9849-0a4f0256af1b-kube-api-access-rrg8k\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-95bqt\" (UID: \"a421de2b-dd41-4862-9849-0a4f0256af1b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95bqt" Oct 30 19:47:19.856054 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.855515 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1afac97f-a617-4113-b8e8-c668caed54f9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2hfrk\" (UID: \"1afac97f-a617-4113-b8e8-c668caed54f9\") " pod="openshift-multus/multus-additional-cni-plugins-2hfrk" Oct 30 19:47:19.856054 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.855517 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-host-run-netns\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.856054 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.855517 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-trusted-ca\") pod \"image-registry-5bc64fc876-25g7d\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:19.856054 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.855656 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b1440092-9901-4eff-8a9d-6174b2dfdada-ca-trust-extracted\") pod \"image-registry-6796cf4b57-b9d8k\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:19.856054 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.855715 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/afb6fa1c-9722-42c4-a914-4941bcbcaaad-nginx-conf\") pod \"networking-console-plugin-86c478bb88-tv4xh\" (UID: \"afb6fa1c-9722-42c4-a914-4941bcbcaaad\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-tv4xh" Oct 30 19:47:19.856054 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.855751 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a421de2b-dd41-4862-9849-0a4f0256af1b-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-95bqt\" (UID: \"a421de2b-dd41-4862-9849-0a4f0256af1b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95bqt" Oct 30 19:47:19.856054 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.855754 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b1440092-9901-4eff-8a9d-6174b2dfdada-ca-trust-extracted\") pod \"image-registry-6796cf4b57-b9d8k\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:19.856054 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.855774 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da850825-59ab-480f-9047-afd7d2bdc19f-trusted-ca\") pod \"console-operator-7cd56b8dbc-qdj4h\" (UID: \"da850825-59ab-480f-9047-afd7d2bdc19f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qdj4h" Oct 30 19:47:19.856054 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.855876 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d1dca14b-4840-4dd6-a7cd-7d795adb9532-registration-dir\") pod \"aws-ebs-csi-driver-node-glksn\" (UID: \"d1dca14b-4840-4dd6-a7cd-7d795adb9532\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" Oct 30 19:47:19.856054 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.855928 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d808d253-3f30-48e8-918c-bd3880c2af8f-iptables-alerter-script\") pod \"iptables-alerter-882qh\" (UID: \"d808d253-3f30-48e8-918c-bd3880c2af8f\") " pod="openshift-network-operator/iptables-alerter-882qh" Oct 30 19:47:19.856054 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.855946 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/afb6fa1c-9722-42c4-a914-4941bcbcaaad-nginx-conf\") pod \"networking-console-plugin-86c478bb88-tv4xh\" (UID: \"afb6fa1c-9722-42c4-a914-4941bcbcaaad\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-tv4xh" Oct 30 19:47:19.856054 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.855976 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-host-cni-bin\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.856054 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.855990 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da850825-59ab-480f-9047-afd7d2bdc19f-trusted-ca\") pod \"console-operator-7cd56b8dbc-qdj4h\" (UID: \"da850825-59ab-480f-9047-afd7d2bdc19f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qdj4h" Oct 30 19:47:19.856054 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.856006 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1afac97f-a617-4113-b8e8-c668caed54f9-system-cni-dir\") pod \"multus-additional-cni-plugins-2hfrk\" (UID: \"1afac97f-a617-4113-b8e8-c668caed54f9\") " pod="openshift-multus/multus-additional-cni-plugins-2hfrk" Oct 30 19:47:19.856054 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.856040 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73bb0a1f-597a-4078-8a01-20d0ed24715b-original-pull-secret\") pod \"global-pull-secret-syncer-hz6jn\" (UID: \"73bb0a1f-597a-4078-8a01-20d0ed24715b\") " pod="kube-system/global-pull-secret-syncer-hz6jn" Oct 30 19:47:19.857006 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.856090 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b1440092-9901-4eff-8a9d-6174b2dfdada-bound-sa-token\") pod \"image-registry-6796cf4b57-b9d8k\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:19.857006 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.856158 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/66999038-807b-406b-9242-c68c04e521f3-multus-daemon-config\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.857006 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.856184 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d808d253-3f30-48e8-918c-bd3880c2af8f-iptables-alerter-script\") pod \"iptables-alerter-882qh\" (UID: \"d808d253-3f30-48e8-918c-bd3880c2af8f\") " pod="openshift-network-operator/iptables-alerter-882qh" Oct 30 19:47:19.857006 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.856239 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8e3ed9d-5156-4698-82c3-df72fa728f94-tmp-dir\") pod \"dns-default-ml5lp\" (UID: \"b8e3ed9d-5156-4698-82c3-df72fa728f94\") " pod="openshift-dns/dns-default-ml5lp" Oct 30 19:47:19.857006 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.856270 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtksm\" (UniqueName: \"kubernetes.io/projected/b8e3ed9d-5156-4698-82c3-df72fa728f94-kube-api-access-xtksm\") pod \"dns-default-ml5lp\" (UID: \"b8e3ed9d-5156-4698-82c3-df72fa728f94\") " pod="openshift-dns/dns-default-ml5lp" Oct 30 19:47:19.857006 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.856291 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73bb0a1f-597a-4078-8a01-20d0ed24715b-original-pull-secret\") pod \"global-pull-secret-syncer-hz6jn\" (UID: \"73bb0a1f-597a-4078-8a01-20d0ed24715b\") " pod="kube-system/global-pull-secret-syncer-hz6jn" Oct 30 19:47:19.857006 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.856298 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-run-openvswitch\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.857006 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.856323 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/66999038-807b-406b-9242-c68c04e521f3-multus-daemon-config\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.857006 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.856326 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1afac97f-a617-4113-b8e8-c668caed54f9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2hfrk\" (UID: \"1afac97f-a617-4113-b8e8-c668caed54f9\") " pod="openshift-multus/multus-additional-cni-plugins-2hfrk" Oct 30 19:47:19.857006 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.856356 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-image-registry-private-configuration\") pod \"image-registry-5bc64fc876-25g7d\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:19.857006 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.856363 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8e3ed9d-5156-4698-82c3-df72fa728f94-tmp-dir\") pod \"dns-default-ml5lp\" (UID: \"b8e3ed9d-5156-4698-82c3-df72fa728f94\") " pod="openshift-dns/dns-default-ml5lp" Oct 30 19:47:19.857006 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.856407 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94r98\" (UniqueName: \"kubernetes.io/projected/2e03cb52-fd85-4e35-87b4-38833cbc7d74-kube-api-access-94r98\") pod \"node-ca-hkvn6\" (UID: \"2e03cb52-fd85-4e35-87b4-38833cbc7d74\") " pod="openshift-image-registry/node-ca-hkvn6" Oct 30 19:47:19.857006 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.856439 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-etc-modprobe-d\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:47:19.857006 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.856461 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-etc-sysconfig\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:47:19.857006 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.856500 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-host-var-lib-cni-bin\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.857006 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.856510 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1afac97f-a617-4113-b8e8-c668caed54f9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2hfrk\" (UID: \"1afac97f-a617-4113-b8e8-c668caed54f9\") " pod="openshift-multus/multus-additional-cni-plugins-2hfrk" Oct 30 19:47:19.857006 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.856548 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4h6p\" (UniqueName: \"kubernetes.io/projected/d71659c8-f08f-4221-a160-56ac78bd8473-kube-api-access-s4h6p\") pod \"service-ca-5fffc97bdf-pfwg8\" (UID: \"d71659c8-f08f-4221-a160-56ac78bd8473\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pfwg8" Oct 30 19:47:19.857006 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.856579 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-image-registry-private-configuration\") pod \"image-registry-5bc64fc876-25g7d\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:19.857006 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.856652 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-ca-trust-extracted\") pod \"image-registry-5bc64fc876-25g7d\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:19.857006 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.856681 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj8ws\" (UniqueName: \"kubernetes.io/projected/1342ef67-5f63-47fb-a337-64c693e11228-kube-api-access-vj8ws\") pod \"cluster-samples-operator-6ddb6c8b94-n75h8\" (UID: \"1342ef67-5f63-47fb-a337-64c693e11228\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n75h8" Oct 30 19:47:19.857006 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.856707 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d1dca14b-4840-4dd6-a7cd-7d795adb9532-etc-selinux\") pod \"aws-ebs-csi-driver-node-glksn\" (UID: \"d1dca14b-4840-4dd6-a7cd-7d795adb9532\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" Oct 30 19:47:19.857006 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.856732 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-run\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:47:19.857006 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.856756 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-ca-trust-extracted\") pod \"image-registry-5bc64fc876-25g7d\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:19.857006 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.856787 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-host-run-multus-certs\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.857006 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.856816 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-host-run-ovn-kubernetes\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.857006 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.856854 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb8sb\" (UniqueName: \"kubernetes.io/projected/57ddd2a0-f081-4f90-95a6-0f6f8df2f433-kube-api-access-wb8sb\") pod \"network-metrics-daemon-7f4jl\" (UID: \"57ddd2a0-f081-4f90-95a6-0f6f8df2f433\") " pod="openshift-multus/network-metrics-daemon-7f4jl" Oct 30 19:47:19.857006 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.856890 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/005bece8-8d37-4a58-954c-0ae3dab9dadc-archives-path\") pod \"periodic-gathering-d7ptc-wwx7j\" (UID: \"005bece8-8d37-4a58-954c-0ae3dab9dadc\") " pod="openshift-insights/periodic-gathering-d7ptc-wwx7j" Oct 30 19:47:19.857898 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.857020 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d1dca14b-4840-4dd6-a7cd-7d795adb9532-sys-fs\") pod \"aws-ebs-csi-driver-node-glksn\" (UID: \"d1dca14b-4840-4dd6-a7cd-7d795adb9532\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" Oct 30 19:47:19.857898 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.857041 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/005bece8-8d37-4a58-954c-0ae3dab9dadc-archives-path\") pod \"periodic-gathering-d7ptc-wwx7j\" (UID: \"005bece8-8d37-4a58-954c-0ae3dab9dadc\") " pod="openshift-insights/periodic-gathering-d7ptc-wwx7j" Oct 30 19:47:19.857898 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.857092 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49lpb\" (UniqueName: \"kubernetes.io/projected/08b5a0ca-cfe1-4366-a404-5ebbaeb7c149-kube-api-access-49lpb\") pod \"node-resolver-l9fpp\" (UID: \"08b5a0ca-cfe1-4366-a404-5ebbaeb7c149\") " pod="openshift-dns/node-resolver-l9fpp" Oct 30 19:47:19.857898 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.857115 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-host-run-k8s-cni-cncf-io\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.857898 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.857470 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkmdz\" (UniqueName: \"kubernetes.io/projected/d808d253-3f30-48e8-918c-bd3880c2af8f-kube-api-access-mkmdz\") pod \"iptables-alerter-882qh\" (UID: \"d808d253-3f30-48e8-918c-bd3880c2af8f\") " pod="openshift-network-operator/iptables-alerter-882qh" Oct 30 19:47:19.857898 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.857516 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-host-kubelet\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.857898 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.857555 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1342ef67-5f63-47fb-a337-64c693e11228-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-n75h8\" (UID: \"1342ef67-5f63-47fb-a337-64c693e11228\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n75h8" Oct 30 19:47:19.857898 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.857588 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41534930-917f-44ec-bccd-86d6c6fc7c70-metrics-certs\") pod \"router-default-79458f7dbc-vbb28\" (UID: \"41534930-917f-44ec-bccd-86d6c6fc7c70\") " pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:19.857898 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.857631 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a160cec7-bc1f-4202-8382-e84318122fd9-crio-socket\") pod \"insights-runtime-extractor-8txdr\" (UID: \"a160cec7-bc1f-4202-8382-e84318122fd9\") " pod="openshift-insights/insights-runtime-extractor-8txdr" Oct 30 19:47:19.857898 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.857895 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1342ef67-5f63-47fb-a337-64c693e11228-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-n75h8\" (UID: \"1342ef67-5f63-47fb-a337-64c693e11228\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n75h8" Oct 30 19:47:19.858305 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.857914 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08b5a0ca-cfe1-4366-a404-5ebbaeb7c149-tmp-dir\") pod \"node-resolver-l9fpp\" (UID: \"08b5a0ca-cfe1-4366-a404-5ebbaeb7c149\") " pod="openshift-dns/node-resolver-l9fpp" Oct 30 19:47:19.858305 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.857948 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d808d253-3f30-48e8-918c-bd3880c2af8f-host-slash\") pod \"iptables-alerter-882qh\" (UID: \"d808d253-3f30-48e8-918c-bd3880c2af8f\") " pod="openshift-network-operator/iptables-alerter-882qh" Oct 30 19:47:19.858305 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.857982 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e03cb52-fd85-4e35-87b4-38833cbc7d74-host\") pod \"node-ca-hkvn6\" (UID: \"2e03cb52-fd85-4e35-87b4-38833cbc7d74\") " pod="openshift-image-registry/node-ca-hkvn6" Oct 30 19:47:19.858305 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.858015 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41534930-917f-44ec-bccd-86d6c6fc7c70-service-ca-bundle\") pod \"router-default-79458f7dbc-vbb28\" (UID: \"41534930-917f-44ec-bccd-86d6c6fc7c70\") " pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:19.858305 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.858036 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08b5a0ca-cfe1-4366-a404-5ebbaeb7c149-tmp-dir\") pod \"node-resolver-l9fpp\" (UID: \"08b5a0ca-cfe1-4366-a404-5ebbaeb7c149\") " pod="openshift-dns/node-resolver-l9fpp" Oct 30 19:47:19.858305 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.858052 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v26lp\" (UniqueName: \"kubernetes.io/projected/da850825-59ab-480f-9047-afd7d2bdc19f-kube-api-access-v26lp\") pod \"console-operator-7cd56b8dbc-qdj4h\" (UID: \"da850825-59ab-480f-9047-afd7d2bdc19f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qdj4h" Oct 30 19:47:19.858305 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.858081 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/41534930-917f-44ec-bccd-86d6c6fc7c70-stats-auth\") pod \"router-default-79458f7dbc-vbb28\" (UID: \"41534930-917f-44ec-bccd-86d6c6fc7c70\") " pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:19.858305 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.858113 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x59q2\" (UniqueName: \"kubernetes.io/projected/41534930-917f-44ec-bccd-86d6c6fc7c70-kube-api-access-x59q2\") pod \"router-default-79458f7dbc-vbb28\" (UID: \"41534930-917f-44ec-bccd-86d6c6fc7c70\") " pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:19.858305 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.858143 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-sys\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:47:19.858305 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.858200 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-run-ovn\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.858305 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.858233 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/688d079a-932d-4f63-97da-5cd6222a03e1-ovnkube-script-lib\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.858305 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.858259 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d78606e4-ccb9-421a-8ae5-8fbecbcd3831-serving-cert\") pod \"insights-operator-7c4986d449-bnp7s\" (UID: \"d78606e4-ccb9-421a-8ae5-8fbecbcd3831\") " pod="openshift-insights/insights-operator-7c4986d449-bnp7s" Oct 30 19:47:19.858305 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.858289 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d71659c8-f08f-4221-a160-56ac78bd8473-signing-key\") pod \"service-ca-5fffc97bdf-pfwg8\" (UID: \"d71659c8-f08f-4221-a160-56ac78bd8473\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pfwg8" Oct 30 19:47:19.858873 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.858321 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-multus-cni-dir\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.858873 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.858588 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/688d079a-932d-4f63-97da-5cd6222a03e1-ovnkube-script-lib\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.858873 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.858593 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d71659c8-f08f-4221-a160-56ac78bd8473-signing-key\") pod \"service-ca-5fffc97bdf-pfwg8\" (UID: \"d71659c8-f08f-4221-a160-56ac78bd8473\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pfwg8" Oct 30 19:47:19.858873 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.858767 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d78606e4-ccb9-421a-8ae5-8fbecbcd3831-serving-cert\") pod \"insights-operator-7c4986d449-bnp7s\" (UID: \"d78606e4-ccb9-421a-8ae5-8fbecbcd3831\") " pod="openshift-insights/insights-operator-7c4986d449-bnp7s" Oct 30 19:47:19.858873 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.858832 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d71659c8-f08f-4221-a160-56ac78bd8473-signing-cabundle\") pod \"service-ca-5fffc97bdf-pfwg8\" (UID: \"d71659c8-f08f-4221-a160-56ac78bd8473\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pfwg8" Oct 30 19:47:19.858873 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.858871 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzg2k\" (UniqueName: \"kubernetes.io/projected/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-kube-api-access-rzg2k\") pod \"image-registry-5bc64fc876-25g7d\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:19.859113 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.858904 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/005bece8-8d37-4a58-954c-0ae3dab9dadc-trusted-ca-bundle\") pod \"periodic-gathering-d7ptc-wwx7j\" (UID: \"005bece8-8d37-4a58-954c-0ae3dab9dadc\") " pod="openshift-insights/periodic-gathering-d7ptc-wwx7j" Oct 30 19:47:19.859113 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.858934 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbnlm\" (UniqueName: \"kubernetes.io/projected/74b7c739-483f-4756-b97b-e65d37387b17-kube-api-access-gbnlm\") pod \"service-ca-operator-544d749c8c-6wqfc\" (UID: \"74b7c739-483f-4756-b97b-e65d37387b17\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6wqfc" Oct 30 19:47:19.859113 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.858966 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpkkb\" (UniqueName: \"kubernetes.io/projected/cba5fe49-b5b5-413e-9085-e35762430fa9-kube-api-access-hpkkb\") pod \"network-check-source-6f78d59f99-k8mq4\" (UID: \"cba5fe49-b5b5-413e-9085-e35762430fa9\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-k8mq4" Oct 30 19:47:19.859113 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.858996 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/13962162-194c-4e37-ac6b-b3bbc810cedb-agent-certs\") pod \"konnectivity-agent-djkzp\" (UID: \"13962162-194c-4e37-ac6b-b3bbc810cedb\") " pod="kube-system/konnectivity-agent-djkzp" Oct 30 19:47:19.859113 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.859028 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rwww\" (UniqueName: \"kubernetes.io/projected/20f593c8-594e-4a64-9c5e-881311b269a3-kube-api-access-6rwww\") pod \"cluster-monitoring-operator-884576c57-qdv4s\" (UID: \"20f593c8-594e-4a64-9c5e-881311b269a3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qdv4s" Oct 30 19:47:19.859113 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.859056 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d1dca14b-4840-4dd6-a7cd-7d795adb9532-device-dir\") pod \"aws-ebs-csi-driver-node-glksn\" (UID: \"d1dca14b-4840-4dd6-a7cd-7d795adb9532\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" Oct 30 19:47:19.859113 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.859056 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/41534930-917f-44ec-bccd-86d6c6fc7c70-stats-auth\") pod \"router-default-79458f7dbc-vbb28\" (UID: \"41534930-917f-44ec-bccd-86d6c6fc7c70\") " pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:19.859113 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.859092 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-etc-tuned\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:47:19.859427 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.859124 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5rr5\" (UniqueName: \"kubernetes.io/projected/66999038-807b-406b-9242-c68c04e521f3-kube-api-access-t5rr5\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.859427 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.859156 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-run-systemd\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.859427 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.859191 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-etc-sysctl-d\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:47:19.859427 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.859247 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm7rk\" (UniqueName: \"kubernetes.io/projected/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-kube-api-access-cm7rk\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:47:19.859427 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.859270 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d71659c8-f08f-4221-a160-56ac78bd8473-signing-cabundle\") pod \"service-ca-5fffc97bdf-pfwg8\" (UID: \"d71659c8-f08f-4221-a160-56ac78bd8473\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pfwg8" Oct 30 19:47:19.859427 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.859277 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-cnibin\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.859427 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.859300 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-host-slash\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.859427 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.859330 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58q6j\" (UniqueName: \"kubernetes.io/projected/908c2874-510c-43c4-a07c-c1771b924e8d-kube-api-access-58q6j\") pod \"kas-connection-checker-685895c7c9-c7n9t\" (UID: \"908c2874-510c-43c4-a07c-c1771b924e8d\") " pod="kube-system/kas-connection-checker-685895c7c9-c7n9t" Oct 30 19:47:19.859427 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.859363 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b1440092-9901-4eff-8a9d-6174b2dfdada-image-registry-private-configuration\") pod \"image-registry-6796cf4b57-b9d8k\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:19.859427 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.859394 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xtvg\" (UniqueName: \"kubernetes.io/projected/b1440092-9901-4eff-8a9d-6174b2dfdada-kube-api-access-8xtvg\") pod \"image-registry-6796cf4b57-b9d8k\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:19.859427 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.859425 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a160cec7-bc1f-4202-8382-e84318122fd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8txdr\" (UID: \"a160cec7-bc1f-4202-8382-e84318122fd9\") " pod="openshift-insights/insights-runtime-extractor-8txdr" Oct 30 19:47:19.859876 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.859452 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/08b5a0ca-cfe1-4366-a404-5ebbaeb7c149-hosts-file\") pod \"node-resolver-l9fpp\" (UID: \"08b5a0ca-cfe1-4366-a404-5ebbaeb7c149\") " pod="openshift-dns/node-resolver-l9fpp" Oct 30 19:47:19.859876 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.859482 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-log-socket\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.859876 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.859515 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.859876 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.859550 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b1440092-9901-4eff-8a9d-6174b2dfdada-registry-certificates\") pod \"image-registry-6796cf4b57-b9d8k\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:19.859876 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.859577 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/da850825-59ab-480f-9047-afd7d2bdc19f-serving-cert\") pod \"console-operator-7cd56b8dbc-qdj4h\" (UID: \"da850825-59ab-480f-9047-afd7d2bdc19f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qdj4h" Oct 30 19:47:19.859876 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.859626 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-hostroot\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.859876 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.859657 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-etc-kubernetes\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.859876 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.859688 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lph46\" (UniqueName: \"kubernetes.io/projected/116a1241-c9a3-4664-8217-60c2b6aa8b76-kube-api-access-lph46\") pod \"network-check-target-kvw9t\" (UID: \"116a1241-c9a3-4664-8217-60c2b6aa8b76\") " pod="openshift-network-diagnostics/network-check-target-kvw9t" Oct 30 19:47:19.859876 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.859720 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a160cec7-bc1f-4202-8382-e84318122fd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8txdr\" (UID: \"a160cec7-bc1f-4202-8382-e84318122fd9\") " pod="openshift-insights/insights-runtime-extractor-8txdr" Oct 30 19:47:19.859876 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.859745 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1afac97f-a617-4113-b8e8-c668caed54f9-cnibin\") pod \"multus-additional-cni-plugins-2hfrk\" (UID: \"1afac97f-a617-4113-b8e8-c668caed54f9\") " pod="openshift-multus/multus-additional-cni-plugins-2hfrk" Oct 30 19:47:19.859876 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.859776 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/13962162-194c-4e37-ac6b-b3bbc810cedb-konnectivity-ca\") pod \"konnectivity-agent-djkzp\" (UID: \"13962162-194c-4e37-ac6b-b3bbc810cedb\") " pod="kube-system/konnectivity-agent-djkzp" Oct 30 19:47:19.859876 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.859778 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/005bece8-8d37-4a58-954c-0ae3dab9dadc-trusted-ca-bundle\") pod \"periodic-gathering-d7ptc-wwx7j\" (UID: \"005bece8-8d37-4a58-954c-0ae3dab9dadc\") " pod="openshift-insights/periodic-gathering-d7ptc-wwx7j" Oct 30 19:47:19.859876 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.859805 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-lib-modules\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:47:19.859876 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.859839 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a160cec7-bc1f-4202-8382-e84318122fd9-data-volume\") pod \"insights-runtime-extractor-8txdr\" (UID: \"a160cec7-bc1f-4202-8382-e84318122fd9\") " pod="openshift-insights/insights-runtime-extractor-8txdr" Oct 30 19:47:19.859876 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.859870 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-host-var-lib-cni-multus\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.860475 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.859895 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d78606e4-ccb9-421a-8ae5-8fbecbcd3831-tmp\") pod \"insights-operator-7c4986d449-bnp7s\" (UID: \"d78606e4-ccb9-421a-8ae5-8fbecbcd3831\") " pod="openshift-insights/insights-operator-7c4986d449-bnp7s" Oct 30 19:47:19.860475 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.859926 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d1dca14b-4840-4dd6-a7cd-7d795adb9532-socket-dir\") pod \"aws-ebs-csi-driver-node-glksn\" (UID: \"d1dca14b-4840-4dd6-a7cd-7d795adb9532\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" Oct 30 19:47:19.860475 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.859955 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-var-lib-kubelet\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:47:19.860475 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.859989 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blstw\" (UniqueName: \"kubernetes.io/projected/a160cec7-bc1f-4202-8382-e84318122fd9-kube-api-access-blstw\") pod \"insights-runtime-extractor-8txdr\" (UID: \"a160cec7-bc1f-4202-8382-e84318122fd9\") " pod="openshift-insights/insights-runtime-extractor-8txdr" Oct 30 19:47:19.860475 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.859997 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b1440092-9901-4eff-8a9d-6174b2dfdada-image-registry-private-configuration\") pod \"image-registry-6796cf4b57-b9d8k\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:19.860475 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.860016 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1afac97f-a617-4113-b8e8-c668caed54f9-cni-binary-copy\") pod \"multus-additional-cni-plugins-2hfrk\" (UID: \"1afac97f-a617-4113-b8e8-c668caed54f9\") " pod="openshift-multus/multus-additional-cni-plugins-2hfrk" Oct 30 19:47:19.860475 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.860049 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1440092-9901-4eff-8a9d-6174b2dfdada-registry-tls\") pod \"image-registry-6796cf4b57-b9d8k\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:19.860475 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.860079 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-etc-systemd\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:47:19.860475 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.860110 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-host\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:47:19.860475 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.860139 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-etc-openvswitch\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.860475 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.860165 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/73bb0a1f-597a-4078-8a01-20d0ed24715b-kubelet-config\") pod \"global-pull-secret-syncer-hz6jn\" (UID: \"73bb0a1f-597a-4078-8a01-20d0ed24715b\") " pod="kube-system/global-pull-secret-syncer-hz6jn" Oct 30 19:47:19.860475 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.860197 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/20f593c8-594e-4a64-9c5e-881311b269a3-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-qdv4s\" (UID: \"20f593c8-594e-4a64-9c5e-881311b269a3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qdv4s" Oct 30 19:47:19.860475 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.860228 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-multus-socket-dir-parent\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.860475 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.860470 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-systemd-units\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.861055 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.860515 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/688d079a-932d-4f63-97da-5cd6222a03e1-ovnkube-config\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.861055 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.860550 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-installation-pull-secrets\") pod \"image-registry-5bc64fc876-25g7d\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:19.861055 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.860584 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2e03cb52-fd85-4e35-87b4-38833cbc7d74-serviceca\") pod \"node-ca-hkvn6\" (UID: \"2e03cb52-fd85-4e35-87b4-38833cbc7d74\") " pod="openshift-image-registry/node-ca-hkvn6" Oct 30 19:47:19.861055 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.860641 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/74b7c739-483f-4756-b97b-e65d37387b17-config\") pod \"service-ca-operator-544d749c8c-6wqfc\" (UID: \"74b7c739-483f-4756-b97b-e65d37387b17\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6wqfc" Oct 30 19:47:19.861055 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.860676 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1342ef67-5f63-47fb-a337-64c693e11228-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n75h8\" (UID: \"1342ef67-5f63-47fb-a337-64c693e11228\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n75h8" Oct 30 19:47:19.861055 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.860708 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-multus-conf-dir\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.861055 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.860734 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1afac97f-a617-4113-b8e8-c668caed54f9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2hfrk\" (UID: \"1afac97f-a617-4113-b8e8-c668caed54f9\") " pod="openshift-multus/multus-additional-cni-plugins-2hfrk" Oct 30 19:47:19.861055 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.860738 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/13962162-194c-4e37-ac6b-b3bbc810cedb-agent-certs\") pod \"konnectivity-agent-djkzp\" (UID: \"13962162-194c-4e37-ac6b-b3bbc810cedb\") " pod="kube-system/konnectivity-agent-djkzp" Oct 30 19:47:19.861379 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.861084 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-etc-tuned\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:47:19.861596 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.861574 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/20f593c8-594e-4a64-9c5e-881311b269a3-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-qdv4s\" (UID: \"20f593c8-594e-4a64-9c5e-881311b269a3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qdv4s" Oct 30 19:47:19.861803 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.861775 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/13962162-194c-4e37-ac6b-b3bbc810cedb-konnectivity-ca\") pod \"konnectivity-agent-djkzp\" (UID: \"13962162-194c-4e37-ac6b-b3bbc810cedb\") " pod="kube-system/konnectivity-agent-djkzp" Oct 30 19:47:19.861861 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.861799 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b1440092-9901-4eff-8a9d-6174b2dfdada-registry-certificates\") pod \"image-registry-6796cf4b57-b9d8k\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:19.861861 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.861854 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a160cec7-bc1f-4202-8382-e84318122fd9-data-volume\") pod \"insights-runtime-extractor-8txdr\" (UID: \"a160cec7-bc1f-4202-8382-e84318122fd9\") " pod="openshift-insights/insights-runtime-extractor-8txdr" Oct 30 19:47:19.861981 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.861933 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d78606e4-ccb9-421a-8ae5-8fbecbcd3831-tmp\") pod \"insights-operator-7c4986d449-bnp7s\" (UID: \"d78606e4-ccb9-421a-8ae5-8fbecbcd3831\") " pod="openshift-insights/insights-operator-7c4986d449-bnp7s" Oct 30 19:47:19.862149 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.862133 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/688d079a-932d-4f63-97da-5cd6222a03e1-ovnkube-config\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.862381 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.862363 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-installation-pull-secrets\") pod \"image-registry-5bc64fc876-25g7d\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:19.862475 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.862455 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/da850825-59ab-480f-9047-afd7d2bdc19f-serving-cert\") pod \"console-operator-7cd56b8dbc-qdj4h\" (UID: \"da850825-59ab-480f-9047-afd7d2bdc19f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qdj4h" Oct 30 19:47:19.862606 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.862589 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2e03cb52-fd85-4e35-87b4-38833cbc7d74-serviceca\") pod \"node-ca-hkvn6\" (UID: \"2e03cb52-fd85-4e35-87b4-38833cbc7d74\") " pod="openshift-image-registry/node-ca-hkvn6" Oct 30 19:47:19.862711 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.862680 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1afac97f-a617-4113-b8e8-c668caed54f9-cni-binary-copy\") pod \"multus-additional-cni-plugins-2hfrk\" (UID: \"1afac97f-a617-4113-b8e8-c668caed54f9\") " pod="openshift-multus/multus-additional-cni-plugins-2hfrk" Oct 30 19:47:19.862833 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.862816 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/74b7c739-483f-4756-b97b-e65d37387b17-config\") pod \"service-ca-operator-544d749c8c-6wqfc\" (UID: \"74b7c739-483f-4756-b97b-e65d37387b17\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6wqfc" Oct 30 19:47:19.879319 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.879275 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg9ks\" (UniqueName: \"kubernetes.io/projected/aab2acf3-4af5-4d2e-8336-05b01a2dca1e-kube-api-access-lg9ks\") pod \"kas-connection-checker-685895c7c9-kflff\" (UID: \"aab2acf3-4af5-4d2e-8336-05b01a2dca1e\") " pod="kube-system/kas-connection-checker-685895c7c9-kflff" Oct 30 19:47:19.879635 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.879593 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv4p7\" (UniqueName: \"kubernetes.io/projected/9b6dd02b-d4bd-4959-9a06-8941d6baf1a5-kube-api-access-sv4p7\") pod \"volume-data-source-validator-58c9f5f46d-xlgtn\" (UID: \"9b6dd02b-d4bd-4959-9a06-8941d6baf1a5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xlgtn" Oct 30 19:47:19.879700 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.879642 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrg8k\" (UniqueName: \"kubernetes.io/projected/a421de2b-dd41-4862-9849-0a4f0256af1b-kube-api-access-rrg8k\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-95bqt\" (UID: \"a421de2b-dd41-4862-9849-0a4f0256af1b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95bqt" Oct 30 19:47:19.879780 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.879763 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljrnk\" (UniqueName: \"kubernetes.io/projected/d78606e4-ccb9-421a-8ae5-8fbecbcd3831-kube-api-access-ljrnk\") pod \"insights-operator-7c4986d449-bnp7s\" (UID: \"d78606e4-ccb9-421a-8ae5-8fbecbcd3831\") " pod="openshift-insights/insights-operator-7c4986d449-bnp7s" Oct 30 19:47:19.892330 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.892303 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkbkr\" (UniqueName: \"kubernetes.io/projected/2962e3da-8439-4316-9c3a-1679842609b7-kube-api-access-rkbkr\") pod \"kas-connection-checker-685895c7c9-r7djq\" (UID: \"2962e3da-8439-4316-9c3a-1679842609b7\") " pod="kube-system/kas-connection-checker-685895c7c9-r7djq" Oct 30 19:47:19.892474 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.892425 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtqp8\" (UniqueName: \"kubernetes.io/projected/d1dca14b-4840-4dd6-a7cd-7d795adb9532-kube-api-access-xtqp8\") pod \"aws-ebs-csi-driver-node-glksn\" (UID: \"d1dca14b-4840-4dd6-a7cd-7d795adb9532\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" Oct 30 19:47:19.892554 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.892510 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr5dn\" (UniqueName: \"kubernetes.io/projected/1afac97f-a617-4113-b8e8-c668caed54f9-kube-api-access-nr5dn\") pod \"multus-additional-cni-plugins-2hfrk\" (UID: \"1afac97f-a617-4113-b8e8-c668caed54f9\") " pod="openshift-multus/multus-additional-cni-plugins-2hfrk" Oct 30 19:47:19.892554 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.892510 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvcvq\" (UniqueName: \"kubernetes.io/projected/0547f118-5c20-403a-beca-c6a5a671a1ff-kube-api-access-vvcvq\") pod \"migrator-66df5d496b-b7sz6\" (UID: \"0547f118-5c20-403a-beca-c6a5a671a1ff\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b7sz6" Oct 30 19:47:19.894090 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.892888 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94r98\" (UniqueName: \"kubernetes.io/projected/2e03cb52-fd85-4e35-87b4-38833cbc7d74-kube-api-access-94r98\") pod \"node-ca-hkvn6\" (UID: \"2e03cb52-fd85-4e35-87b4-38833cbc7d74\") " pod="openshift-image-registry/node-ca-hkvn6" Oct 30 19:47:19.894090 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.893006 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b1440092-9901-4eff-8a9d-6174b2dfdada-bound-sa-token\") pod \"image-registry-6796cf4b57-b9d8k\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:19.894090 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.893026 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfbsr\" (UniqueName: \"kubernetes.io/projected/688d079a-932d-4f63-97da-5cd6222a03e1-kube-api-access-tfbsr\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.894090 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.893033 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb8sb\" (UniqueName: \"kubernetes.io/projected/57ddd2a0-f081-4f90-95a6-0f6f8df2f433-kube-api-access-wb8sb\") pod \"network-metrics-daemon-7f4jl\" (UID: \"57ddd2a0-f081-4f90-95a6-0f6f8df2f433\") " pod="openshift-multus/network-metrics-daemon-7f4jl" Oct 30 19:47:19.894090 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.893232 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4h6p\" (UniqueName: \"kubernetes.io/projected/d71659c8-f08f-4221-a160-56ac78bd8473-kube-api-access-s4h6p\") pod \"service-ca-5fffc97bdf-pfwg8\" (UID: \"d71659c8-f08f-4221-a160-56ac78bd8473\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pfwg8" Oct 30 19:47:19.894090 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.893270 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtksm\" (UniqueName: \"kubernetes.io/projected/b8e3ed9d-5156-4698-82c3-df72fa728f94-kube-api-access-xtksm\") pod \"dns-default-ml5lp\" (UID: \"b8e3ed9d-5156-4698-82c3-df72fa728f94\") " pod="openshift-dns/dns-default-ml5lp" Oct 30 19:47:19.894090 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.893503 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v26lp\" (UniqueName: \"kubernetes.io/projected/da850825-59ab-480f-9047-afd7d2bdc19f-kube-api-access-v26lp\") pod \"console-operator-7cd56b8dbc-qdj4h\" (UID: \"da850825-59ab-480f-9047-afd7d2bdc19f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qdj4h" Oct 30 19:47:19.894090 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.893503 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkmdz\" (UniqueName: \"kubernetes.io/projected/d808d253-3f30-48e8-918c-bd3880c2af8f-kube-api-access-mkmdz\") pod \"iptables-alerter-882qh\" (UID: \"d808d253-3f30-48e8-918c-bd3880c2af8f\") " pod="openshift-network-operator/iptables-alerter-882qh" Oct 30 19:47:19.894090 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.893537 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68nwv\" (UniqueName: \"kubernetes.io/projected/005bece8-8d37-4a58-954c-0ae3dab9dadc-kube-api-access-68nwv\") pod \"periodic-gathering-d7ptc-wwx7j\" (UID: \"005bece8-8d37-4a58-954c-0ae3dab9dadc\") " pod="openshift-insights/periodic-gathering-d7ptc-wwx7j" Oct 30 19:47:19.894090 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.893777 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbnlm\" (UniqueName: \"kubernetes.io/projected/74b7c739-483f-4756-b97b-e65d37387b17-kube-api-access-gbnlm\") pod \"service-ca-operator-544d749c8c-6wqfc\" (UID: \"74b7c739-483f-4756-b97b-e65d37387b17\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6wqfc" Oct 30 19:47:19.894090 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.893777 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x59q2\" (UniqueName: \"kubernetes.io/projected/41534930-917f-44ec-bccd-86d6c6fc7c70-kube-api-access-x59q2\") pod \"router-default-79458f7dbc-vbb28\" (UID: \"41534930-917f-44ec-bccd-86d6c6fc7c70\") " pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:19.894090 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.893952 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-bound-sa-token\") pod \"image-registry-5bc64fc876-25g7d\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:19.894090 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.893966 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpkkb\" (UniqueName: \"kubernetes.io/projected/cba5fe49-b5b5-413e-9085-e35762430fa9-kube-api-access-hpkkb\") pod \"network-check-source-6f78d59f99-k8mq4\" (UID: \"cba5fe49-b5b5-413e-9085-e35762430fa9\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-k8mq4" Oct 30 19:47:19.894090 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.893980 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58q6j\" (UniqueName: \"kubernetes.io/projected/908c2874-510c-43c4-a07c-c1771b924e8d-kube-api-access-58q6j\") pod \"kas-connection-checker-685895c7c9-c7n9t\" (UID: \"908c2874-510c-43c4-a07c-c1771b924e8d\") " pod="kube-system/kas-connection-checker-685895c7c9-c7n9t" Oct 30 19:47:19.894090 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.893974 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xtvg\" (UniqueName: \"kubernetes.io/projected/b1440092-9901-4eff-8a9d-6174b2dfdada-kube-api-access-8xtvg\") pod \"image-registry-6796cf4b57-b9d8k\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:19.894090 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.894030 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5rr5\" (UniqueName: \"kubernetes.io/projected/66999038-807b-406b-9242-c68c04e521f3-kube-api-access-t5rr5\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.895604 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.894298 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blstw\" (UniqueName: \"kubernetes.io/projected/a160cec7-bc1f-4202-8382-e84318122fd9-kube-api-access-blstw\") pod \"insights-runtime-extractor-8txdr\" (UID: \"a160cec7-bc1f-4202-8382-e84318122fd9\") " pod="openshift-insights/insights-runtime-extractor-8txdr" Oct 30 19:47:19.895604 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.894469 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj8ws\" (UniqueName: \"kubernetes.io/projected/1342ef67-5f63-47fb-a337-64c693e11228-kube-api-access-vj8ws\") pod \"cluster-samples-operator-6ddb6c8b94-n75h8\" (UID: \"1342ef67-5f63-47fb-a337-64c693e11228\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n75h8" Oct 30 19:47:19.895604 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.894499 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tng88\" (UniqueName: \"kubernetes.io/projected/7a570505-5193-4564-b776-0f83df60d4e9-kube-api-access-tng88\") pod \"migrator-66df5d496b-vdklw\" (UID: \"7a570505-5193-4564-b776-0f83df60d4e9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vdklw" Oct 30 19:47:19.895604 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.894655 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm7rk\" (UniqueName: \"kubernetes.io/projected/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-kube-api-access-cm7rk\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:47:19.895604 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.894729 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lph46\" (UniqueName: \"kubernetes.io/projected/116a1241-c9a3-4664-8217-60c2b6aa8b76-kube-api-access-lph46\") pod \"network-check-target-kvw9t\" (UID: \"116a1241-c9a3-4664-8217-60c2b6aa8b76\") " pod="openshift-network-diagnostics/network-check-target-kvw9t" Oct 30 19:47:19.895604 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.894859 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxq6z\" (UniqueName: \"kubernetes.io/projected/5ad8f4b0-50ba-4191-8df4-25930717acb0-kube-api-access-lxq6z\") pod \"ingress-canary-lk574\" (UID: \"5ad8f4b0-50ba-4191-8df4-25930717acb0\") " pod="openshift-ingress-canary/ingress-canary-lk574" Oct 30 19:47:19.895604 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.894883 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49lpb\" (UniqueName: \"kubernetes.io/projected/08b5a0ca-cfe1-4366-a404-5ebbaeb7c149-kube-api-access-49lpb\") pod \"node-resolver-l9fpp\" (UID: \"08b5a0ca-cfe1-4366-a404-5ebbaeb7c149\") " pod="openshift-dns/node-resolver-l9fpp" Oct 30 19:47:19.895604 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.895159 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzg2k\" (UniqueName: \"kubernetes.io/projected/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-kube-api-access-rzg2k\") pod \"image-registry-5bc64fc876-25g7d\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:19.895604 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.895312 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rwww\" (UniqueName: \"kubernetes.io/projected/20f593c8-594e-4a64-9c5e-881311b269a3-kube-api-access-6rwww\") pod \"cluster-monitoring-operator-884576c57-qdv4s\" (UID: \"20f593c8-594e-4a64-9c5e-881311b269a3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qdv4s" Oct 30 19:47:19.936757 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.936696 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-b7sz6_0547f118-5c20-403a-beca-c6a5a671a1ff/migrator/0.log" Oct 30 19:47:19.961549 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.961526 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-registry-tls\") pod \"image-registry-5bc64fc876-25g7d\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:19.961689 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.961554 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/20f593c8-594e-4a64-9c5e-881311b269a3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qdv4s\" (UID: \"20f593c8-594e-4a64-9c5e-881311b269a3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qdv4s" Oct 30 19:47:19.961689 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.961574 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8e3ed9d-5156-4698-82c3-df72fa728f94-metrics-tls\") pod \"dns-default-ml5lp\" (UID: \"b8e3ed9d-5156-4698-82c3-df72fa728f94\") " pod="openshift-dns/dns-default-ml5lp" Oct 30 19:47:19.961689 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.961589 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d1dca14b-4840-4dd6-a7cd-7d795adb9532-kubelet-dir\") pod \"aws-ebs-csi-driver-node-glksn\" (UID: \"d1dca14b-4840-4dd6-a7cd-7d795adb9532\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" Oct 30 19:47:19.961689 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.961607 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/afb6fa1c-9722-42c4-a914-4941bcbcaaad-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-tv4xh\" (UID: \"afb6fa1c-9722-42c4-a914-4941bcbcaaad\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-tv4xh" Oct 30 19:47:19.961838 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.961686 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/73bb0a1f-597a-4078-8a01-20d0ed24715b-dbus\") pod \"global-pull-secret-syncer-hz6jn\" (UID: \"73bb0a1f-597a-4078-8a01-20d0ed24715b\") " pod="kube-system/global-pull-secret-syncer-hz6jn" Oct 30 19:47:19.961838 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.961718 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d1dca14b-4840-4dd6-a7cd-7d795adb9532-kubelet-dir\") pod \"aws-ebs-csi-driver-node-glksn\" (UID: \"d1dca14b-4840-4dd6-a7cd-7d795adb9532\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" Oct 30 19:47:19.961838 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.961729 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-host-cni-netd\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.961838 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:47:19.961708 5088 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 19:47:19.961838 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:47:19.961764 5088 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:47:19.961838 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.961782 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-etc-kubernetes\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:47:19.961838 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.961795 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/73bb0a1f-597a-4078-8a01-20d0ed24715b-dbus\") pod \"global-pull-secret-syncer-hz6jn\" (UID: \"73bb0a1f-597a-4078-8a01-20d0ed24715b\") " pod="kube-system/global-pull-secret-syncer-hz6jn" Oct 30 19:47:19.961838 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.961807 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-host-var-lib-kubelet\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.961838 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:47:19.961709 5088 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 19:47:19.961838 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:47:19.961835 5088 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8e3ed9d-5156-4698-82c3-df72fa728f94-metrics-tls podName:b8e3ed9d-5156-4698-82c3-df72fa728f94 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:20.461815786 +0000 UTC m=+1.874529556 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b8e3ed9d-5156-4698-82c3-df72fa728f94-metrics-tls") pod "dns-default-ml5lp" (UID: "b8e3ed9d-5156-4698-82c3-df72fa728f94") : secret "dns-default-metrics-tls" not found Oct 30 19:47:19.962161 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.961843 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-host-var-lib-kubelet\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.962161 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:47:19.961872 5088 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afb6fa1c-9722-42c4-a914-4941bcbcaaad-networking-console-plugin-cert podName:afb6fa1c-9722-42c4-a914-4941bcbcaaad nodeName:}" failed. No retries permitted until 2026-10-30 19:47:20.461863023 +0000 UTC m=+1.874576783 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/afb6fa1c-9722-42c4-a914-4941bcbcaaad-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-tv4xh" (UID: "afb6fa1c-9722-42c4-a914-4941bcbcaaad") : secret "networking-console-plugin-cert" not found Oct 30 19:47:19.962161 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.961882 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-host-cni-netd\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.962161 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:47:19.961888 5088 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20f593c8-594e-4a64-9c5e-881311b269a3-cluster-monitoring-operator-tls podName:20f593c8-594e-4a64-9c5e-881311b269a3 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:20.461879379 +0000 UTC m=+1.874593137 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/20f593c8-594e-4a64-9c5e-881311b269a3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-qdv4s" (UID: "20f593c8-594e-4a64-9c5e-881311b269a3") : secret "cluster-monitoring-operator-tls" not found Oct 30 19:47:19.962161 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.961901 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-etc-kubernetes\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:47:19.962161 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.961914 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-var-lib-openvswitch\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.962161 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.961931 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-var-lib-openvswitch\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.962161 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.961938 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-node-log\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.962161 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.961962 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-system-cni-dir\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.962161 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.961966 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-node-log\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.962161 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.961963 5088 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 19:47:19.962161 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.961985 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57ddd2a0-f081-4f90-95a6-0f6f8df2f433-metrics-certs\") pod \"network-metrics-daemon-7f4jl\" (UID: \"57ddd2a0-f081-4f90-95a6-0f6f8df2f433\") " pod="openshift-multus/network-metrics-daemon-7f4jl" Oct 30 19:47:19.962161 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962014 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-host-run-netns\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.962161 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962032 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d1dca14b-4840-4dd6-a7cd-7d795adb9532-registration-dir\") pod \"aws-ebs-csi-driver-node-glksn\" (UID: \"d1dca14b-4840-4dd6-a7cd-7d795adb9532\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" Oct 30 19:47:19.962161 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962079 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d1dca14b-4840-4dd6-a7cd-7d795adb9532-registration-dir\") pod \"aws-ebs-csi-driver-node-glksn\" (UID: \"d1dca14b-4840-4dd6-a7cd-7d795adb9532\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" Oct 30 19:47:19.962161 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962078 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-host-run-netns\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.962161 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962078 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-system-cni-dir\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.962161 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962098 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-host-cni-bin\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.962161 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962121 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-host-cni-bin\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.962161 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962124 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1afac97f-a617-4113-b8e8-c668caed54f9-system-cni-dir\") pod \"multus-additional-cni-plugins-2hfrk\" (UID: \"1afac97f-a617-4113-b8e8-c668caed54f9\") " pod="openshift-multus/multus-additional-cni-plugins-2hfrk" Oct 30 19:47:19.962161 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962143 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1afac97f-a617-4113-b8e8-c668caed54f9-system-cni-dir\") pod \"multus-additional-cni-plugins-2hfrk\" (UID: \"1afac97f-a617-4113-b8e8-c668caed54f9\") " pod="openshift-multus/multus-additional-cni-plugins-2hfrk" Oct 30 19:47:19.962161 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962164 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-run-openvswitch\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962188 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-etc-modprobe-d\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962210 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-etc-sysconfig\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962232 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-host-var-lib-cni-bin\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962209 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-run-openvswitch\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962257 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d1dca14b-4840-4dd6-a7cd-7d795adb9532-etc-selinux\") pod \"aws-ebs-csi-driver-node-glksn\" (UID: \"d1dca14b-4840-4dd6-a7cd-7d795adb9532\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962278 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-etc-sysconfig\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962284 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-run\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962298 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d1dca14b-4840-4dd6-a7cd-7d795adb9532-etc-selinux\") pod \"aws-ebs-csi-driver-node-glksn\" (UID: \"d1dca14b-4840-4dd6-a7cd-7d795adb9532\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962304 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-host-var-lib-cni-bin\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962316 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-host-run-multus-certs\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962319 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-etc-modprobe-d\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962341 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-host-run-ovn-kubernetes\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962324 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-run\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962373 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-host-run-multus-certs\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962381 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-host-run-ovn-kubernetes\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962385 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d1dca14b-4840-4dd6-a7cd-7d795adb9532-sys-fs\") pod \"aws-ebs-csi-driver-node-glksn\" (UID: \"d1dca14b-4840-4dd6-a7cd-7d795adb9532\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962418 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d1dca14b-4840-4dd6-a7cd-7d795adb9532-sys-fs\") pod \"aws-ebs-csi-driver-node-glksn\" (UID: \"d1dca14b-4840-4dd6-a7cd-7d795adb9532\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962432 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-host-run-k8s-cni-cncf-io\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962458 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-host-kubelet\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962482 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41534930-917f-44ec-bccd-86d6c6fc7c70-metrics-certs\") pod \"router-default-79458f7dbc-vbb28\" (UID: \"41534930-917f-44ec-bccd-86d6c6fc7c70\") " pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962494 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-host-run-k8s-cni-cncf-io\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962505 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a160cec7-bc1f-4202-8382-e84318122fd9-crio-socket\") pod \"insights-runtime-extractor-8txdr\" (UID: \"a160cec7-bc1f-4202-8382-e84318122fd9\") " pod="openshift-insights/insights-runtime-extractor-8txdr" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962529 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d808d253-3f30-48e8-918c-bd3880c2af8f-host-slash\") pod \"iptables-alerter-882qh\" (UID: \"d808d253-3f30-48e8-918c-bd3880c2af8f\") " pod="openshift-network-operator/iptables-alerter-882qh" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962537 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-host-kubelet\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962560 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e03cb52-fd85-4e35-87b4-38833cbc7d74-host\") pod \"node-ca-hkvn6\" (UID: \"2e03cb52-fd85-4e35-87b4-38833cbc7d74\") " pod="openshift-image-registry/node-ca-hkvn6" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:47:19.962556 5088 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962580 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d808d253-3f30-48e8-918c-bd3880c2af8f-host-slash\") pod \"iptables-alerter-882qh\" (UID: \"d808d253-3f30-48e8-918c-bd3880c2af8f\") " pod="openshift-network-operator/iptables-alerter-882qh" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962586 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41534930-917f-44ec-bccd-86d6c6fc7c70-service-ca-bundle\") pod \"router-default-79458f7dbc-vbb28\" (UID: \"41534930-917f-44ec-bccd-86d6c6fc7c70\") " pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962565 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a160cec7-bc1f-4202-8382-e84318122fd9-crio-socket\") pod \"insights-runtime-extractor-8txdr\" (UID: \"a160cec7-bc1f-4202-8382-e84318122fd9\") " pod="openshift-insights/insights-runtime-extractor-8txdr" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:47:19.962624 5088 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41534930-917f-44ec-bccd-86d6c6fc7c70-metrics-certs podName:41534930-917f-44ec-bccd-86d6c6fc7c70 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:20.462593748 +0000 UTC m=+1.875307525 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/41534930-917f-44ec-bccd-86d6c6fc7c70-metrics-certs") pod "router-default-79458f7dbc-vbb28" (UID: "41534930-917f-44ec-bccd-86d6c6fc7c70") : secret "router-metrics-certs-default" not found Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962625 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e03cb52-fd85-4e35-87b4-38833cbc7d74-host\") pod \"node-ca-hkvn6\" (UID: \"2e03cb52-fd85-4e35-87b4-38833cbc7d74\") " pod="openshift-image-registry/node-ca-hkvn6" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962656 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-sys\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962672 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-run-ovn\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:47:19.962710 5088 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/41534930-917f-44ec-bccd-86d6c6fc7c70-service-ca-bundle podName:41534930-917f-44ec-bccd-86d6c6fc7c70 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:20.462694949 +0000 UTC m=+1.875408719 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/41534930-917f-44ec-bccd-86d6c6fc7c70-service-ca-bundle") pod "router-default-79458f7dbc-vbb28" (UID: "41534930-917f-44ec-bccd-86d6c6fc7c70") : configmap references non-existent config key: service-ca.crt Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962723 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-sys\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962732 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-multus-cni-dir\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962767 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d1dca14b-4840-4dd6-a7cd-7d795adb9532-device-dir\") pod \"aws-ebs-csi-driver-node-glksn\" (UID: \"d1dca14b-4840-4dd6-a7cd-7d795adb9532\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962795 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-run-systemd\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962716 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-run-ovn\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962823 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-run-systemd\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962803 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-multus-cni-dir\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962838 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d1dca14b-4840-4dd6-a7cd-7d795adb9532-device-dir\") pod \"aws-ebs-csi-driver-node-glksn\" (UID: \"d1dca14b-4840-4dd6-a7cd-7d795adb9532\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962858 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-etc-sysctl-d\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962885 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-cnibin\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962908 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-host-slash\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962945 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a160cec7-bc1f-4202-8382-e84318122fd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8txdr\" (UID: \"a160cec7-bc1f-4202-8382-e84318122fd9\") " pod="openshift-insights/insights-runtime-extractor-8txdr" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962971 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/08b5a0ca-cfe1-4366-a404-5ebbaeb7c149-hosts-file\") pod \"node-resolver-l9fpp\" (UID: \"08b5a0ca-cfe1-4366-a404-5ebbaeb7c149\") " pod="openshift-dns/node-resolver-l9fpp" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962981 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-etc-sysctl-d\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962990 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-host-slash\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.962993 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-log-socket\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.963010 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-cnibin\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.963051 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/08b5a0ca-cfe1-4366-a404-5ebbaeb7c149-hosts-file\") pod \"node-resolver-l9fpp\" (UID: \"08b5a0ca-cfe1-4366-a404-5ebbaeb7c149\") " pod="openshift-dns/node-resolver-l9fpp" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.963049 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:47:19.963062 5088 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.963072 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-log-socket\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.963084 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-hostroot\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:47:19.963100 5088 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a160cec7-bc1f-4202-8382-e84318122fd9-insights-runtime-extractor-tls podName:a160cec7-bc1f-4202-8382-e84318122fd9 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:20.463087747 +0000 UTC m=+1.875801509 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a160cec7-bc1f-4202-8382-e84318122fd9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8txdr" (UID: "a160cec7-bc1f-4202-8382-e84318122fd9") : secret "insights-runtime-extractor-tls" not found Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.963101 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.963116 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-hostroot\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.963132 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-etc-kubernetes\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.963153 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a160cec7-bc1f-4202-8382-e84318122fd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8txdr\" (UID: \"a160cec7-bc1f-4202-8382-e84318122fd9\") " pod="openshift-insights/insights-runtime-extractor-8txdr" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.963171 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1afac97f-a617-4113-b8e8-c668caed54f9-cnibin\") pod \"multus-additional-cni-plugins-2hfrk\" (UID: \"1afac97f-a617-4113-b8e8-c668caed54f9\") " pod="openshift-multus/multus-additional-cni-plugins-2hfrk" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.963182 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-etc-kubernetes\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.963194 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-lib-modules\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:47:19.963163 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.963215 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-host-var-lib-cni-multus\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.965393 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.963223 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1afac97f-a617-4113-b8e8-c668caed54f9-cnibin\") pod \"multus-additional-cni-plugins-2hfrk\" (UID: \"1afac97f-a617-4113-b8e8-c668caed54f9\") " pod="openshift-multus/multus-additional-cni-plugins-2hfrk" Oct 30 19:47:19.965393 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.963231 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d1dca14b-4840-4dd6-a7cd-7d795adb9532-socket-dir\") pod \"aws-ebs-csi-driver-node-glksn\" (UID: \"d1dca14b-4840-4dd6-a7cd-7d795adb9532\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" Oct 30 19:47:19.965393 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.963247 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-var-lib-kubelet\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:47:19.965393 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.963270 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1440092-9901-4eff-8a9d-6174b2dfdada-registry-tls\") pod \"image-registry-6796cf4b57-b9d8k\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:19.965393 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.963295 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-etc-systemd\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:47:19.965393 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.963304 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d1dca14b-4840-4dd6-a7cd-7d795adb9532-socket-dir\") pod \"aws-ebs-csi-driver-node-glksn\" (UID: \"d1dca14b-4840-4dd6-a7cd-7d795adb9532\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glksn" Oct 30 19:47:19.965393 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.963309 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-lib-modules\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:47:19.965393 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.963315 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-host\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:47:19.965393 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.963340 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-etc-openvswitch\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.965393 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.963352 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-host-var-lib-cni-multus\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.965393 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.963362 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/73bb0a1f-597a-4078-8a01-20d0ed24715b-kubelet-config\") pod \"global-pull-secret-syncer-hz6jn\" (UID: \"73bb0a1f-597a-4078-8a01-20d0ed24715b\") " pod="kube-system/global-pull-secret-syncer-hz6jn" Oct 30 19:47:19.965393 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.963388 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-multus-socket-dir-parent\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.965393 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.963397 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-var-lib-kubelet\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:47:19.965393 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.963412 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-systemd-units\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.965393 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.963441 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1342ef67-5f63-47fb-a337-64c693e11228-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n75h8\" (UID: \"1342ef67-5f63-47fb-a337-64c693e11228\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n75h8" Oct 30 19:47:19.965393 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.963469 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-multus-conf-dir\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.965393 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.963477 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-etc-systemd\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:47:19.965393 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.963495 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1afac97f-a617-4113-b8e8-c668caed54f9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2hfrk\" (UID: \"1afac97f-a617-4113-b8e8-c668caed54f9\") " pod="openshift-multus/multus-additional-cni-plugins-2hfrk" Oct 30 19:47:19.965393 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.963513 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-host\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:47:19.965393 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.963520 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-etc-sysctl-conf\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:47:19.965393 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.963546 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-host-run-netns\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.965393 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.963564 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/73bb0a1f-597a-4078-8a01-20d0ed24715b-kubelet-config\") pod \"global-pull-secret-syncer-hz6jn\" (UID: \"73bb0a1f-597a-4078-8a01-20d0ed24715b\") " pod="kube-system/global-pull-secret-syncer-hz6jn" Oct 30 19:47:19.965393 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.963442 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-etc-openvswitch\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.965393 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.963632 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-multus-socket-dir-parent\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.965393 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:47:19.963671 5088 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 19:47:19.965393 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.963690 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66999038-807b-406b-9242-c68c04e521f3-multus-conf-dir\") pod \"multus-76qp7\" (UID: \"66999038-807b-406b-9242-c68c04e521f3\") " pod="openshift-multus/multus-76qp7" Oct 30 19:47:19.965393 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:47:19.963706 5088 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1342ef67-5f63-47fb-a337-64c693e11228-samples-operator-tls podName:1342ef67-5f63-47fb-a337-64c693e11228 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:20.463696138 +0000 UTC m=+1.876409908 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1342ef67-5f63-47fb-a337-64c693e11228-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-n75h8" (UID: "1342ef67-5f63-47fb-a337-64c693e11228") : secret "samples-operator-tls" not found Oct 30 19:47:19.965393 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.963715 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a160cec7-bc1f-4202-8382-e84318122fd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8txdr\" (UID: \"a160cec7-bc1f-4202-8382-e84318122fd9\") " pod="openshift-insights/insights-runtime-extractor-8txdr" Oct 30 19:47:19.965393 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.963727 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-host-run-netns\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.965393 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.963744 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/688d079a-932d-4f63-97da-5cd6222a03e1-systemd-units\") pod \"ovnkube-node-xpjpn\" (UID: \"688d079a-932d-4f63-97da-5cd6222a03e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:19.965393 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.963764 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e39329dd-fe4e-4ef9-853b-d8e5fe9d0617-etc-sysctl-conf\") pod \"tuned-rggcq\" (UID: \"e39329dd-fe4e-4ef9-853b-d8e5fe9d0617\") " pod="openshift-cluster-node-tuning-operator/tuned-rggcq" Oct 30 19:47:19.965393 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.963781 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1afac97f-a617-4113-b8e8-c668caed54f9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2hfrk\" (UID: \"1afac97f-a617-4113-b8e8-c668caed54f9\") " pod="openshift-multus/multus-additional-cni-plugins-2hfrk" Oct 30 19:47:19.965393 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.964396 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57ddd2a0-f081-4f90-95a6-0f6f8df2f433-metrics-certs\") pod \"network-metrics-daemon-7f4jl\" (UID: \"57ddd2a0-f081-4f90-95a6-0f6f8df2f433\") " pod="openshift-multus/network-metrics-daemon-7f4jl" Oct 30 19:47:19.965393 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.964741 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-registry-tls\") pod \"image-registry-5bc64fc876-25g7d\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:19.965393 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:19.965354 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1440092-9901-4eff-8a9d-6174b2dfdada-registry-tls\") pod \"image-registry-6796cf4b57-b9d8k\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:20.071003 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:20.070971 5088 scope.go:122] "RemoveContainer" containerID="77f9a154bdfbbdcca31431c7016e021fc1250443b58b9925789af279a4b64a0f" Oct 30 19:47:20.072013 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:20.071987 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:20.074303 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:20.074278 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:20.077522 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:20.077507 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7f4jl" Oct 30 19:47:20.138552 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:20.138527 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-vdklw_7a570505-5193-4564-b776-0f83df60d4e9/migrator/0.log" Oct 30 19:47:20.338087 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:20.338050 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-95bqt_a421de2b-dd41-4862-9849-0a4f0256af1b/kube-storage-version-migrator-operator/0.log" Oct 30 19:47:20.413205 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:20.413177 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6796cf4b57-b9d8k"] Oct 30 19:47:20.416302 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:20.416268 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1440092_9901_4eff_8a9d_6174b2dfdada.slice/crio-3c141530abb191bd2325d7b767fbac05d13e06a7f61cf87c4323fd1736e9190c WatchSource:0}: Error finding container 3c141530abb191bd2325d7b767fbac05d13e06a7f61cf87c4323fd1736e9190c: Status 404 returned error can't find the container with id 3c141530abb191bd2325d7b767fbac05d13e06a7f61cf87c4323fd1736e9190c Oct 30 19:47:20.457079 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:20.457056 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5bc64fc876-25g7d"] Oct 30 19:47:20.458384 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:20.458360 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a8ef63e_bcc4_4b71_b345_3fcbc48f4396.slice/crio-275923771ca2752326255995ad1da4193d0a2dba0490cd8d15fb15d9a4286408 WatchSource:0}: Error finding container 275923771ca2752326255995ad1da4193d0a2dba0490cd8d15fb15d9a4286408: Status 404 returned error can't find the container with id 275923771ca2752326255995ad1da4193d0a2dba0490cd8d15fb15d9a4286408 Oct 30 19:47:20.467023 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:20.466999 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7f4jl"] Oct 30 19:47:20.469369 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:20.469350 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a160cec7-bc1f-4202-8382-e84318122fd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8txdr\" (UID: \"a160cec7-bc1f-4202-8382-e84318122fd9\") " pod="openshift-insights/insights-runtime-extractor-8txdr" Oct 30 19:47:20.469444 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:20.469388 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1342ef67-5f63-47fb-a337-64c693e11228-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n75h8\" (UID: \"1342ef67-5f63-47fb-a337-64c693e11228\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n75h8" Oct 30 19:47:20.469444 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:20.469416 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/20f593c8-594e-4a64-9c5e-881311b269a3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qdv4s\" (UID: \"20f593c8-594e-4a64-9c5e-881311b269a3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qdv4s" Oct 30 19:47:20.469551 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:20.469441 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8e3ed9d-5156-4698-82c3-df72fa728f94-metrics-tls\") pod \"dns-default-ml5lp\" (UID: \"b8e3ed9d-5156-4698-82c3-df72fa728f94\") " pod="openshift-dns/dns-default-ml5lp" Oct 30 19:47:20.469551 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:20.469467 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/afb6fa1c-9722-42c4-a914-4941bcbcaaad-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-tv4xh\" (UID: \"afb6fa1c-9722-42c4-a914-4941bcbcaaad\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-tv4xh" Oct 30 19:47:20.469551 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:47:20.469511 5088 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:47:20.469551 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:20.469522 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41534930-917f-44ec-bccd-86d6c6fc7c70-metrics-certs\") pod \"router-default-79458f7dbc-vbb28\" (UID: \"41534930-917f-44ec-bccd-86d6c6fc7c70\") " pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:20.469800 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:47:20.469575 5088 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:47:20.469800 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:47:20.469571 5088 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a160cec7-bc1f-4202-8382-e84318122fd9-insights-runtime-extractor-tls podName:a160cec7-bc1f-4202-8382-e84318122fd9 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:21.469551429 +0000 UTC m=+2.882265203 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a160cec7-bc1f-4202-8382-e84318122fd9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8txdr" (UID: "a160cec7-bc1f-4202-8382-e84318122fd9") : secret "insights-runtime-extractor-tls" not found Oct 30 19:47:20.469800 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:20.469663 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41534930-917f-44ec-bccd-86d6c6fc7c70-service-ca-bundle\") pod \"router-default-79458f7dbc-vbb28\" (UID: \"41534930-917f-44ec-bccd-86d6c6fc7c70\") " pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:20.469800 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:47:20.469696 5088 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 19:47:20.469800 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:47:20.469720 5088 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8e3ed9d-5156-4698-82c3-df72fa728f94-metrics-tls podName:b8e3ed9d-5156-4698-82c3-df72fa728f94 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:21.46970606 +0000 UTC m=+2.882419829 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b8e3ed9d-5156-4698-82c3-df72fa728f94-metrics-tls") pod "dns-default-ml5lp" (UID: "b8e3ed9d-5156-4698-82c3-df72fa728f94") : secret "dns-default-metrics-tls" not found Oct 30 19:47:20.469800 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:47:20.469738 5088 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afb6fa1c-9722-42c4-a914-4941bcbcaaad-networking-console-plugin-cert podName:afb6fa1c-9722-42c4-a914-4941bcbcaaad nodeName:}" failed. No retries permitted until 2026-10-30 19:47:21.469729206 +0000 UTC m=+2.882442964 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/afb6fa1c-9722-42c4-a914-4941bcbcaaad-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-tv4xh" (UID: "afb6fa1c-9722-42c4-a914-4941bcbcaaad") : secret "networking-console-plugin-cert" not found Oct 30 19:47:20.470101 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:47:20.469806 5088 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/41534930-917f-44ec-bccd-86d6c6fc7c70-service-ca-bundle podName:41534930-917f-44ec-bccd-86d6c6fc7c70 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:21.469773686 +0000 UTC m=+2.882487462 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/41534930-917f-44ec-bccd-86d6c6fc7c70-service-ca-bundle") pod "router-default-79458f7dbc-vbb28" (UID: "41534930-917f-44ec-bccd-86d6c6fc7c70") : configmap references non-existent config key: service-ca.crt Oct 30 19:47:20.470101 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:47:20.469831 5088 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 19:47:20.470101 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:47:20.469871 5088 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1342ef67-5f63-47fb-a337-64c693e11228-samples-operator-tls podName:1342ef67-5f63-47fb-a337-64c693e11228 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:21.469860748 +0000 UTC m=+2.882574531 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1342ef67-5f63-47fb-a337-64c693e11228-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-n75h8" (UID: "1342ef67-5f63-47fb-a337-64c693e11228") : secret "samples-operator-tls" not found Oct 30 19:47:20.470823 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:20.470804 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57ddd2a0_f081_4f90_95a6_0f6f8df2f433.slice/crio-41da4d757092557f5df7b4ff9e7321437e02c9a983579d69c4f4374bf47e23c0 WatchSource:0}: Error finding container 41da4d757092557f5df7b4ff9e7321437e02c9a983579d69c4f4374bf47e23c0: Status 404 returned error can't find the container with id 41da4d757092557f5df7b4ff9e7321437e02c9a983579d69c4f4374bf47e23c0 Oct 30 19:47:20.472219 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:20.472202 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41534930-917f-44ec-bccd-86d6c6fc7c70-metrics-certs\") pod \"router-default-79458f7dbc-vbb28\" (UID: \"41534930-917f-44ec-bccd-86d6c6fc7c70\") " pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:20.472306 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:20.472204 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/20f593c8-594e-4a64-9c5e-881311b269a3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qdv4s\" (UID: \"20f593c8-594e-4a64-9c5e-881311b269a3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qdv4s" Oct 30 19:47:20.473318 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:20.473300 5088 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:47:20.515263 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:20.515239 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:20.671432 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:20.671391 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qdv4s" Oct 30 19:47:20.798525 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:20.798491 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" event={"ID":"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396","Type":"ContainerStarted","Data":"3afabe566656ee8e35be184b729246f5dfde6a0337aaedbc4f5b51f20a7bcc27"} Oct 30 19:47:20.799049 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:20.799025 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" event={"ID":"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396","Type":"ContainerStarted","Data":"275923771ca2752326255995ad1da4193d0a2dba0490cd8d15fb15d9a4286408"} Oct 30 19:47:20.802232 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:20.802212 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qdj4h_da850825-59ab-480f-9047-afd7d2bdc19f/console-operator/1.log" Oct 30 19:47:20.802432 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:20.802415 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-qdj4h" event={"ID":"da850825-59ab-480f-9047-afd7d2bdc19f","Type":"ContainerStarted","Data":"10b2c5c14be1ca865936b5e3c84ffa576d86403ffba0656607850a7aa4d7abfe"} Oct 30 19:47:20.804155 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:20.804127 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" event={"ID":"b1440092-9901-4eff-8a9d-6174b2dfdada","Type":"ContainerStarted","Data":"f3221a11e7e16a34abf16f1c11207c4ec32012fdfd813688f49c9bcec17a4675"} Oct 30 19:47:20.804270 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:20.804177 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" event={"ID":"b1440092-9901-4eff-8a9d-6174b2dfdada","Type":"ContainerStarted","Data":"3c141530abb191bd2325d7b767fbac05d13e06a7f61cf87c4323fd1736e9190c"} Oct 30 19:47:20.806098 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:20.805447 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7f4jl" event={"ID":"57ddd2a0-f081-4f90-95a6-0f6f8df2f433","Type":"ContainerStarted","Data":"41da4d757092557f5df7b4ff9e7321437e02c9a983579d69c4f4374bf47e23c0"} Oct 30 19:47:20.985092 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:20.985063 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-qdv4s"] Oct 30 19:47:20.988897 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:20.988871 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20f593c8_594e_4a64_9c5e_881311b269a3.slice/crio-221831c7a6c967544fb6f2e6ab33c350e9d696f077f081c17bb73b712cb95cfd WatchSource:0}: Error finding container 221831c7a6c967544fb6f2e6ab33c350e9d696f077f081c17bb73b712cb95cfd: Status 404 returned error can't find the container with id 221831c7a6c967544fb6f2e6ab33c350e9d696f077f081c17bb73b712cb95cfd Oct 30 19:47:21.481821 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:21.481798 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1342ef67-5f63-47fb-a337-64c693e11228-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n75h8\" (UID: \"1342ef67-5f63-47fb-a337-64c693e11228\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n75h8" Oct 30 19:47:21.481921 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:21.481833 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8e3ed9d-5156-4698-82c3-df72fa728f94-metrics-tls\") pod \"dns-default-ml5lp\" (UID: \"b8e3ed9d-5156-4698-82c3-df72fa728f94\") " pod="openshift-dns/dns-default-ml5lp" Oct 30 19:47:21.481921 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:21.481856 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/afb6fa1c-9722-42c4-a914-4941bcbcaaad-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-tv4xh\" (UID: \"afb6fa1c-9722-42c4-a914-4941bcbcaaad\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-tv4xh" Oct 30 19:47:21.481921 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:21.481903 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41534930-917f-44ec-bccd-86d6c6fc7c70-service-ca-bundle\") pod \"router-default-79458f7dbc-vbb28\" (UID: \"41534930-917f-44ec-bccd-86d6c6fc7c70\") " pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:21.482126 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:21.481928 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a160cec7-bc1f-4202-8382-e84318122fd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8txdr\" (UID: \"a160cec7-bc1f-4202-8382-e84318122fd9\") " pod="openshift-insights/insights-runtime-extractor-8txdr" Oct 30 19:47:21.482126 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:47:21.481948 5088 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:47:21.482126 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:47:21.482002 5088 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8e3ed9d-5156-4698-82c3-df72fa728f94-metrics-tls podName:b8e3ed9d-5156-4698-82c3-df72fa728f94 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:23.481988362 +0000 UTC m=+4.894702118 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b8e3ed9d-5156-4698-82c3-df72fa728f94-metrics-tls") pod "dns-default-ml5lp" (UID: "b8e3ed9d-5156-4698-82c3-df72fa728f94") : secret "dns-default-metrics-tls" not found Oct 30 19:47:21.482126 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:47:21.482027 5088 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 19:47:21.482126 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:47:21.482072 5088 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/41534930-917f-44ec-bccd-86d6c6fc7c70-service-ca-bundle podName:41534930-917f-44ec-bccd-86d6c6fc7c70 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:23.48205382 +0000 UTC m=+4.894767590 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/41534930-917f-44ec-bccd-86d6c6fc7c70-service-ca-bundle") pod "router-default-79458f7dbc-vbb28" (UID: "41534930-917f-44ec-bccd-86d6c6fc7c70") : configmap references non-existent config key: service-ca.crt Oct 30 19:47:21.482126 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:47:21.482097 5088 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afb6fa1c-9722-42c4-a914-4941bcbcaaad-networking-console-plugin-cert podName:afb6fa1c-9722-42c4-a914-4941bcbcaaad nodeName:}" failed. No retries permitted until 2026-10-30 19:47:23.482088236 +0000 UTC m=+4.894801995 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/afb6fa1c-9722-42c4-a914-4941bcbcaaad-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-tv4xh" (UID: "afb6fa1c-9722-42c4-a914-4941bcbcaaad") : secret "networking-console-plugin-cert" not found Oct 30 19:47:21.482402 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:47:21.482179 5088 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:47:21.482402 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:47:21.482220 5088 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a160cec7-bc1f-4202-8382-e84318122fd9-insights-runtime-extractor-tls podName:a160cec7-bc1f-4202-8382-e84318122fd9 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:23.482207944 +0000 UTC m=+4.894921714 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a160cec7-bc1f-4202-8382-e84318122fd9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8txdr" (UID: "a160cec7-bc1f-4202-8382-e84318122fd9") : secret "insights-runtime-extractor-tls" not found Oct 30 19:47:21.484188 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:21.484162 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1342ef67-5f63-47fb-a337-64c693e11228-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n75h8\" (UID: \"1342ef67-5f63-47fb-a337-64c693e11228\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n75h8" Oct 30 19:47:21.572083 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:21.572019 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n75h8" Oct 30 19:47:21.811192 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:21.811158 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7f4jl" event={"ID":"57ddd2a0-f081-4f90-95a6-0f6f8df2f433","Type":"ContainerStarted","Data":"44e48f07e34208308818119005d8056a90fd56376691c9a2cf2913cdd9fb9efa"} Oct 30 19:47:21.811654 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:21.811251 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7f4jl" event={"ID":"57ddd2a0-f081-4f90-95a6-0f6f8df2f433","Type":"ContainerStarted","Data":"9ed342e7c3d1f6e2623e1233e0db5074c89ccee901bf4411b7ddaa45e7099d94"} Oct 30 19:47:21.812547 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:21.812524 5088 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:47:21.812697 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:21.812590 5088 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:47:21.812697 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:21.812525 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qdv4s" event={"ID":"20f593c8-594e-4a64-9c5e-881311b269a3","Type":"ContainerStarted","Data":"221831c7a6c967544fb6f2e6ab33c350e9d696f077f081c17bb73b712cb95cfd"} Oct 30 19:47:21.876583 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:21.876504 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n75h8"] Oct 30 19:47:22.277180 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:22.277153 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:22.277419 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:22.277396 5088 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:47:22.365570 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:22.365538 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-djkzp" Oct 30 19:47:22.818276 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:22.818241 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n75h8" event={"ID":"1342ef67-5f63-47fb-a337-64c693e11228","Type":"ContainerStarted","Data":"396075a4a7bb617b9893c4c5dfd66549f7c46ce430497a08383b0cd0307f4580"} Oct 30 19:47:23.114950 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:23.114298 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:23.501833 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:23.501802 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8e3ed9d-5156-4698-82c3-df72fa728f94-metrics-tls\") pod \"dns-default-ml5lp\" (UID: \"b8e3ed9d-5156-4698-82c3-df72fa728f94\") " pod="openshift-dns/dns-default-ml5lp" Oct 30 19:47:23.501999 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:23.501846 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/afb6fa1c-9722-42c4-a914-4941bcbcaaad-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-tv4xh\" (UID: \"afb6fa1c-9722-42c4-a914-4941bcbcaaad\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-tv4xh" Oct 30 19:47:23.502071 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:23.502026 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41534930-917f-44ec-bccd-86d6c6fc7c70-service-ca-bundle\") pod \"router-default-79458f7dbc-vbb28\" (UID: \"41534930-917f-44ec-bccd-86d6c6fc7c70\") " pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:23.502124 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:23.502078 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a160cec7-bc1f-4202-8382-e84318122fd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8txdr\" (UID: \"a160cec7-bc1f-4202-8382-e84318122fd9\") " pod="openshift-insights/insights-runtime-extractor-8txdr" Oct 30 19:47:23.502228 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:47:23.502211 5088 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:47:23.502328 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:47:23.502215 5088 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/41534930-917f-44ec-bccd-86d6c6fc7c70-service-ca-bundle podName:41534930-917f-44ec-bccd-86d6c6fc7c70 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:27.502197097 +0000 UTC m=+8.914910874 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/41534930-917f-44ec-bccd-86d6c6fc7c70-service-ca-bundle") pod "router-default-79458f7dbc-vbb28" (UID: "41534930-917f-44ec-bccd-86d6c6fc7c70") : configmap references non-existent config key: service-ca.crt Oct 30 19:47:23.502678 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:47:23.502303 5088 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a160cec7-bc1f-4202-8382-e84318122fd9-insights-runtime-extractor-tls podName:a160cec7-bc1f-4202-8382-e84318122fd9 nodeName:}" failed. No retries permitted until 2026-10-30 19:47:27.502256451 +0000 UTC m=+8.914970221 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a160cec7-bc1f-4202-8382-e84318122fd9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8txdr" (UID: "a160cec7-bc1f-4202-8382-e84318122fd9") : secret "insights-runtime-extractor-tls" not found Oct 30 19:47:23.504344 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:23.504316 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8e3ed9d-5156-4698-82c3-df72fa728f94-metrics-tls\") pod \"dns-default-ml5lp\" (UID: \"b8e3ed9d-5156-4698-82c3-df72fa728f94\") " pod="openshift-dns/dns-default-ml5lp" Oct 30 19:47:23.504436 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:23.504383 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/afb6fa1c-9722-42c4-a914-4941bcbcaaad-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-tv4xh\" (UID: \"afb6fa1c-9722-42c4-a914-4941bcbcaaad\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-tv4xh" Oct 30 19:47:23.674402 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:23.674371 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-tv4xh" Oct 30 19:47:23.682042 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:23.682016 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ml5lp" Oct 30 19:47:23.824389 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:23.824351 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qdv4s" event={"ID":"20f593c8-594e-4a64-9c5e-881311b269a3","Type":"ContainerStarted","Data":"b7d1f045f4b3de15e4f92730a635c10f69796349ff6e49efe79a89ac01f26743"} Oct 30 19:47:23.826486 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:23.826457 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n75h8" event={"ID":"1342ef67-5f63-47fb-a337-64c693e11228","Type":"ContainerStarted","Data":"af36e798a6b253cf72b72eefb5d041932c99de1f03a13ce39f1f9451aa64c73f"} Oct 30 19:47:23.826590 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:23.826511 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n75h8" event={"ID":"1342ef67-5f63-47fb-a337-64c693e11228","Type":"ContainerStarted","Data":"374237f20c96919b763f4306d7a6c9cd7b838b3d3a86c71fe6cae4e3baa58034"} Oct 30 19:47:23.976007 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:23.975976 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-tv4xh"] Oct 30 19:47:23.978851 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:23.978826 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafb6fa1c_9722_42c4_a914_4941bcbcaaad.slice/crio-bcd619944ce28b37961887c2fa64e07cbe5766c6bceccb296ad657277894706f WatchSource:0}: Error finding container bcd619944ce28b37961887c2fa64e07cbe5766c6bceccb296ad657277894706f: Status 404 returned error can't find the container with id bcd619944ce28b37961887c2fa64e07cbe5766c6bceccb296ad657277894706f Oct 30 19:47:23.995279 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:23.995261 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ml5lp"] Oct 30 19:47:23.997542 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:23.997388 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8e3ed9d_5156_4698_82c3_df72fa728f94.slice/crio-9545a93fc17fa96fda5eee17f1fd30adc9970be3c3e064db2717e2be9bca1631 WatchSource:0}: Error finding container 9545a93fc17fa96fda5eee17f1fd30adc9970be3c3e064db2717e2be9bca1631: Status 404 returned error can't find the container with id 9545a93fc17fa96fda5eee17f1fd30adc9970be3c3e064db2717e2be9bca1631 Oct 30 19:47:24.591813 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:24.591777 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-qdj4h" Oct 30 19:47:24.591991 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:24.591929 5088 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:47:24.597287 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:24.597260 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-qdj4h" Oct 30 19:47:24.833797 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:24.833723 5088 generic.go:356] "Generic (PLEG): container finished" podID="d78606e4-ccb9-421a-8ae5-8fbecbcd3831" containerID="0311b6a67457029147850bb5d461ad63c35e9cdc77c6212ede73fc2499803caf" exitCode=0 Oct 30 19:47:24.833797 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:24.833784 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-bnp7s" event={"ID":"d78606e4-ccb9-421a-8ae5-8fbecbcd3831","Type":"ContainerDied","Data":"0311b6a67457029147850bb5d461ad63c35e9cdc77c6212ede73fc2499803caf"} Oct 30 19:47:24.834280 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:24.834148 5088 scope.go:122] "RemoveContainer" containerID="0311b6a67457029147850bb5d461ad63c35e9cdc77c6212ede73fc2499803caf" Oct 30 19:47:24.836132 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:24.836060 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-tv4xh" event={"ID":"afb6fa1c-9722-42c4-a914-4941bcbcaaad","Type":"ContainerStarted","Data":"bcd619944ce28b37961887c2fa64e07cbe5766c6bceccb296ad657277894706f"} Oct 30 19:47:24.837582 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:24.837551 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ml5lp" event={"ID":"b8e3ed9d-5156-4698-82c3-df72fa728f94","Type":"ContainerStarted","Data":"9545a93fc17fa96fda5eee17f1fd30adc9970be3c3e064db2717e2be9bca1631"} Oct 30 19:47:24.841697 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:24.841672 5088 generic.go:356] "Generic (PLEG): container finished" podID="a421de2b-dd41-4862-9849-0a4f0256af1b" containerID="068a6c75ad9f87aab05aee31cfa9d24e47fdcb138ed92b01e8b68b4f1c0c8359" exitCode=0 Oct 30 19:47:24.841789 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:24.841749 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95bqt" event={"ID":"a421de2b-dd41-4862-9849-0a4f0256af1b","Type":"ContainerDied","Data":"068a6c75ad9f87aab05aee31cfa9d24e47fdcb138ed92b01e8b68b4f1c0c8359"} Oct 30 19:47:24.842512 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:24.842050 5088 scope.go:122] "RemoveContainer" containerID="068a6c75ad9f87aab05aee31cfa9d24e47fdcb138ed92b01e8b68b4f1c0c8359" Oct 30 19:47:24.844957 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:24.844912 5088 generic.go:356] "Generic (PLEG): container finished" podID="74b7c739-483f-4756-b97b-e65d37387b17" containerID="1f2235f3507dd80283f2bd971b8b3cd676d6501ed14e0ed9485dc488cdf82209" exitCode=0 Oct 30 19:47:24.845416 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:24.845392 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6wqfc" event={"ID":"74b7c739-483f-4756-b97b-e65d37387b17","Type":"ContainerDied","Data":"1f2235f3507dd80283f2bd971b8b3cd676d6501ed14e0ed9485dc488cdf82209"} Oct 30 19:47:24.845676 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:24.845656 5088 scope.go:122] "RemoveContainer" containerID="1f2235f3507dd80283f2bd971b8b3cd676d6501ed14e0ed9485dc488cdf82209" Oct 30 19:47:25.857234 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:25.857197 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-95bqt" event={"ID":"a421de2b-dd41-4862-9849-0a4f0256af1b","Type":"ContainerStarted","Data":"3e893837c94553fc1e0c46beccebaeba388038dc4ef8fa9cf44c45158a99d0e0"} Oct 30 19:47:25.859418 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:25.859389 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6wqfc" event={"ID":"74b7c739-483f-4756-b97b-e65d37387b17","Type":"ContainerStarted","Data":"c0234d955ab04bd4df7e5211186c626b80e7665b1a32b36d9373ed5a71ac14cd"} Oct 30 19:47:25.861408 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:25.861386 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-bnp7s" event={"ID":"d78606e4-ccb9-421a-8ae5-8fbecbcd3831","Type":"ContainerStarted","Data":"b6e8c79e765c82ab981316b7753653230e2b3e74e996f9691637b377afa3bb54"} Oct 30 19:47:25.862974 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:25.862951 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-tv4xh" event={"ID":"afb6fa1c-9722-42c4-a914-4941bcbcaaad","Type":"ContainerStarted","Data":"032670ed1b735400b0d0256a3b04104ce78b6c61901c2ded06d97a51dae7e1b9"} Oct 30 19:47:26.428038 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:26.427964 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:26.868980 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:26.868944 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ml5lp" event={"ID":"b8e3ed9d-5156-4698-82c3-df72fa728f94","Type":"ContainerStarted","Data":"f47e65b0df272df9d1595d3c50ce39dcc2fc6aad41021717359767478add3a9c"} Oct 30 19:47:26.869433 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:26.869006 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ml5lp" event={"ID":"b8e3ed9d-5156-4698-82c3-df72fa728f94","Type":"ContainerStarted","Data":"b67f08fed8d1382f8fc577e03ca09072e098d6736534a9c3f695b25eb0ec1d21"} Oct 30 19:47:26.869433 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:26.869119 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ml5lp" Oct 30 19:47:27.441396 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:27.441361 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kvw9t" Oct 30 19:47:27.548883 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:27.548856 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a160cec7-bc1f-4202-8382-e84318122fd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8txdr\" (UID: \"a160cec7-bc1f-4202-8382-e84318122fd9\") " pod="openshift-insights/insights-runtime-extractor-8txdr" Oct 30 19:47:27.549033 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:27.549015 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41534930-917f-44ec-bccd-86d6c6fc7c70-service-ca-bundle\") pod \"router-default-79458f7dbc-vbb28\" (UID: \"41534930-917f-44ec-bccd-86d6c6fc7c70\") " pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:27.549586 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:27.549558 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41534930-917f-44ec-bccd-86d6c6fc7c70-service-ca-bundle\") pod \"router-default-79458f7dbc-vbb28\" (UID: \"41534930-917f-44ec-bccd-86d6c6fc7c70\") " pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:27.550988 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:27.550967 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a160cec7-bc1f-4202-8382-e84318122fd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8txdr\" (UID: \"a160cec7-bc1f-4202-8382-e84318122fd9\") " pod="openshift-insights/insights-runtime-extractor-8txdr" Oct 30 19:47:27.571080 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:27.571059 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:27.584267 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:27.584246 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8txdr" Oct 30 19:47:27.717486 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:27.717428 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xpjpn" Oct 30 19:47:27.890739 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:27.890707 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-79458f7dbc-vbb28"] Oct 30 19:47:27.893150 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:27.893126 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41534930_917f_44ec_bccd_86d6c6fc7c70.slice/crio-5a404fffafea0dce4c8c2255e938a2aec356a59cc0c73983e224f13679917a31 WatchSource:0}: Error finding container 5a404fffafea0dce4c8c2255e938a2aec356a59cc0c73983e224f13679917a31: Status 404 returned error can't find the container with id 5a404fffafea0dce4c8c2255e938a2aec356a59cc0c73983e224f13679917a31 Oct 30 19:47:27.913683 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:27.913664 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8txdr"] Oct 30 19:47:27.922685 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:27.922665 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda160cec7_bc1f_4202_8382_e84318122fd9.slice/crio-018ef7cd3bfe81d5d63e93f3b558a82a4aa0ca03e2f5bd86b88e402090c47dc7 WatchSource:0}: Error finding container 018ef7cd3bfe81d5d63e93f3b558a82a4aa0ca03e2f5bd86b88e402090c47dc7: Status 404 returned error can't find the container with id 018ef7cd3bfe81d5d63e93f3b558a82a4aa0ca03e2f5bd86b88e402090c47dc7 Oct 30 19:47:28.876921 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:28.876836 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-79458f7dbc-vbb28" event={"ID":"41534930-917f-44ec-bccd-86d6c6fc7c70","Type":"ContainerStarted","Data":"6da9d3a48148498e38bd19ea39d31f9037424215dcbae3feec3ce4fb2b4f1004"} Oct 30 19:47:28.876921 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:28.876892 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-79458f7dbc-vbb28" event={"ID":"41534930-917f-44ec-bccd-86d6c6fc7c70","Type":"ContainerStarted","Data":"5a404fffafea0dce4c8c2255e938a2aec356a59cc0c73983e224f13679917a31"} Oct 30 19:47:28.878532 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:28.878507 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8txdr" event={"ID":"a160cec7-bc1f-4202-8382-e84318122fd9","Type":"ContainerStarted","Data":"9c3f1da94ea0e4c5d5f0edd5bc34c501ae23d69abf5f6696680eff2b4f8e23ea"} Oct 30 19:47:28.878636 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:28.878555 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8txdr" event={"ID":"a160cec7-bc1f-4202-8382-e84318122fd9","Type":"ContainerStarted","Data":"9e3f8eb3145ea0f8ae97b0c45567bec255a3888b6fd1631632763c5f9476e508"} Oct 30 19:47:28.878636 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:28.878581 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8txdr" event={"ID":"a160cec7-bc1f-4202-8382-e84318122fd9","Type":"ContainerStarted","Data":"018ef7cd3bfe81d5d63e93f3b558a82a4aa0ca03e2f5bd86b88e402090c47dc7"} Oct 30 19:47:29.571794 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:29.571758 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:29.574749 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:29.574726 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:30.889286 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:30.889253 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8txdr" event={"ID":"a160cec7-bc1f-4202-8382-e84318122fd9","Type":"ContainerStarted","Data":"86ad7e3f85fc0cb227a58518877c4a51d5056cc46db33997bc217eb1220e1029"} Oct 30 19:47:30.889798 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:30.889307 5088 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:47:34.687949 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:34.687914 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:34.688518 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:34.688043 5088 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:47:34.689256 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:34.689234 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-79458f7dbc-vbb28" Oct 30 19:47:36.432490 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.432458 5088 patch_prober.go:29] interesting pod/image-registry-5bc64fc876-25g7d container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 19:47:36.432989 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.432553 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" podUID="9a8ef63e-bcc4-4b71-b345-3fcbc48f4396" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:47:36.778430 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.778392 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-nmpzq-wm6lg"] Oct 30 19:47:36.783308 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.783287 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-nmpzq-wm6lg" Oct 30 19:47:36.815922 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.815892 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-nmpzq-wm6lg"] Oct 30 19:47:36.823838 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.823816 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-796595f99b-8nbrk"] Oct 30 19:47:36.827105 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.827080 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nw8kx"] Oct 30 19:47:36.827229 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.827166 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-796595f99b-8nbrk" Oct 30 19:47:36.830539 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.830518 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-mbstf"] Oct 30 19:47:36.830672 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.830600 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nw8kx" Oct 30 19:47:36.833273 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.833219 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w68mv\" (UniqueName: \"kubernetes.io/projected/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-kube-api-access-w68mv\") pod \"console-796595f99b-8nbrk\" (UID: \"a16466d6-9ae1-4791-ad64-092e8d1cfa6f\") " pod="openshift-console/console-796595f99b-8nbrk" Oct 30 19:47:36.833369 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.833297 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/269934ab-9b1e-4dab-8daf-86bd074bc85a-archives-path\") pod \"periodic-gathering-nmpzq-wm6lg\" (UID: \"269934ab-9b1e-4dab-8daf-86bd074bc85a\") " pod="openshift-insights/periodic-gathering-nmpzq-wm6lg" Oct 30 19:47:36.833369 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.833323 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/269934ab-9b1e-4dab-8daf-86bd074bc85a-trusted-ca-bundle\") pod \"periodic-gathering-nmpzq-wm6lg\" (UID: \"269934ab-9b1e-4dab-8daf-86bd074bc85a\") " pod="openshift-insights/periodic-gathering-nmpzq-wm6lg" Oct 30 19:47:36.833369 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.833346 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-console-serving-cert\") pod \"console-796595f99b-8nbrk\" (UID: \"a16466d6-9ae1-4791-ad64-092e8d1cfa6f\") " pod="openshift-console/console-796595f99b-8nbrk" Oct 30 19:47:36.833548 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.833504 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns9wx\" (UniqueName: \"kubernetes.io/projected/269934ab-9b1e-4dab-8daf-86bd074bc85a-kube-api-access-ns9wx\") pod \"periodic-gathering-nmpzq-wm6lg\" (UID: \"269934ab-9b1e-4dab-8daf-86bd074bc85a\") " pod="openshift-insights/periodic-gathering-nmpzq-wm6lg" Oct 30 19:47:36.833600 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.833578 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-tmp\") pod \"console-796595f99b-8nbrk\" (UID: \"a16466d6-9ae1-4791-ad64-092e8d1cfa6f\") " pod="openshift-console/console-796595f99b-8nbrk" Oct 30 19:47:36.833684 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.833604 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7ef6ce3a-0191-4e2b-a483-3c1bde6e2444-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-nw8kx\" (UID: \"7ef6ce3a-0191-4e2b-a483-3c1bde6e2444\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nw8kx" Oct 30 19:47:36.833684 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.833657 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/269934ab-9b1e-4dab-8daf-86bd074bc85a-service-ca-bundle\") pod \"periodic-gathering-nmpzq-wm6lg\" (UID: \"269934ab-9b1e-4dab-8daf-86bd074bc85a\") " pod="openshift-insights/periodic-gathering-nmpzq-wm6lg" Oct 30 19:47:36.833789 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.833681 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-oauth-serving-cert\") pod \"console-796595f99b-8nbrk\" (UID: \"a16466d6-9ae1-4791-ad64-092e8d1cfa6f\") " pod="openshift-console/console-796595f99b-8nbrk" Oct 30 19:47:36.833789 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.833716 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-console-config\") pod \"console-796595f99b-8nbrk\" (UID: \"a16466d6-9ae1-4791-ad64-092e8d1cfa6f\") " pod="openshift-console/console-796595f99b-8nbrk" Oct 30 19:47:36.833789 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.833767 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-console-oauth-config\") pod \"console-796595f99b-8nbrk\" (UID: \"a16466d6-9ae1-4791-ad64-092e8d1cfa6f\") " pod="openshift-console/console-796595f99b-8nbrk" Oct 30 19:47:36.833912 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.833804 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-service-ca\") pod \"console-796595f99b-8nbrk\" (UID: \"a16466d6-9ae1-4791-ad64-092e8d1cfa6f\") " pod="openshift-console/console-796595f99b-8nbrk" Oct 30 19:47:36.836251 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.836229 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-mbstf" Oct 30 19:47:36.848705 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.848677 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nw8kx"] Oct 30 19:47:36.854893 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.854043 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 19:47:36.854893 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.854159 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 19:47:36.857990 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.856873 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 19:47:36.857990 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.856934 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 19:47:36.857990 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.857488 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rgk57\"" Oct 30 19:47:36.857990 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.857564 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 19:47:36.857990 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.857654 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 19:47:36.857990 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.857705 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 19:47:36.859946 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.859920 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-796595f99b-8nbrk"] Oct 30 19:47:36.861489 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.861453 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-mbstf"] Oct 30 19:47:36.867759 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.867741 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6796cf4b57-b9d8k"] Oct 30 19:47:36.872933 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.872900 5088 patch_prober.go:29] interesting pod/image-registry-6796cf4b57-b9d8k container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 19:47:36.873031 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.872960 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" podUID="b1440092-9901-4eff-8a9d-6174b2dfdada" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:47:36.874322 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.874305 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ml5lp" Oct 30 19:47:36.934543 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.934514 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/269934ab-9b1e-4dab-8daf-86bd074bc85a-service-ca-bundle\") pod \"periodic-gathering-nmpzq-wm6lg\" (UID: \"269934ab-9b1e-4dab-8daf-86bd074bc85a\") " pod="openshift-insights/periodic-gathering-nmpzq-wm6lg" Oct 30 19:47:36.934543 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.934543 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-oauth-serving-cert\") pod \"console-796595f99b-8nbrk\" (UID: \"a16466d6-9ae1-4791-ad64-092e8d1cfa6f\") " pod="openshift-console/console-796595f99b-8nbrk" Oct 30 19:47:36.934775 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.934572 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-console-config\") pod \"console-796595f99b-8nbrk\" (UID: \"a16466d6-9ae1-4791-ad64-092e8d1cfa6f\") " pod="openshift-console/console-796595f99b-8nbrk" Oct 30 19:47:36.934775 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.934600 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-console-oauth-config\") pod \"console-796595f99b-8nbrk\" (UID: \"a16466d6-9ae1-4791-ad64-092e8d1cfa6f\") " pod="openshift-console/console-796595f99b-8nbrk" Oct 30 19:47:36.934775 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.934638 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfrc5\" (UniqueName: \"kubernetes.io/projected/8d6ab503-19e8-4585-b7c7-612f33db3e07-kube-api-access-bfrc5\") pod \"downloads-6dd789c8b7-mbstf\" (UID: \"8d6ab503-19e8-4585-b7c7-612f33db3e07\") " pod="openshift-console/downloads-6dd789c8b7-mbstf" Oct 30 19:47:36.934938 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.934808 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-service-ca\") pod \"console-796595f99b-8nbrk\" (UID: \"a16466d6-9ae1-4791-ad64-092e8d1cfa6f\") " pod="openshift-console/console-796595f99b-8nbrk" Oct 30 19:47:36.934938 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.934854 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w68mv\" (UniqueName: \"kubernetes.io/projected/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-kube-api-access-w68mv\") pod \"console-796595f99b-8nbrk\" (UID: \"a16466d6-9ae1-4791-ad64-092e8d1cfa6f\") " pod="openshift-console/console-796595f99b-8nbrk" Oct 30 19:47:36.934938 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.934914 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/269934ab-9b1e-4dab-8daf-86bd074bc85a-archives-path\") pod \"periodic-gathering-nmpzq-wm6lg\" (UID: \"269934ab-9b1e-4dab-8daf-86bd074bc85a\") " pod="openshift-insights/periodic-gathering-nmpzq-wm6lg" Oct 30 19:47:36.935100 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.934942 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/269934ab-9b1e-4dab-8daf-86bd074bc85a-trusted-ca-bundle\") pod \"periodic-gathering-nmpzq-wm6lg\" (UID: \"269934ab-9b1e-4dab-8daf-86bd074bc85a\") " pod="openshift-insights/periodic-gathering-nmpzq-wm6lg" Oct 30 19:47:36.935100 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.934984 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-console-serving-cert\") pod \"console-796595f99b-8nbrk\" (UID: \"a16466d6-9ae1-4791-ad64-092e8d1cfa6f\") " pod="openshift-console/console-796595f99b-8nbrk" Oct 30 19:47:36.935100 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.935018 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d6ab503-19e8-4585-b7c7-612f33db3e07-tmp\") pod \"downloads-6dd789c8b7-mbstf\" (UID: \"8d6ab503-19e8-4585-b7c7-612f33db3e07\") " pod="openshift-console/downloads-6dd789c8b7-mbstf" Oct 30 19:47:36.935100 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.935095 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns9wx\" (UniqueName: \"kubernetes.io/projected/269934ab-9b1e-4dab-8daf-86bd074bc85a-kube-api-access-ns9wx\") pod \"periodic-gathering-nmpzq-wm6lg\" (UID: \"269934ab-9b1e-4dab-8daf-86bd074bc85a\") " pod="openshift-insights/periodic-gathering-nmpzq-wm6lg" Oct 30 19:47:36.935451 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.935128 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-tmp\") pod \"console-796595f99b-8nbrk\" (UID: \"a16466d6-9ae1-4791-ad64-092e8d1cfa6f\") " pod="openshift-console/console-796595f99b-8nbrk" Oct 30 19:47:36.935451 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.935180 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7ef6ce3a-0191-4e2b-a483-3c1bde6e2444-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-nw8kx\" (UID: \"7ef6ce3a-0191-4e2b-a483-3c1bde6e2444\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nw8kx" Oct 30 19:47:36.935451 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.935352 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/269934ab-9b1e-4dab-8daf-86bd074bc85a-archives-path\") pod \"periodic-gathering-nmpzq-wm6lg\" (UID: \"269934ab-9b1e-4dab-8daf-86bd074bc85a\") " pod="openshift-insights/periodic-gathering-nmpzq-wm6lg" Oct 30 19:47:36.935451 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.935408 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-oauth-serving-cert\") pod \"console-796595f99b-8nbrk\" (UID: \"a16466d6-9ae1-4791-ad64-092e8d1cfa6f\") " pod="openshift-console/console-796595f99b-8nbrk" Oct 30 19:47:36.935451 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.935408 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-console-config\") pod \"console-796595f99b-8nbrk\" (UID: \"a16466d6-9ae1-4791-ad64-092e8d1cfa6f\") " pod="openshift-console/console-796595f99b-8nbrk" Oct 30 19:47:36.935917 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.935788 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-tmp\") pod \"console-796595f99b-8nbrk\" (UID: \"a16466d6-9ae1-4791-ad64-092e8d1cfa6f\") " pod="openshift-console/console-796595f99b-8nbrk" Oct 30 19:47:36.935917 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.935829 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/269934ab-9b1e-4dab-8daf-86bd074bc85a-service-ca-bundle\") pod \"periodic-gathering-nmpzq-wm6lg\" (UID: \"269934ab-9b1e-4dab-8daf-86bd074bc85a\") " pod="openshift-insights/periodic-gathering-nmpzq-wm6lg" Oct 30 19:47:36.936386 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.936328 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-service-ca\") pod \"console-796595f99b-8nbrk\" (UID: \"a16466d6-9ae1-4791-ad64-092e8d1cfa6f\") " pod="openshift-console/console-796595f99b-8nbrk" Oct 30 19:47:36.936493 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.936443 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/269934ab-9b1e-4dab-8daf-86bd074bc85a-trusted-ca-bundle\") pod \"periodic-gathering-nmpzq-wm6lg\" (UID: \"269934ab-9b1e-4dab-8daf-86bd074bc85a\") " pod="openshift-insights/periodic-gathering-nmpzq-wm6lg" Oct 30 19:47:36.938337 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.938312 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-console-serving-cert\") pod \"console-796595f99b-8nbrk\" (UID: \"a16466d6-9ae1-4791-ad64-092e8d1cfa6f\") " pod="openshift-console/console-796595f99b-8nbrk" Oct 30 19:47:36.938774 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.938754 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-console-oauth-config\") pod \"console-796595f99b-8nbrk\" (UID: \"a16466d6-9ae1-4791-ad64-092e8d1cfa6f\") " pod="openshift-console/console-796595f99b-8nbrk" Oct 30 19:47:36.954527 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.954507 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-lx2lw\"" Oct 30 19:47:36.954671 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.954656 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-jg67s\"" Oct 30 19:47:36.954671 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.954665 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 19:47:36.957670 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.957651 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7ef6ce3a-0191-4e2b-a483-3c1bde6e2444-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-nw8kx\" (UID: \"7ef6ce3a-0191-4e2b-a483-3c1bde6e2444\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nw8kx" Oct 30 19:47:36.964089 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.964068 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-857cdcc958-b2kxp"] Oct 30 19:47:36.968636 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.968593 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-857cdcc958-b2kxp" Oct 30 19:47:36.972292 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.972273 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns9wx\" (UniqueName: \"kubernetes.io/projected/269934ab-9b1e-4dab-8daf-86bd074bc85a-kube-api-access-ns9wx\") pod \"periodic-gathering-nmpzq-wm6lg\" (UID: \"269934ab-9b1e-4dab-8daf-86bd074bc85a\") " pod="openshift-insights/periodic-gathering-nmpzq-wm6lg" Oct 30 19:47:36.976021 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.975994 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w68mv\" (UniqueName: \"kubernetes.io/projected/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-kube-api-access-w68mv\") pod \"console-796595f99b-8nbrk\" (UID: \"a16466d6-9ae1-4791-ad64-092e8d1cfa6f\") " pod="openshift-console/console-796595f99b-8nbrk" Oct 30 19:47:36.993438 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:36.993419 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-857cdcc958-b2kxp"] Oct 30 19:47:37.035846 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.035794 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7a81c307-a7a7-492b-bce4-cca4b59e1de4-trusted-ca\") pod \"image-registry-857cdcc958-b2kxp\" (UID: \"7a81c307-a7a7-492b-bce4-cca4b59e1de4\") " pod="openshift-image-registry/image-registry-857cdcc958-b2kxp" Oct 30 19:47:37.035846 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.035824 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7a81c307-a7a7-492b-bce4-cca4b59e1de4-registry-certificates\") pod \"image-registry-857cdcc958-b2kxp\" (UID: \"7a81c307-a7a7-492b-bce4-cca4b59e1de4\") " pod="openshift-image-registry/image-registry-857cdcc958-b2kxp" Oct 30 19:47:37.035846 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.035843 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7a81c307-a7a7-492b-bce4-cca4b59e1de4-registry-tls\") pod \"image-registry-857cdcc958-b2kxp\" (UID: \"7a81c307-a7a7-492b-bce4-cca4b59e1de4\") " pod="openshift-image-registry/image-registry-857cdcc958-b2kxp" Oct 30 19:47:37.036071 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.035980 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7a81c307-a7a7-492b-bce4-cca4b59e1de4-ca-trust-extracted\") pod \"image-registry-857cdcc958-b2kxp\" (UID: \"7a81c307-a7a7-492b-bce4-cca4b59e1de4\") " pod="openshift-image-registry/image-registry-857cdcc958-b2kxp" Oct 30 19:47:37.036071 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.036015 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7a81c307-a7a7-492b-bce4-cca4b59e1de4-installation-pull-secrets\") pod \"image-registry-857cdcc958-b2kxp\" (UID: \"7a81c307-a7a7-492b-bce4-cca4b59e1de4\") " pod="openshift-image-registry/image-registry-857cdcc958-b2kxp" Oct 30 19:47:37.036071 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.036039 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzwfs\" (UniqueName: \"kubernetes.io/projected/7a81c307-a7a7-492b-bce4-cca4b59e1de4-kube-api-access-fzwfs\") pod \"image-registry-857cdcc958-b2kxp\" (UID: \"7a81c307-a7a7-492b-bce4-cca4b59e1de4\") " pod="openshift-image-registry/image-registry-857cdcc958-b2kxp" Oct 30 19:47:37.036221 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.036137 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7a81c307-a7a7-492b-bce4-cca4b59e1de4-bound-sa-token\") pod \"image-registry-857cdcc958-b2kxp\" (UID: \"7a81c307-a7a7-492b-bce4-cca4b59e1de4\") " pod="openshift-image-registry/image-registry-857cdcc958-b2kxp" Oct 30 19:47:37.036221 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.036190 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfrc5\" (UniqueName: \"kubernetes.io/projected/8d6ab503-19e8-4585-b7c7-612f33db3e07-kube-api-access-bfrc5\") pod \"downloads-6dd789c8b7-mbstf\" (UID: \"8d6ab503-19e8-4585-b7c7-612f33db3e07\") " pod="openshift-console/downloads-6dd789c8b7-mbstf" Oct 30 19:47:37.036318 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.036247 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7a81c307-a7a7-492b-bce4-cca4b59e1de4-image-registry-private-configuration\") pod \"image-registry-857cdcc958-b2kxp\" (UID: \"7a81c307-a7a7-492b-bce4-cca4b59e1de4\") " pod="openshift-image-registry/image-registry-857cdcc958-b2kxp" Oct 30 19:47:37.036318 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.036302 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d6ab503-19e8-4585-b7c7-612f33db3e07-tmp\") pod \"downloads-6dd789c8b7-mbstf\" (UID: \"8d6ab503-19e8-4585-b7c7-612f33db3e07\") " pod="openshift-console/downloads-6dd789c8b7-mbstf" Oct 30 19:47:37.036666 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.036647 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d6ab503-19e8-4585-b7c7-612f33db3e07-tmp\") pod \"downloads-6dd789c8b7-mbstf\" (UID: \"8d6ab503-19e8-4585-b7c7-612f33db3e07\") " pod="openshift-console/downloads-6dd789c8b7-mbstf" Oct 30 19:47:37.068394 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.068376 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfrc5\" (UniqueName: \"kubernetes.io/projected/8d6ab503-19e8-4585-b7c7-612f33db3e07-kube-api-access-bfrc5\") pod \"downloads-6dd789c8b7-mbstf\" (UID: \"8d6ab503-19e8-4585-b7c7-612f33db3e07\") " pod="openshift-console/downloads-6dd789c8b7-mbstf" Oct 30 19:47:37.094578 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.094558 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-nmpzq-wm6lg" Oct 30 19:47:37.137025 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.136984 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7a81c307-a7a7-492b-bce4-cca4b59e1de4-ca-trust-extracted\") pod \"image-registry-857cdcc958-b2kxp\" (UID: \"7a81c307-a7a7-492b-bce4-cca4b59e1de4\") " pod="openshift-image-registry/image-registry-857cdcc958-b2kxp" Oct 30 19:47:37.137195 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.137031 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7a81c307-a7a7-492b-bce4-cca4b59e1de4-installation-pull-secrets\") pod \"image-registry-857cdcc958-b2kxp\" (UID: \"7a81c307-a7a7-492b-bce4-cca4b59e1de4\") " pod="openshift-image-registry/image-registry-857cdcc958-b2kxp" Oct 30 19:47:37.137195 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.137059 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzwfs\" (UniqueName: \"kubernetes.io/projected/7a81c307-a7a7-492b-bce4-cca4b59e1de4-kube-api-access-fzwfs\") pod \"image-registry-857cdcc958-b2kxp\" (UID: \"7a81c307-a7a7-492b-bce4-cca4b59e1de4\") " pod="openshift-image-registry/image-registry-857cdcc958-b2kxp" Oct 30 19:47:37.137195 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.137087 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7a81c307-a7a7-492b-bce4-cca4b59e1de4-bound-sa-token\") pod \"image-registry-857cdcc958-b2kxp\" (UID: \"7a81c307-a7a7-492b-bce4-cca4b59e1de4\") " pod="openshift-image-registry/image-registry-857cdcc958-b2kxp" Oct 30 19:47:37.137347 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.137291 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7a81c307-a7a7-492b-bce4-cca4b59e1de4-image-registry-private-configuration\") pod \"image-registry-857cdcc958-b2kxp\" (UID: \"7a81c307-a7a7-492b-bce4-cca4b59e1de4\") " pod="openshift-image-registry/image-registry-857cdcc958-b2kxp" Oct 30 19:47:37.137400 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.137380 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7a81c307-a7a7-492b-bce4-cca4b59e1de4-trusted-ca\") pod \"image-registry-857cdcc958-b2kxp\" (UID: \"7a81c307-a7a7-492b-bce4-cca4b59e1de4\") " pod="openshift-image-registry/image-registry-857cdcc958-b2kxp" Oct 30 19:47:37.137449 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.137396 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7a81c307-a7a7-492b-bce4-cca4b59e1de4-ca-trust-extracted\") pod \"image-registry-857cdcc958-b2kxp\" (UID: \"7a81c307-a7a7-492b-bce4-cca4b59e1de4\") " pod="openshift-image-registry/image-registry-857cdcc958-b2kxp" Oct 30 19:47:37.137449 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.137418 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7a81c307-a7a7-492b-bce4-cca4b59e1de4-registry-certificates\") pod \"image-registry-857cdcc958-b2kxp\" (UID: \"7a81c307-a7a7-492b-bce4-cca4b59e1de4\") " pod="openshift-image-registry/image-registry-857cdcc958-b2kxp" Oct 30 19:47:37.138658 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.138223 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7a81c307-a7a7-492b-bce4-cca4b59e1de4-registry-certificates\") pod \"image-registry-857cdcc958-b2kxp\" (UID: \"7a81c307-a7a7-492b-bce4-cca4b59e1de4\") " pod="openshift-image-registry/image-registry-857cdcc958-b2kxp" Oct 30 19:47:37.138658 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.138282 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7a81c307-a7a7-492b-bce4-cca4b59e1de4-registry-tls\") pod \"image-registry-857cdcc958-b2kxp\" (UID: \"7a81c307-a7a7-492b-bce4-cca4b59e1de4\") " pod="openshift-image-registry/image-registry-857cdcc958-b2kxp" Oct 30 19:47:37.138658 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.138550 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7a81c307-a7a7-492b-bce4-cca4b59e1de4-trusted-ca\") pod \"image-registry-857cdcc958-b2kxp\" (UID: \"7a81c307-a7a7-492b-bce4-cca4b59e1de4\") " pod="openshift-image-registry/image-registry-857cdcc958-b2kxp" Oct 30 19:47:37.140092 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.140069 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7a81c307-a7a7-492b-bce4-cca4b59e1de4-installation-pull-secrets\") pod \"image-registry-857cdcc958-b2kxp\" (UID: \"7a81c307-a7a7-492b-bce4-cca4b59e1de4\") " pod="openshift-image-registry/image-registry-857cdcc958-b2kxp" Oct 30 19:47:37.141216 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.141173 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7a81c307-a7a7-492b-bce4-cca4b59e1de4-image-registry-private-configuration\") pod \"image-registry-857cdcc958-b2kxp\" (UID: \"7a81c307-a7a7-492b-bce4-cca4b59e1de4\") " pod="openshift-image-registry/image-registry-857cdcc958-b2kxp" Oct 30 19:47:37.143515 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.143496 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7a81c307-a7a7-492b-bce4-cca4b59e1de4-registry-tls\") pod \"image-registry-857cdcc958-b2kxp\" (UID: \"7a81c307-a7a7-492b-bce4-cca4b59e1de4\") " pod="openshift-image-registry/image-registry-857cdcc958-b2kxp" Oct 30 19:47:37.144950 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.144740 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-796595f99b-8nbrk" Oct 30 19:47:37.160710 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.160682 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nw8kx" Oct 30 19:47:37.165391 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.165365 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-mbstf" Oct 30 19:47:37.171045 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.170993 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzwfs\" (UniqueName: \"kubernetes.io/projected/7a81c307-a7a7-492b-bce4-cca4b59e1de4-kube-api-access-fzwfs\") pod \"image-registry-857cdcc958-b2kxp\" (UID: \"7a81c307-a7a7-492b-bce4-cca4b59e1de4\") " pod="openshift-image-registry/image-registry-857cdcc958-b2kxp" Oct 30 19:47:37.171943 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.171897 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7a81c307-a7a7-492b-bce4-cca4b59e1de4-bound-sa-token\") pod \"image-registry-857cdcc958-b2kxp\" (UID: \"7a81c307-a7a7-492b-bce4-cca4b59e1de4\") " pod="openshift-image-registry/image-registry-857cdcc958-b2kxp" Oct 30 19:47:37.292285 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.292218 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-857cdcc958-b2kxp" Oct 30 19:47:37.392762 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.392736 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-nmpzq-wm6lg"] Oct 30 19:47:37.394624 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:37.394583 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod269934ab_9b1e_4dab_8daf_86bd074bc85a.slice/crio-daa0869c1fa90993b15220900b3fbcff993bb6f9a998d3989861058c948f4c8a WatchSource:0}: Error finding container daa0869c1fa90993b15220900b3fbcff993bb6f9a998d3989861058c948f4c8a: Status 404 returned error can't find the container with id daa0869c1fa90993b15220900b3fbcff993bb6f9a998d3989861058c948f4c8a Oct 30 19:47:37.460865 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.460836 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-796595f99b-8nbrk"] Oct 30 19:47:37.462626 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:37.462587 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda16466d6_9ae1_4791_ad64_092e8d1cfa6f.slice/crio-ac43139b9917b0b6fe7c5ed3045a74700ded8c8a28968bb88bcc6b8663c17dba WatchSource:0}: Error finding container ac43139b9917b0b6fe7c5ed3045a74700ded8c8a28968bb88bcc6b8663c17dba: Status 404 returned error can't find the container with id ac43139b9917b0b6fe7c5ed3045a74700ded8c8a28968bb88bcc6b8663c17dba Oct 30 19:47:37.484985 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.482548 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-mbstf"] Oct 30 19:47:37.484985 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.484392 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nw8kx"] Oct 30 19:47:37.485073 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:37.485007 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d6ab503_19e8_4585_b7c7_612f33db3e07.slice/crio-24fa8352bff0ae6ad8f6269774267c2b884151ab13a116c7948b8c56fa5fc8d2 WatchSource:0}: Error finding container 24fa8352bff0ae6ad8f6269774267c2b884151ab13a116c7948b8c56fa5fc8d2: Status 404 returned error can't find the container with id 24fa8352bff0ae6ad8f6269774267c2b884151ab13a116c7948b8c56fa5fc8d2 Oct 30 19:47:37.487084 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:37.487066 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ef6ce3a_0191_4e2b_a483_3c1bde6e2444.slice/crio-f505334235e121dbb3cf79fe3df324c9b0f28b959884d1f1b1fa52cc88273d69 WatchSource:0}: Error finding container f505334235e121dbb3cf79fe3df324c9b0f28b959884d1f1b1fa52cc88273d69: Status 404 returned error can't find the container with id f505334235e121dbb3cf79fe3df324c9b0f28b959884d1f1b1fa52cc88273d69 Oct 30 19:47:37.595081 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.595058 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-857cdcc958-b2kxp"] Oct 30 19:47:37.596964 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:37.596942 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a81c307_a7a7_492b_bce4_cca4b59e1de4.slice/crio-8b5d0aac1cbb2add581bd6ce079f1b6fedfc042babb87396863370a826a578f4 WatchSource:0}: Error finding container 8b5d0aac1cbb2add581bd6ce079f1b6fedfc042babb87396863370a826a578f4: Status 404 returned error can't find the container with id 8b5d0aac1cbb2add581bd6ce079f1b6fedfc042babb87396863370a826a578f4 Oct 30 19:47:37.940943 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.940765 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nw8kx" event={"ID":"7ef6ce3a-0191-4e2b-a483-3c1bde6e2444","Type":"ContainerStarted","Data":"f505334235e121dbb3cf79fe3df324c9b0f28b959884d1f1b1fa52cc88273d69"} Oct 30 19:47:37.943928 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.943030 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-nmpzq-wm6lg" event={"ID":"269934ab-9b1e-4dab-8daf-86bd074bc85a","Type":"ContainerStarted","Data":"4468aa564ca1874f89756ea15ed177014ac56af1a2aa0ba56d3c42e48a782636"} Oct 30 19:47:37.943928 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.943083 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-nmpzq-wm6lg" event={"ID":"269934ab-9b1e-4dab-8daf-86bd074bc85a","Type":"ContainerStarted","Data":"daa0869c1fa90993b15220900b3fbcff993bb6f9a998d3989861058c948f4c8a"} Oct 30 19:47:37.946125 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.946098 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-857cdcc958-b2kxp" event={"ID":"7a81c307-a7a7-492b-bce4-cca4b59e1de4","Type":"ContainerStarted","Data":"a8ce79ede2f60b7de351c9d34f9f2ce8d7f8aef2f5a1be47dc2844e15fa92932"} Oct 30 19:47:37.946230 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.946146 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-857cdcc958-b2kxp" event={"ID":"7a81c307-a7a7-492b-bce4-cca4b59e1de4","Type":"ContainerStarted","Data":"8b5d0aac1cbb2add581bd6ce079f1b6fedfc042babb87396863370a826a578f4"} Oct 30 19:47:37.946666 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.946646 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-857cdcc958-b2kxp" Oct 30 19:47:37.949487 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.949435 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-mbstf" event={"ID":"8d6ab503-19e8-4585-b7c7-612f33db3e07","Type":"ContainerStarted","Data":"24fa8352bff0ae6ad8f6269774267c2b884151ab13a116c7948b8c56fa5fc8d2"} Oct 30 19:47:37.951783 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:37.951755 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-796595f99b-8nbrk" event={"ID":"a16466d6-9ae1-4791-ad64-092e8d1cfa6f","Type":"ContainerStarted","Data":"ac43139b9917b0b6fe7c5ed3045a74700ded8c8a28968bb88bcc6b8663c17dba"} Oct 30 19:47:38.039402 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:38.039343 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-nmpzq-wm6lg" podStartSLOduration=2.03932558 podStartE2EDuration="2.03932558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:37.987861077 +0000 UTC m=+19.400574863" watchObservedRunningTime="2026-10-30 19:47:38.03932558 +0000 UTC m=+19.452039356" Oct 30 19:47:38.039991 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:38.039954 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-857cdcc958-b2kxp" podStartSLOduration=2.039943532 podStartE2EDuration="2.039943532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:38.039262192 +0000 UTC m=+19.451975967" watchObservedRunningTime="2026-10-30 19:47:38.039943532 +0000 UTC m=+19.452657308" Oct 30 19:47:38.959365 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:38.959243 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nw8kx" event={"ID":"7ef6ce3a-0191-4e2b-a483-3c1bde6e2444","Type":"ContainerStarted","Data":"211205b21115bb2166997f3475432e1aa8ca8f075e605e5c557ea41254d54f23"} Oct 30 19:47:39.006802 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:39.005630 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nw8kx" podStartSLOduration=1.866628854 podStartE2EDuration="3.005598417s" totalImagesPullingTime="1.138969563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:39.00529897 +0000 UTC m=+20.418012749" watchObservedRunningTime="2026-10-30 19:47:39.005598417 +0000 UTC m=+20.418312194" Oct 30 19:47:39.964954 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:39.964923 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nw8kx" Oct 30 19:47:39.970522 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:39.970495 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nw8kx" Oct 30 19:47:40.771961 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:40.771927 5088 patch_prober.go:29] interesting pod/image-registry-5bc64fc876-25g7d container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 19:47:40.772127 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:40.771983 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" podUID="9a8ef63e-bcc4-4b71-b345-3fcbc48f4396" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:47:40.971914 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:40.971821 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-796595f99b-8nbrk" event={"ID":"a16466d6-9ae1-4791-ad64-092e8d1cfa6f","Type":"ContainerStarted","Data":"3d6af27263ae2a8cc40b93017b5ad4ded4feac6d77639383e67878dae56bcb8b"} Oct 30 19:47:41.020014 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:41.019959 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-796595f99b-8nbrk" podStartSLOduration=1.951419094 podStartE2EDuration="5.019947578s" totalImagesPullingTime="3.068528484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:41.019264948 +0000 UTC m=+22.431978750" watchObservedRunningTime="2026-10-30 19:47:41.019947578 +0000 UTC m=+22.432661428" Oct 30 19:47:42.038857 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:42.038829 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-tv4xh_afb6fa1c-9722-42c4-a914-4941bcbcaaad/networking-console-plugin/0.log" Oct 30 19:47:42.239537 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:42.239506 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qdj4h_da850825-59ab-480f-9047-afd7d2bdc19f/console-operator/1.log" Oct 30 19:47:42.440109 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:42.440041 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qdj4h_da850825-59ab-480f-9047-afd7d2bdc19f/console-operator/2.log" Oct 30 19:47:42.644958 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:42.644931 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-796595f99b-8nbrk_a16466d6-9ae1-4791-ad64-092e8d1cfa6f/console/0.log" Oct 30 19:47:43.439153 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:43.439123 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ml5lp_b8e3ed9d-5156-4698-82c3-df72fa728f94/dns/0.log" Oct 30 19:47:43.638455 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:43.638419 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ml5lp_b8e3ed9d-5156-4698-82c3-df72fa728f94/kube-rbac-proxy/0.log" Oct 30 19:47:44.038766 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:44.038739 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l9fpp_08b5a0ca-cfe1-4366-a404-5ebbaeb7c149/dns-node-resolver/0.log" Oct 30 19:47:44.440128 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:44.440056 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5bc64fc876-25g7d_9a8ef63e-bcc4-4b71-b345-3fcbc48f4396/registry/0.log" Oct 30 19:47:44.638828 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:44.638746 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6796cf4b57-b9d8k_b1440092-9901-4eff-8a9d-6174b2dfdada/registry/0.log" Oct 30 19:47:44.838985 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:44.838955 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-857cdcc958-b2kxp_7a81c307-a7a7-492b-bce4-cca4b59e1de4/registry/0.log" Oct 30 19:47:45.446559 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:45.446504 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hkvn6_2e03cb52-fd85-4e35-87b4-38833cbc7d74/node-ca/0.log" Oct 30 19:47:45.641032 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:45.640997 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-79458f7dbc-vbb28_41534930-917f-44ec-bccd-86d6c6fc7c70/router/0.log" Oct 30 19:47:45.802342 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:45.802305 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nxvww"] Oct 30 19:47:45.835784 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:45.835754 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nxvww" Oct 30 19:47:45.901148 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:45.900921 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 19:47:45.901148 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:45.900976 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 19:47:45.901148 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:45.901032 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 19:47:45.901148 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:45.901045 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-67dw4\"" Oct 30 19:47:45.901148 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:45.901119 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 19:47:45.922553 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:45.922520 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b86bf452-ef24-4715-b1d1-c4af3e689cfc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nxvww\" (UID: \"b86bf452-ef24-4715-b1d1-c4af3e689cfc\") " pod="openshift-monitoring/node-exporter-nxvww" Oct 30 19:47:45.922697 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:45.922593 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b86bf452-ef24-4715-b1d1-c4af3e689cfc-node-exporter-accelerators-collector-config\") pod \"node-exporter-nxvww\" (UID: \"b86bf452-ef24-4715-b1d1-c4af3e689cfc\") " pod="openshift-monitoring/node-exporter-nxvww" Oct 30 19:47:45.922697 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:45.922675 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b86bf452-ef24-4715-b1d1-c4af3e689cfc-metrics-client-ca\") pod \"node-exporter-nxvww\" (UID: \"b86bf452-ef24-4715-b1d1-c4af3e689cfc\") " pod="openshift-monitoring/node-exporter-nxvww" Oct 30 19:47:45.922791 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:45.922697 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6btr\" (UniqueName: \"kubernetes.io/projected/b86bf452-ef24-4715-b1d1-c4af3e689cfc-kube-api-access-b6btr\") pod \"node-exporter-nxvww\" (UID: \"b86bf452-ef24-4715-b1d1-c4af3e689cfc\") " pod="openshift-monitoring/node-exporter-nxvww" Oct 30 19:47:45.922791 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:45.922747 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b86bf452-ef24-4715-b1d1-c4af3e689cfc-node-exporter-textfile\") pod \"node-exporter-nxvww\" (UID: \"b86bf452-ef24-4715-b1d1-c4af3e689cfc\") " pod="openshift-monitoring/node-exporter-nxvww" Oct 30 19:47:45.922791 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:45.922765 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b86bf452-ef24-4715-b1d1-c4af3e689cfc-node-exporter-tls\") pod \"node-exporter-nxvww\" (UID: \"b86bf452-ef24-4715-b1d1-c4af3e689cfc\") " pod="openshift-monitoring/node-exporter-nxvww" Oct 30 19:47:45.922889 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:45.922793 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b86bf452-ef24-4715-b1d1-c4af3e689cfc-sys\") pod \"node-exporter-nxvww\" (UID: \"b86bf452-ef24-4715-b1d1-c4af3e689cfc\") " pod="openshift-monitoring/node-exporter-nxvww" Oct 30 19:47:45.922889 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:45.922811 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b86bf452-ef24-4715-b1d1-c4af3e689cfc-root\") pod \"node-exporter-nxvww\" (UID: \"b86bf452-ef24-4715-b1d1-c4af3e689cfc\") " pod="openshift-monitoring/node-exporter-nxvww" Oct 30 19:47:45.922889 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:45.922842 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b86bf452-ef24-4715-b1d1-c4af3e689cfc-node-exporter-wtmp\") pod \"node-exporter-nxvww\" (UID: \"b86bf452-ef24-4715-b1d1-c4af3e689cfc\") " pod="openshift-monitoring/node-exporter-nxvww" Oct 30 19:47:46.023850 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:46.023814 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b86bf452-ef24-4715-b1d1-c4af3e689cfc-node-exporter-accelerators-collector-config\") pod \"node-exporter-nxvww\" (UID: \"b86bf452-ef24-4715-b1d1-c4af3e689cfc\") " pod="openshift-monitoring/node-exporter-nxvww" Oct 30 19:47:46.024012 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:46.023858 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b86bf452-ef24-4715-b1d1-c4af3e689cfc-metrics-client-ca\") pod \"node-exporter-nxvww\" (UID: \"b86bf452-ef24-4715-b1d1-c4af3e689cfc\") " pod="openshift-monitoring/node-exporter-nxvww" Oct 30 19:47:46.024012 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:46.023882 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6btr\" (UniqueName: \"kubernetes.io/projected/b86bf452-ef24-4715-b1d1-c4af3e689cfc-kube-api-access-b6btr\") pod \"node-exporter-nxvww\" (UID: \"b86bf452-ef24-4715-b1d1-c4af3e689cfc\") " pod="openshift-monitoring/node-exporter-nxvww" Oct 30 19:47:46.024012 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:46.023912 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b86bf452-ef24-4715-b1d1-c4af3e689cfc-node-exporter-textfile\") pod \"node-exporter-nxvww\" (UID: \"b86bf452-ef24-4715-b1d1-c4af3e689cfc\") " pod="openshift-monitoring/node-exporter-nxvww" Oct 30 19:47:46.024012 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:46.023933 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b86bf452-ef24-4715-b1d1-c4af3e689cfc-node-exporter-tls\") pod \"node-exporter-nxvww\" (UID: \"b86bf452-ef24-4715-b1d1-c4af3e689cfc\") " pod="openshift-monitoring/node-exporter-nxvww" Oct 30 19:47:46.024012 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:46.023973 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b86bf452-ef24-4715-b1d1-c4af3e689cfc-sys\") pod \"node-exporter-nxvww\" (UID: \"b86bf452-ef24-4715-b1d1-c4af3e689cfc\") " pod="openshift-monitoring/node-exporter-nxvww" Oct 30 19:47:46.024012 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:46.024001 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b86bf452-ef24-4715-b1d1-c4af3e689cfc-root\") pod \"node-exporter-nxvww\" (UID: \"b86bf452-ef24-4715-b1d1-c4af3e689cfc\") " pod="openshift-monitoring/node-exporter-nxvww" Oct 30 19:47:46.024293 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:46.024042 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b86bf452-ef24-4715-b1d1-c4af3e689cfc-node-exporter-wtmp\") pod \"node-exporter-nxvww\" (UID: \"b86bf452-ef24-4715-b1d1-c4af3e689cfc\") " pod="openshift-monitoring/node-exporter-nxvww" Oct 30 19:47:46.024293 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:46.024073 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b86bf452-ef24-4715-b1d1-c4af3e689cfc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nxvww\" (UID: \"b86bf452-ef24-4715-b1d1-c4af3e689cfc\") " pod="openshift-monitoring/node-exporter-nxvww" Oct 30 19:47:46.024521 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:46.024488 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b86bf452-ef24-4715-b1d1-c4af3e689cfc-node-exporter-accelerators-collector-config\") pod \"node-exporter-nxvww\" (UID: \"b86bf452-ef24-4715-b1d1-c4af3e689cfc\") " pod="openshift-monitoring/node-exporter-nxvww" Oct 30 19:47:46.025577 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:46.025316 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b86bf452-ef24-4715-b1d1-c4af3e689cfc-metrics-client-ca\") pod \"node-exporter-nxvww\" (UID: \"b86bf452-ef24-4715-b1d1-c4af3e689cfc\") " pod="openshift-monitoring/node-exporter-nxvww" Oct 30 19:47:46.025577 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:46.025320 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b86bf452-ef24-4715-b1d1-c4af3e689cfc-sys\") pod \"node-exporter-nxvww\" (UID: \"b86bf452-ef24-4715-b1d1-c4af3e689cfc\") " pod="openshift-monitoring/node-exporter-nxvww" Oct 30 19:47:46.025577 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:46.025366 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b86bf452-ef24-4715-b1d1-c4af3e689cfc-root\") pod \"node-exporter-nxvww\" (UID: \"b86bf452-ef24-4715-b1d1-c4af3e689cfc\") " pod="openshift-monitoring/node-exporter-nxvww" Oct 30 19:47:46.025577 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:46.025523 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b86bf452-ef24-4715-b1d1-c4af3e689cfc-node-exporter-wtmp\") pod \"node-exporter-nxvww\" (UID: \"b86bf452-ef24-4715-b1d1-c4af3e689cfc\") " pod="openshift-monitoring/node-exporter-nxvww" Oct 30 19:47:46.025923 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:46.025672 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b86bf452-ef24-4715-b1d1-c4af3e689cfc-node-exporter-textfile\") pod \"node-exporter-nxvww\" (UID: \"b86bf452-ef24-4715-b1d1-c4af3e689cfc\") " pod="openshift-monitoring/node-exporter-nxvww" Oct 30 19:47:46.026563 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:46.026539 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b86bf452-ef24-4715-b1d1-c4af3e689cfc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nxvww\" (UID: \"b86bf452-ef24-4715-b1d1-c4af3e689cfc\") " pod="openshift-monitoring/node-exporter-nxvww" Oct 30 19:47:46.027801 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:46.026992 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b86bf452-ef24-4715-b1d1-c4af3e689cfc-node-exporter-tls\") pod \"node-exporter-nxvww\" (UID: \"b86bf452-ef24-4715-b1d1-c4af3e689cfc\") " pod="openshift-monitoring/node-exporter-nxvww" Oct 30 19:47:46.039945 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:46.039910 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lk574_5ad8f4b0-50ba-4191-8df4-25930717acb0/serve-healthcheck-canary/0.log" Oct 30 19:47:46.052500 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:46.052446 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6btr\" (UniqueName: \"kubernetes.io/projected/b86bf452-ef24-4715-b1d1-c4af3e689cfc-kube-api-access-b6btr\") pod \"node-exporter-nxvww\" (UID: \"b86bf452-ef24-4715-b1d1-c4af3e689cfc\") " pod="openshift-monitoring/node-exporter-nxvww" Oct 30 19:47:46.197008 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:46.196979 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nxvww" Oct 30 19:47:46.236897 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:47:46.236868 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb86bf452_ef24_4715_b1d1_c4af3e689cfc.slice/crio-f4df79e7489ce52f9814bec74a499541cf7498b244894fa1c40dc9fe06b24d44 WatchSource:0}: Error finding container f4df79e7489ce52f9814bec74a499541cf7498b244894fa1c40dc9fe06b24d44: Status 404 returned error can't find the container with id f4df79e7489ce52f9814bec74a499541cf7498b244894fa1c40dc9fe06b24d44 Oct 30 19:47:46.432542 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:46.432476 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:47:46.874924 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:46.874891 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:47:47.008359 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:47.008322 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nxvww" event={"ID":"b86bf452-ef24-4715-b1d1-c4af3e689cfc","Type":"ContainerStarted","Data":"f4df79e7489ce52f9814bec74a499541cf7498b244894fa1c40dc9fe06b24d44"} Oct 30 19:47:47.145724 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:47.145646 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-796595f99b-8nbrk" Oct 30 19:47:47.145724 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:47.145701 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-796595f99b-8nbrk" Oct 30 19:47:47.151299 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:47.151068 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-796595f99b-8nbrk" Oct 30 19:47:47.444779 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:47.444710 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kvw9t" Oct 30 19:47:48.015922 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:48.015885 5088 generic.go:356] "Generic (PLEG): container finished" podID="b86bf452-ef24-4715-b1d1-c4af3e689cfc" containerID="6bef86c618524afc8c3fc46a8d32be7e0461d8b9ae64d308cf851e0bf29382bd" exitCode=0 Oct 30 19:47:48.016974 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:48.016944 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nxvww" event={"ID":"b86bf452-ef24-4715-b1d1-c4af3e689cfc","Type":"ContainerDied","Data":"6bef86c618524afc8c3fc46a8d32be7e0461d8b9ae64d308cf851e0bf29382bd"} Oct 30 19:47:48.024728 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:48.024703 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-796595f99b-8nbrk" Oct 30 19:47:49.024694 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:49.024661 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nxvww" event={"ID":"b86bf452-ef24-4715-b1d1-c4af3e689cfc","Type":"ContainerStarted","Data":"7464755e7a46bf84e9fd71e7575323794812dd82cb955bc1eb1845a6db5c4902"} Oct 30 19:47:49.025120 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:49.024719 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nxvww" event={"ID":"b86bf452-ef24-4715-b1d1-c4af3e689cfc","Type":"ContainerStarted","Data":"eebddca1a866254e22dd0d96b24825b430c30e6b49b2f3e465a49d98ec381ebc"} Oct 30 19:47:49.108675 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:49.108604 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nxvww" podStartSLOduration=2.655581113 podStartE2EDuration="4.108589619s" totalImagesPullingTime="1.360309231s" totalInitContainerRuntime="92.699275ms" isStatefulPod=true podCreationTimestamp="2026-10-30 19:47:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:49.093522832 +0000 UTC m=+30.506236626" watchObservedRunningTime="2026-10-30 19:47:49.108589619 +0000 UTC m=+30.521303426" Oct 30 19:47:52.439653 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:47:52.439601 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 19:47:52.852642 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:52.852595 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-796595f99b-8nbrk"] Oct 30 19:47:54.050569 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:54.050529 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-mbstf" event={"ID":"8d6ab503-19e8-4585-b7c7-612f33db3e07","Type":"ContainerStarted","Data":"fff368855a530868eb8ad767bc0f5842c5bfac88e9854583bc7266471a1390f0"} Oct 30 19:47:54.051070 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:54.050604 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-mbstf" Oct 30 19:47:54.063063 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:54.063035 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-mbstf" Oct 30 19:47:54.124016 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:54.123957 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-mbstf" podStartSLOduration=2.376936295 podStartE2EDuration="18.123940397s" totalImagesPullingTime="15.747004102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:54.123080481 +0000 UTC m=+35.535794256" watchObservedRunningTime="2026-10-30 19:47:54.123940397 +0000 UTC m=+35.536654178" Oct 30 19:47:59.970630 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:47:59.970579 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-857cdcc958-b2kxp" Oct 30 19:48:00.113180 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:00.113146 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5bc64fc876-25g7d"] Oct 30 19:48:01.889104 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:01.889016 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" podUID="b1440092-9901-4eff-8a9d-6174b2dfdada" containerName="registry" containerID="cri-o://f3221a11e7e16a34abf16f1c11207c4ec32012fdfd813688f49c9bcec17a4675" gracePeriod=30 Oct 30 19:48:02.114226 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:02.114195 5088 generic.go:356] "Generic (PLEG): container finished" podID="b1440092-9901-4eff-8a9d-6174b2dfdada" containerID="f3221a11e7e16a34abf16f1c11207c4ec32012fdfd813688f49c9bcec17a4675" exitCode=0 Oct 30 19:48:02.114376 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:02.114240 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" event={"ID":"b1440092-9901-4eff-8a9d-6174b2dfdada","Type":"ContainerDied","Data":"f3221a11e7e16a34abf16f1c11207c4ec32012fdfd813688f49c9bcec17a4675"} Oct 30 19:48:02.186004 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:02.185975 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:48:02.285790 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:02.285758 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1440092-9901-4eff-8a9d-6174b2dfdada-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b1440092-9901-4eff-8a9d-6174b2dfdada-installation-pull-secrets\") pod \"b1440092-9901-4eff-8a9d-6174b2dfdada\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " Oct 30 19:48:02.285961 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:02.285799 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1440092-9901-4eff-8a9d-6174b2dfdada-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b1440092-9901-4eff-8a9d-6174b2dfdada-image-registry-private-configuration\") pod \"b1440092-9901-4eff-8a9d-6174b2dfdada\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " Oct 30 19:48:02.285961 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:02.285830 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1440092-9901-4eff-8a9d-6174b2dfdada-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b1440092-9901-4eff-8a9d-6174b2dfdada-trusted-ca\") pod \"b1440092-9901-4eff-8a9d-6174b2dfdada\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " Oct 30 19:48:02.285961 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:02.285855 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1440092-9901-4eff-8a9d-6174b2dfdada-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b1440092-9901-4eff-8a9d-6174b2dfdada-registry-certificates\") pod \"b1440092-9901-4eff-8a9d-6174b2dfdada\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " Oct 30 19:48:02.286122 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:02.286067 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1440092-9901-4eff-8a9d-6174b2dfdada-kube-api-access-8xtvg\" (UniqueName: \"kubernetes.io/projected/b1440092-9901-4eff-8a9d-6174b2dfdada-kube-api-access-8xtvg\") pod \"b1440092-9901-4eff-8a9d-6174b2dfdada\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " Oct 30 19:48:02.286212 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:02.286169 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1440092-9901-4eff-8a9d-6174b2dfdada-registry-tls\" (UniqueName: \"kubernetes.io/projected/b1440092-9901-4eff-8a9d-6174b2dfdada-registry-tls\") pod \"b1440092-9901-4eff-8a9d-6174b2dfdada\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " Oct 30 19:48:02.286299 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:02.286215 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1440092-9901-4eff-8a9d-6174b2dfdada-registry-certificates" pod "b1440092-9901-4eff-8a9d-6174b2dfdada" (UID: "b1440092-9901-4eff-8a9d-6174b2dfdada"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:48:02.286299 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:02.286272 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1440092-9901-4eff-8a9d-6174b2dfdada-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b1440092-9901-4eff-8a9d-6174b2dfdada-bound-sa-token\") pod \"b1440092-9901-4eff-8a9d-6174b2dfdada\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " Oct 30 19:48:02.286431 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:02.286336 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1440092-9901-4eff-8a9d-6174b2dfdada-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b1440092-9901-4eff-8a9d-6174b2dfdada-ca-trust-extracted\") pod \"b1440092-9901-4eff-8a9d-6174b2dfdada\" (UID: \"b1440092-9901-4eff-8a9d-6174b2dfdada\") " Oct 30 19:48:02.286483 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:02.286457 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1440092-9901-4eff-8a9d-6174b2dfdada-trusted-ca" pod "b1440092-9901-4eff-8a9d-6174b2dfdada" (UID: "b1440092-9901-4eff-8a9d-6174b2dfdada"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:48:02.286752 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:02.286728 5088 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b1440092-9901-4eff-8a9d-6174b2dfdada-trusted-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 19:48:02.286848 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:02.286758 5088 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b1440092-9901-4eff-8a9d-6174b2dfdada-registry-certificates\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 19:48:02.288538 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:02.288511 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1440092-9901-4eff-8a9d-6174b2dfdada-registry-tls" pod "b1440092-9901-4eff-8a9d-6174b2dfdada" (UID: "b1440092-9901-4eff-8a9d-6174b2dfdada"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:48:02.289116 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:02.289048 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1440092-9901-4eff-8a9d-6174b2dfdada-installation-pull-secrets" pod "b1440092-9901-4eff-8a9d-6174b2dfdada" (UID: "b1440092-9901-4eff-8a9d-6174b2dfdada"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:48:02.289295 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:02.289162 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1440092-9901-4eff-8a9d-6174b2dfdada-bound-sa-token" pod "b1440092-9901-4eff-8a9d-6174b2dfdada" (UID: "b1440092-9901-4eff-8a9d-6174b2dfdada"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:48:02.289295 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:02.289259 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1440092-9901-4eff-8a9d-6174b2dfdada-image-registry-private-configuration" pod "b1440092-9901-4eff-8a9d-6174b2dfdada" (UID: "b1440092-9901-4eff-8a9d-6174b2dfdada"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:48:02.290798 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:02.290765 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1440092-9901-4eff-8a9d-6174b2dfdada-kube-api-access-8xtvg" pod "b1440092-9901-4eff-8a9d-6174b2dfdada" (UID: "b1440092-9901-4eff-8a9d-6174b2dfdada"). InnerVolumeSpecName "kube-api-access-8xtvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:48:02.296125 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:02.296094 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1440092-9901-4eff-8a9d-6174b2dfdada-ca-trust-extracted" pod "b1440092-9901-4eff-8a9d-6174b2dfdada" (UID: "b1440092-9901-4eff-8a9d-6174b2dfdada"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:48:02.387371 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:02.387346 5088 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b1440092-9901-4eff-8a9d-6174b2dfdada-ca-trust-extracted\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 19:48:02.387371 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:02.387369 5088 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b1440092-9901-4eff-8a9d-6174b2dfdada-installation-pull-secrets\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 19:48:02.387493 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:02.387380 5088 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b1440092-9901-4eff-8a9d-6174b2dfdada-image-registry-private-configuration\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 19:48:02.387493 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:02.387389 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xtvg\" (UniqueName: \"kubernetes.io/projected/b1440092-9901-4eff-8a9d-6174b2dfdada-kube-api-access-8xtvg\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 19:48:02.387493 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:02.387399 5088 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1440092-9901-4eff-8a9d-6174b2dfdada-registry-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 19:48:02.387493 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:02.387407 5088 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b1440092-9901-4eff-8a9d-6174b2dfdada-bound-sa-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 19:48:03.119923 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:03.119853 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" Oct 30 19:48:03.119923 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:03.119858 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6796cf4b57-b9d8k" event={"ID":"b1440092-9901-4eff-8a9d-6174b2dfdada","Type":"ContainerDied","Data":"3c141530abb191bd2325d7b767fbac05d13e06a7f61cf87c4323fd1736e9190c"} Oct 30 19:48:03.119923 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:03.119923 5088 scope.go:122] "RemoveContainer" containerID="f3221a11e7e16a34abf16f1c11207c4ec32012fdfd813688f49c9bcec17a4675" Oct 30 19:48:03.196493 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:03.196459 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6796cf4b57-b9d8k"] Oct 30 19:48:03.202698 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:03.202673 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6796cf4b57-b9d8k"] Oct 30 19:48:04.737077 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:04.737044 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1440092-9901-4eff-8a9d-6174b2dfdada" path="/var/lib/kubelet/pods/b1440092-9901-4eff-8a9d-6174b2dfdada/volumes" Oct 30 19:48:08.329029 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:08.328998 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-d7ptc-wwx7j"] Oct 30 19:48:08.329573 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:08.329247 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-d7ptc-wwx7j" podUID="005bece8-8d37-4a58-954c-0ae3dab9dadc" containerName="insights-gathering" containerID="cri-o://4237f64d37febae57523262a2c633409efd9684b43cea4ca819f7a6623c23386" gracePeriod=30 Oct 30 19:48:08.670588 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:08.670563 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-d7ptc-wwx7j" Oct 30 19:48:08.842820 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:08.842792 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/005bece8-8d37-4a58-954c-0ae3dab9dadc-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/005bece8-8d37-4a58-954c-0ae3dab9dadc-archives-path\") pod \"005bece8-8d37-4a58-954c-0ae3dab9dadc\" (UID: \"005bece8-8d37-4a58-954c-0ae3dab9dadc\") " Oct 30 19:48:08.843012 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:08.842879 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/005bece8-8d37-4a58-954c-0ae3dab9dadc-kube-api-access-68nwv\" (UniqueName: \"kubernetes.io/projected/005bece8-8d37-4a58-954c-0ae3dab9dadc-kube-api-access-68nwv\") pod \"005bece8-8d37-4a58-954c-0ae3dab9dadc\" (UID: \"005bece8-8d37-4a58-954c-0ae3dab9dadc\") " Oct 30 19:48:08.843012 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:08.842959 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/005bece8-8d37-4a58-954c-0ae3dab9dadc-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/005bece8-8d37-4a58-954c-0ae3dab9dadc-trusted-ca-bundle\") pod \"005bece8-8d37-4a58-954c-0ae3dab9dadc\" (UID: \"005bece8-8d37-4a58-954c-0ae3dab9dadc\") " Oct 30 19:48:08.843012 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:08.842987 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/005bece8-8d37-4a58-954c-0ae3dab9dadc-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/005bece8-8d37-4a58-954c-0ae3dab9dadc-service-ca-bundle\") pod \"005bece8-8d37-4a58-954c-0ae3dab9dadc\" (UID: \"005bece8-8d37-4a58-954c-0ae3dab9dadc\") " Oct 30 19:48:08.843247 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:08.843155 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/005bece8-8d37-4a58-954c-0ae3dab9dadc-archives-path" pod "005bece8-8d37-4a58-954c-0ae3dab9dadc" (UID: "005bece8-8d37-4a58-954c-0ae3dab9dadc"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:48:08.843379 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:08.843327 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/005bece8-8d37-4a58-954c-0ae3dab9dadc-trusted-ca-bundle" pod "005bece8-8d37-4a58-954c-0ae3dab9dadc" (UID: "005bece8-8d37-4a58-954c-0ae3dab9dadc"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:48:08.843426 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:08.843387 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/005bece8-8d37-4a58-954c-0ae3dab9dadc-service-ca-bundle" pod "005bece8-8d37-4a58-954c-0ae3dab9dadc" (UID: "005bece8-8d37-4a58-954c-0ae3dab9dadc"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:48:08.845517 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:08.845492 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/005bece8-8d37-4a58-954c-0ae3dab9dadc-kube-api-access-68nwv" pod "005bece8-8d37-4a58-954c-0ae3dab9dadc" (UID: "005bece8-8d37-4a58-954c-0ae3dab9dadc"). InnerVolumeSpecName "kube-api-access-68nwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:48:08.944193 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:08.944127 5088 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/005bece8-8d37-4a58-954c-0ae3dab9dadc-trusted-ca-bundle\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 19:48:08.944193 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:08.944149 5088 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/005bece8-8d37-4a58-954c-0ae3dab9dadc-service-ca-bundle\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 19:48:08.944193 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:08.944159 5088 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/005bece8-8d37-4a58-954c-0ae3dab9dadc-archives-path\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 19:48:08.944193 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:08.944167 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68nwv\" (UniqueName: \"kubernetes.io/projected/005bece8-8d37-4a58-954c-0ae3dab9dadc-kube-api-access-68nwv\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 19:48:09.148174 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:09.148138 5088 generic.go:356] "Generic (PLEG): container finished" podID="005bece8-8d37-4a58-954c-0ae3dab9dadc" containerID="4237f64d37febae57523262a2c633409efd9684b43cea4ca819f7a6623c23386" exitCode=2 Oct 30 19:48:09.148311 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:09.148186 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-d7ptc-wwx7j" Oct 30 19:48:09.148311 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:09.148218 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-d7ptc-wwx7j" event={"ID":"005bece8-8d37-4a58-954c-0ae3dab9dadc","Type":"ContainerDied","Data":"4237f64d37febae57523262a2c633409efd9684b43cea4ca819f7a6623c23386"} Oct 30 19:48:09.148311 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:09.148273 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-d7ptc-wwx7j" event={"ID":"005bece8-8d37-4a58-954c-0ae3dab9dadc","Type":"ContainerDied","Data":"bf54950fee0ea5f71052f3a0ba8d1af23942d0be378225fc4e5d2cfa2fcc1e69"} Oct 30 19:48:09.148311 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:09.148294 5088 scope.go:122] "RemoveContainer" containerID="4237f64d37febae57523262a2c633409efd9684b43cea4ca819f7a6623c23386" Oct 30 19:48:09.159662 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:09.159642 5088 scope.go:122] "RemoveContainer" containerID="4237f64d37febae57523262a2c633409efd9684b43cea4ca819f7a6623c23386" Oct 30 19:48:09.159956 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:48:09.159928 5088 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4237f64d37febae57523262a2c633409efd9684b43cea4ca819f7a6623c23386\": container with ID starting with 4237f64d37febae57523262a2c633409efd9684b43cea4ca819f7a6623c23386 not found: ID does not exist" containerID="4237f64d37febae57523262a2c633409efd9684b43cea4ca819f7a6623c23386" Oct 30 19:48:09.160033 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:09.159964 5088 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4237f64d37febae57523262a2c633409efd9684b43cea4ca819f7a6623c23386"} err="failed to get container status \"4237f64d37febae57523262a2c633409efd9684b43cea4ca819f7a6623c23386\": rpc error: code = NotFound desc = could not find container \"4237f64d37febae57523262a2c633409efd9684b43cea4ca819f7a6623c23386\": container with ID starting with 4237f64d37febae57523262a2c633409efd9684b43cea4ca819f7a6623c23386 not found: ID does not exist" Oct 30 19:48:09.205298 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:09.205246 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-d7ptc-wwx7j"] Oct 30 19:48:09.218847 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:09.218828 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-d7ptc-wwx7j"] Oct 30 19:48:10.736912 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:10.736874 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="005bece8-8d37-4a58-954c-0ae3dab9dadc" path="/var/lib/kubelet/pods/005bece8-8d37-4a58-954c-0ae3dab9dadc/volumes" Oct 30 19:48:17.874102 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:17.874038 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-796595f99b-8nbrk" podUID="a16466d6-9ae1-4791-ad64-092e8d1cfa6f" containerName="console" containerID="cri-o://3d6af27263ae2a8cc40b93017b5ad4ded4feac6d77639383e67878dae56bcb8b" gracePeriod=15 Oct 30 19:48:18.017747 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:18.017709 5088 patch_prober.go:29] interesting pod/console-796595f99b-8nbrk container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.132.0.29:8443/health\": dial tcp 10.132.0.29:8443: connect: connection refused" start-of-body= Oct 30 19:48:18.017896 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:18.017785 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/console-796595f99b-8nbrk" podUID="a16466d6-9ae1-4791-ad64-092e8d1cfa6f" containerName="console" probeResult="failure" output="Get \"https://10.132.0.29:8443/health\": dial tcp 10.132.0.29:8443: connect: connection refused" Oct 30 19:48:18.132424 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:18.132370 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-796595f99b-8nbrk" Oct 30 19:48:18.186742 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:18.186709 5088 generic.go:356] "Generic (PLEG): container finished" podID="a16466d6-9ae1-4791-ad64-092e8d1cfa6f" containerID="3d6af27263ae2a8cc40b93017b5ad4ded4feac6d77639383e67878dae56bcb8b" exitCode=0 Oct 30 19:48:18.186878 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:18.186755 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-796595f99b-8nbrk" Oct 30 19:48:18.186878 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:18.186782 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-796595f99b-8nbrk" event={"ID":"a16466d6-9ae1-4791-ad64-092e8d1cfa6f","Type":"ContainerDied","Data":"3d6af27263ae2a8cc40b93017b5ad4ded4feac6d77639383e67878dae56bcb8b"} Oct 30 19:48:18.186878 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:18.186836 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-796595f99b-8nbrk" event={"ID":"a16466d6-9ae1-4791-ad64-092e8d1cfa6f","Type":"ContainerDied","Data":"ac43139b9917b0b6fe7c5ed3045a74700ded8c8a28968bb88bcc6b8663c17dba"} Oct 30 19:48:18.186878 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:18.186862 5088 scope.go:122] "RemoveContainer" containerID="3d6af27263ae2a8cc40b93017b5ad4ded4feac6d77639383e67878dae56bcb8b" Oct 30 19:48:18.195151 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:18.195131 5088 scope.go:122] "RemoveContainer" containerID="3d6af27263ae2a8cc40b93017b5ad4ded4feac6d77639383e67878dae56bcb8b" Oct 30 19:48:18.195432 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:48:18.195408 5088 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d6af27263ae2a8cc40b93017b5ad4ded4feac6d77639383e67878dae56bcb8b\": container with ID starting with 3d6af27263ae2a8cc40b93017b5ad4ded4feac6d77639383e67878dae56bcb8b not found: ID does not exist" containerID="3d6af27263ae2a8cc40b93017b5ad4ded4feac6d77639383e67878dae56bcb8b" Oct 30 19:48:18.195536 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:18.195444 5088 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d6af27263ae2a8cc40b93017b5ad4ded4feac6d77639383e67878dae56bcb8b"} err="failed to get container status \"3d6af27263ae2a8cc40b93017b5ad4ded4feac6d77639383e67878dae56bcb8b\": rpc error: code = NotFound desc = could not find container \"3d6af27263ae2a8cc40b93017b5ad4ded4feac6d77639383e67878dae56bcb8b\": container with ID starting with 3d6af27263ae2a8cc40b93017b5ad4ded4feac6d77639383e67878dae56bcb8b not found: ID does not exist" Oct 30 19:48:18.210512 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:18.210489 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-console-serving-cert\") pod \"a16466d6-9ae1-4791-ad64-092e8d1cfa6f\" (UID: \"a16466d6-9ae1-4791-ad64-092e8d1cfa6f\") " Oct 30 19:48:18.210671 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:18.210531 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-console-oauth-config\") pod \"a16466d6-9ae1-4791-ad64-092e8d1cfa6f\" (UID: \"a16466d6-9ae1-4791-ad64-092e8d1cfa6f\") " Oct 30 19:48:18.210671 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:18.210582 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-tmp\") pod \"a16466d6-9ae1-4791-ad64-092e8d1cfa6f\" (UID: \"a16466d6-9ae1-4791-ad64-092e8d1cfa6f\") " Oct 30 19:48:18.210671 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:18.210632 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-kube-api-access-w68mv\" (UniqueName: \"kubernetes.io/projected/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-kube-api-access-w68mv\") pod \"a16466d6-9ae1-4791-ad64-092e8d1cfa6f\" (UID: \"a16466d6-9ae1-4791-ad64-092e8d1cfa6f\") " Oct 30 19:48:18.210671 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:18.210659 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-console-config\" (UniqueName: \"kubernetes.io/configmap/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-console-config\") pod \"a16466d6-9ae1-4791-ad64-092e8d1cfa6f\" (UID: \"a16466d6-9ae1-4791-ad64-092e8d1cfa6f\") " Oct 30 19:48:18.210882 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:18.210717 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-service-ca\" (UniqueName: \"kubernetes.io/configmap/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-service-ca\") pod \"a16466d6-9ae1-4791-ad64-092e8d1cfa6f\" (UID: \"a16466d6-9ae1-4791-ad64-092e8d1cfa6f\") " Oct 30 19:48:18.210882 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:18.210754 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-oauth-serving-cert\") pod \"a16466d6-9ae1-4791-ad64-092e8d1cfa6f\" (UID: \"a16466d6-9ae1-4791-ad64-092e8d1cfa6f\") " Oct 30 19:48:18.210984 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:18.210859 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-tmp" pod "a16466d6-9ae1-4791-ad64-092e8d1cfa6f" (UID: "a16466d6-9ae1-4791-ad64-092e8d1cfa6f"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:48:18.211125 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:18.211104 5088 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-tmp\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 19:48:18.211195 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:18.211173 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-oauth-serving-cert" pod "a16466d6-9ae1-4791-ad64-092e8d1cfa6f" (UID: "a16466d6-9ae1-4791-ad64-092e8d1cfa6f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:48:18.211300 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:18.211278 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-service-ca" pod "a16466d6-9ae1-4791-ad64-092e8d1cfa6f" (UID: "a16466d6-9ae1-4791-ad64-092e8d1cfa6f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:48:18.211350 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:18.211303 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-console-config" pod "a16466d6-9ae1-4791-ad64-092e8d1cfa6f" (UID: "a16466d6-9ae1-4791-ad64-092e8d1cfa6f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:48:18.213095 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:18.213046 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-console-serving-cert" pod "a16466d6-9ae1-4791-ad64-092e8d1cfa6f" (UID: "a16466d6-9ae1-4791-ad64-092e8d1cfa6f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:48:18.213200 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:18.213133 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-kube-api-access-w68mv" pod "a16466d6-9ae1-4791-ad64-092e8d1cfa6f" (UID: "a16466d6-9ae1-4791-ad64-092e8d1cfa6f"). InnerVolumeSpecName "kube-api-access-w68mv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:48:18.213200 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:18.213172 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-console-oauth-config" pod "a16466d6-9ae1-4791-ad64-092e8d1cfa6f" (UID: "a16466d6-9ae1-4791-ad64-092e8d1cfa6f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:48:18.312354 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:18.312320 5088 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-oauth-serving-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 19:48:18.312354 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:18.312346 5088 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-console-serving-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 19:48:18.312354 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:18.312357 5088 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-console-oauth-config\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 19:48:18.312579 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:18.312365 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w68mv\" (UniqueName: \"kubernetes.io/projected/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-kube-api-access-w68mv\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 19:48:18.312579 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:18.312374 5088 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-console-config\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 19:48:18.312579 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:18.312383 5088 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a16466d6-9ae1-4791-ad64-092e8d1cfa6f-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 19:48:18.536740 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:18.536710 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-796595f99b-8nbrk"] Oct 30 19:48:18.549535 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:18.549511 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-796595f99b-8nbrk"] Oct 30 19:48:18.736850 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:18.736824 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a16466d6-9ae1-4791-ad64-092e8d1cfa6f" path="/var/lib/kubelet/pods/a16466d6-9ae1-4791-ad64-092e8d1cfa6f/volumes" Oct 30 19:48:22.436068 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:48:22.436031 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 19:48:25.138625 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:25.138556 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" podUID="9a8ef63e-bcc4-4b71-b345-3fcbc48f4396" containerName="registry" containerID="cri-o://3afabe566656ee8e35be184b729246f5dfde6a0337aaedbc4f5b51f20a7bcc27" gracePeriod=30 Oct 30 19:48:25.416523 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:25.416462 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:48:25.471472 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:25.471446 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-trusted-ca\") pod \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " Oct 30 19:48:25.471592 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:25.471492 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-registry-tls\" (UniqueName: \"kubernetes.io/projected/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-registry-tls\") pod \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " Oct 30 19:48:25.471592 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:25.471531 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-image-registry-private-configuration\") pod \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " Oct 30 19:48:25.471592 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:25.471561 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-ca-trust-extracted\") pod \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " Oct 30 19:48:25.471737 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:25.471650 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-kube-api-access-rzg2k\" (UniqueName: \"kubernetes.io/projected/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-kube-api-access-rzg2k\") pod \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " Oct 30 19:48:25.471737 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:25.471678 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-installation-pull-secrets\") pod \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " Oct 30 19:48:25.471737 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:25.471721 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-registry-certificates\") pod \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " Oct 30 19:48:25.471874 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:25.471764 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-bound-sa-token\") pod \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\" (UID: \"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396\") " Oct 30 19:48:25.472043 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:25.471915 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-trusted-ca" pod "9a8ef63e-bcc4-4b71-b345-3fcbc48f4396" (UID: "9a8ef63e-bcc4-4b71-b345-3fcbc48f4396"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:48:25.472152 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:25.472140 5088 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-trusted-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 19:48:25.472324 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:25.472303 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-registry-certificates" pod "9a8ef63e-bcc4-4b71-b345-3fcbc48f4396" (UID: "9a8ef63e-bcc4-4b71-b345-3fcbc48f4396"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:48:25.473988 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:25.473956 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-image-registry-private-configuration" pod "9a8ef63e-bcc4-4b71-b345-3fcbc48f4396" (UID: "9a8ef63e-bcc4-4b71-b345-3fcbc48f4396"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:48:25.474086 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:25.474022 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-kube-api-access-rzg2k" pod "9a8ef63e-bcc4-4b71-b345-3fcbc48f4396" (UID: "9a8ef63e-bcc4-4b71-b345-3fcbc48f4396"). InnerVolumeSpecName "kube-api-access-rzg2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:48:25.474234 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:25.474203 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-installation-pull-secrets" pod "9a8ef63e-bcc4-4b71-b345-3fcbc48f4396" (UID: "9a8ef63e-bcc4-4b71-b345-3fcbc48f4396"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:48:25.474234 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:25.474225 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-registry-tls" pod "9a8ef63e-bcc4-4b71-b345-3fcbc48f4396" (UID: "9a8ef63e-bcc4-4b71-b345-3fcbc48f4396"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:48:25.474389 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:25.474293 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-bound-sa-token" pod "9a8ef63e-bcc4-4b71-b345-3fcbc48f4396" (UID: "9a8ef63e-bcc4-4b71-b345-3fcbc48f4396"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:48:25.478900 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:25.478874 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-ca-trust-extracted" pod "9a8ef63e-bcc4-4b71-b345-3fcbc48f4396" (UID: "9a8ef63e-bcc4-4b71-b345-3fcbc48f4396"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:48:25.572954 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:25.572924 5088 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-registry-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 19:48:25.572954 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:25.572953 5088 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-image-registry-private-configuration\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 19:48:25.573129 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:25.572963 5088 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-ca-trust-extracted\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 19:48:25.573129 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:25.572972 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzg2k\" (UniqueName: \"kubernetes.io/projected/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-kube-api-access-rzg2k\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 19:48:25.573129 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:25.572983 5088 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-installation-pull-secrets\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 19:48:25.573129 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:25.572992 5088 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-registry-certificates\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 19:48:25.573129 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:25.573000 5088 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396-bound-sa-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 19:48:26.219547 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:26.219461 5088 generic.go:356] "Generic (PLEG): container finished" podID="9a8ef63e-bcc4-4b71-b345-3fcbc48f4396" containerID="3afabe566656ee8e35be184b729246f5dfde6a0337aaedbc4f5b51f20a7bcc27" exitCode=0 Oct 30 19:48:26.219547 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:26.219532 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" event={"ID":"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396","Type":"ContainerDied","Data":"3afabe566656ee8e35be184b729246f5dfde6a0337aaedbc4f5b51f20a7bcc27"} Oct 30 19:48:26.220044 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:26.219587 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" event={"ID":"9a8ef63e-bcc4-4b71-b345-3fcbc48f4396","Type":"ContainerDied","Data":"275923771ca2752326255995ad1da4193d0a2dba0490cd8d15fb15d9a4286408"} Oct 30 19:48:26.220044 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:26.219510 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bc64fc876-25g7d" Oct 30 19:48:26.220044 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:26.219634 5088 scope.go:122] "RemoveContainer" containerID="3afabe566656ee8e35be184b729246f5dfde6a0337aaedbc4f5b51f20a7bcc27" Oct 30 19:48:26.228044 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:26.228030 5088 scope.go:122] "RemoveContainer" containerID="3afabe566656ee8e35be184b729246f5dfde6a0337aaedbc4f5b51f20a7bcc27" Oct 30 19:48:26.228306 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:48:26.228289 5088 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3afabe566656ee8e35be184b729246f5dfde6a0337aaedbc4f5b51f20a7bcc27\": container with ID starting with 3afabe566656ee8e35be184b729246f5dfde6a0337aaedbc4f5b51f20a7bcc27 not found: ID does not exist" containerID="3afabe566656ee8e35be184b729246f5dfde6a0337aaedbc4f5b51f20a7bcc27" Oct 30 19:48:26.228365 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:26.228312 5088 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3afabe566656ee8e35be184b729246f5dfde6a0337aaedbc4f5b51f20a7bcc27"} err="failed to get container status \"3afabe566656ee8e35be184b729246f5dfde6a0337aaedbc4f5b51f20a7bcc27\": rpc error: code = NotFound desc = could not find container \"3afabe566656ee8e35be184b729246f5dfde6a0337aaedbc4f5b51f20a7bcc27\": container with ID starting with 3afabe566656ee8e35be184b729246f5dfde6a0337aaedbc4f5b51f20a7bcc27 not found: ID does not exist" Oct 30 19:48:26.291043 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:26.291021 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5bc64fc876-25g7d"] Oct 30 19:48:26.304602 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:26.304582 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5bc64fc876-25g7d"] Oct 30 19:48:26.737941 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:26.737919 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a8ef63e-bcc4-4b71-b345-3fcbc48f4396" path="/var/lib/kubelet/pods/9a8ef63e-bcc4-4b71-b345-3fcbc48f4396/volumes" Oct 30 19:48:36.263291 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:36.263259 5088 generic.go:356] "Generic (PLEG): container finished" podID="d78606e4-ccb9-421a-8ae5-8fbecbcd3831" containerID="b6e8c79e765c82ab981316b7753653230e2b3e74e996f9691637b377afa3bb54" exitCode=0 Oct 30 19:48:36.263777 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:36.263307 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-bnp7s" event={"ID":"d78606e4-ccb9-421a-8ae5-8fbecbcd3831","Type":"ContainerDied","Data":"b6e8c79e765c82ab981316b7753653230e2b3e74e996f9691637b377afa3bb54"} Oct 30 19:48:36.263777 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:36.263355 5088 scope.go:122] "RemoveContainer" containerID="0311b6a67457029147850bb5d461ad63c35e9cdc77c6212ede73fc2499803caf" Oct 30 19:48:36.263777 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:36.263710 5088 scope.go:122] "RemoveContainer" containerID="b6e8c79e765c82ab981316b7753653230e2b3e74e996f9691637b377afa3bb54" Oct 30 19:48:36.263943 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:48:36.263921 5088 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"insights-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=insights-operator pod=insights-operator-7c4986d449-bnp7s_openshift-insights(d78606e4-ccb9-421a-8ae5-8fbecbcd3831)\"" pod="openshift-insights/insights-operator-7c4986d449-bnp7s" podUID="d78606e4-ccb9-421a-8ae5-8fbecbcd3831" Oct 30 19:48:48.733855 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:48.733826 5088 scope.go:122] "RemoveContainer" containerID="b6e8c79e765c82ab981316b7753653230e2b3e74e996f9691637b377afa3bb54" Oct 30 19:48:49.254560 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:49.254531 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-nmpzq-wm6lg"] Oct 30 19:48:49.254828 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:49.254783 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-nmpzq-wm6lg" podUID="269934ab-9b1e-4dab-8daf-86bd074bc85a" containerName="insights-gathering" containerID="cri-o://4468aa564ca1874f89756ea15ed177014ac56af1a2aa0ba56d3c42e48a782636" gracePeriod=30 Oct 30 19:48:49.319648 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:49.319589 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-bnp7s" event={"ID":"d78606e4-ccb9-421a-8ae5-8fbecbcd3831","Type":"ContainerStarted","Data":"1b753b5d28075b9dae28e4ade11d300d32a391b69986e1cd8ed16f84c7396dd0"} Oct 30 19:48:49.518958 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:49.518933 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-nmpzq-wm6lg" Oct 30 19:48:49.655296 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:49.655263 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/269934ab-9b1e-4dab-8daf-86bd074bc85a-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/269934ab-9b1e-4dab-8daf-86bd074bc85a-archives-path\") pod \"269934ab-9b1e-4dab-8daf-86bd074bc85a\" (UID: \"269934ab-9b1e-4dab-8daf-86bd074bc85a\") " Oct 30 19:48:49.655475 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:49.655318 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/269934ab-9b1e-4dab-8daf-86bd074bc85a-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/269934ab-9b1e-4dab-8daf-86bd074bc85a-service-ca-bundle\") pod \"269934ab-9b1e-4dab-8daf-86bd074bc85a\" (UID: \"269934ab-9b1e-4dab-8daf-86bd074bc85a\") " Oct 30 19:48:49.655475 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:49.655372 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/269934ab-9b1e-4dab-8daf-86bd074bc85a-kube-api-access-ns9wx\" (UniqueName: \"kubernetes.io/projected/269934ab-9b1e-4dab-8daf-86bd074bc85a-kube-api-access-ns9wx\") pod \"269934ab-9b1e-4dab-8daf-86bd074bc85a\" (UID: \"269934ab-9b1e-4dab-8daf-86bd074bc85a\") " Oct 30 19:48:49.655475 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:49.655428 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/269934ab-9b1e-4dab-8daf-86bd074bc85a-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/269934ab-9b1e-4dab-8daf-86bd074bc85a-trusted-ca-bundle\") pod \"269934ab-9b1e-4dab-8daf-86bd074bc85a\" (UID: \"269934ab-9b1e-4dab-8daf-86bd074bc85a\") " Oct 30 19:48:49.655681 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:49.655592 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/269934ab-9b1e-4dab-8daf-86bd074bc85a-archives-path" pod "269934ab-9b1e-4dab-8daf-86bd074bc85a" (UID: "269934ab-9b1e-4dab-8daf-86bd074bc85a"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:48:49.655773 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:49.655745 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/269934ab-9b1e-4dab-8daf-86bd074bc85a-service-ca-bundle" pod "269934ab-9b1e-4dab-8daf-86bd074bc85a" (UID: "269934ab-9b1e-4dab-8daf-86bd074bc85a"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:48:49.655908 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:49.655890 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/269934ab-9b1e-4dab-8daf-86bd074bc85a-trusted-ca-bundle" pod "269934ab-9b1e-4dab-8daf-86bd074bc85a" (UID: "269934ab-9b1e-4dab-8daf-86bd074bc85a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:48:49.657470 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:49.657448 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/269934ab-9b1e-4dab-8daf-86bd074bc85a-kube-api-access-ns9wx" pod "269934ab-9b1e-4dab-8daf-86bd074bc85a" (UID: "269934ab-9b1e-4dab-8daf-86bd074bc85a"). InnerVolumeSpecName "kube-api-access-ns9wx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:48:49.756200 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:49.756112 5088 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/269934ab-9b1e-4dab-8daf-86bd074bc85a-archives-path\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 19:48:49.756200 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:49.756145 5088 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/269934ab-9b1e-4dab-8daf-86bd074bc85a-service-ca-bundle\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 19:48:49.756200 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:49.756155 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns9wx\" (UniqueName: \"kubernetes.io/projected/269934ab-9b1e-4dab-8daf-86bd074bc85a-kube-api-access-ns9wx\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 19:48:49.756200 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:49.756164 5088 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/269934ab-9b1e-4dab-8daf-86bd074bc85a-trusted-ca-bundle\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 19:48:50.324391 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:50.324360 5088 generic.go:356] "Generic (PLEG): container finished" podID="269934ab-9b1e-4dab-8daf-86bd074bc85a" containerID="4468aa564ca1874f89756ea15ed177014ac56af1a2aa0ba56d3c42e48a782636" exitCode=2 Oct 30 19:48:50.324578 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:50.324401 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-nmpzq-wm6lg" event={"ID":"269934ab-9b1e-4dab-8daf-86bd074bc85a","Type":"ContainerDied","Data":"4468aa564ca1874f89756ea15ed177014ac56af1a2aa0ba56d3c42e48a782636"} Oct 30 19:48:50.324578 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:50.324408 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-nmpzq-wm6lg" Oct 30 19:48:50.324578 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:50.324442 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-nmpzq-wm6lg" event={"ID":"269934ab-9b1e-4dab-8daf-86bd074bc85a","Type":"ContainerDied","Data":"daa0869c1fa90993b15220900b3fbcff993bb6f9a998d3989861058c948f4c8a"} Oct 30 19:48:50.324578 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:50.324462 5088 scope.go:122] "RemoveContainer" containerID="4468aa564ca1874f89756ea15ed177014ac56af1a2aa0ba56d3c42e48a782636" Oct 30 19:48:50.333195 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:50.333172 5088 scope.go:122] "RemoveContainer" containerID="4468aa564ca1874f89756ea15ed177014ac56af1a2aa0ba56d3c42e48a782636" Oct 30 19:48:50.333450 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:48:50.333431 5088 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4468aa564ca1874f89756ea15ed177014ac56af1a2aa0ba56d3c42e48a782636\": container with ID starting with 4468aa564ca1874f89756ea15ed177014ac56af1a2aa0ba56d3c42e48a782636 not found: ID does not exist" containerID="4468aa564ca1874f89756ea15ed177014ac56af1a2aa0ba56d3c42e48a782636" Oct 30 19:48:50.333497 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:50.333458 5088 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4468aa564ca1874f89756ea15ed177014ac56af1a2aa0ba56d3c42e48a782636"} err="failed to get container status \"4468aa564ca1874f89756ea15ed177014ac56af1a2aa0ba56d3c42e48a782636\": rpc error: code = NotFound desc = could not find container \"4468aa564ca1874f89756ea15ed177014ac56af1a2aa0ba56d3c42e48a782636\": container with ID starting with 4468aa564ca1874f89756ea15ed177014ac56af1a2aa0ba56d3c42e48a782636 not found: ID does not exist" Oct 30 19:48:50.388260 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:50.388232 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-nmpzq-wm6lg"] Oct 30 19:48:50.401727 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:50.401699 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-nmpzq-wm6lg"] Oct 30 19:48:50.737420 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:48:50.737391 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="269934ab-9b1e-4dab-8daf-86bd074bc85a" path="/var/lib/kubelet/pods/269934ab-9b1e-4dab-8daf-86bd074bc85a/volumes" Oct 30 19:48:52.436434 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:48:52.436390 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 19:49:22.434862 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:49:22.434829 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 19:49:52.435866 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:49:52.435781 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 19:50:22.435080 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:50:22.435034 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 19:50:52.436119 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:50:52.436082 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 19:51:22.434573 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:51:22.434529 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 19:51:52.435439 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:51:52.435399 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 19:52:15.342723 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:15.342667 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f6c7b487f-g84tm"] Oct 30 19:52:15.343150 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:15.342959 5088 cpu_manager.go:414] "containerMap: removing container" podUID="b1440092-9901-4eff-8a9d-6174b2dfdada" containerName="registry" Oct 30 19:52:15.343150 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:15.342969 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1440092-9901-4eff-8a9d-6174b2dfdada" containerName="registry" Oct 30 19:52:15.343150 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:15.342992 5088 cpu_manager.go:414] "containerMap: removing container" podUID="005bece8-8d37-4a58-954c-0ae3dab9dadc" containerName="insights-gathering" Oct 30 19:52:15.343150 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:15.342997 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="005bece8-8d37-4a58-954c-0ae3dab9dadc" containerName="insights-gathering" Oct 30 19:52:15.343150 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:15.343010 5088 cpu_manager.go:414] "containerMap: removing container" podUID="269934ab-9b1e-4dab-8daf-86bd074bc85a" containerName="insights-gathering" Oct 30 19:52:15.343150 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:15.343015 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="269934ab-9b1e-4dab-8daf-86bd074bc85a" containerName="insights-gathering" Oct 30 19:52:15.343150 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:15.343022 5088 cpu_manager.go:414] "containerMap: removing container" podUID="a16466d6-9ae1-4791-ad64-092e8d1cfa6f" containerName="console" Oct 30 19:52:15.343150 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:15.343029 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a16466d6-9ae1-4791-ad64-092e8d1cfa6f" containerName="console" Oct 30 19:52:15.343150 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:15.343036 5088 cpu_manager.go:414] "containerMap: removing container" podUID="9a8ef63e-bcc4-4b71-b345-3fcbc48f4396" containerName="registry" Oct 30 19:52:15.343150 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:15.343041 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a8ef63e-bcc4-4b71-b345-3fcbc48f4396" containerName="registry" Oct 30 19:52:15.343150 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:15.343088 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="269934ab-9b1e-4dab-8daf-86bd074bc85a" containerName="insights-gathering" Oct 30 19:52:15.343150 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:15.343095 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="a16466d6-9ae1-4791-ad64-092e8d1cfa6f" containerName="console" Oct 30 19:52:15.343150 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:15.343101 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a8ef63e-bcc4-4b71-b345-3fcbc48f4396" containerName="registry" Oct 30 19:52:15.343150 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:15.343106 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1440092-9901-4eff-8a9d-6174b2dfdada" containerName="registry" Oct 30 19:52:15.343150 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:15.343114 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="005bece8-8d37-4a58-954c-0ae3dab9dadc" containerName="insights-gathering" Oct 30 19:52:15.346061 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:15.346046 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f6c7b487f-g84tm" Oct 30 19:52:15.374704 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:15.374683 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 19:52:15.379288 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:15.379264 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 19:52:15.379288 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:15.379283 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\"" Oct 30 19:52:15.379413 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:15.379273 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 19:52:15.379413 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:15.379321 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 19:52:15.379413 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:15.379328 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-4r48g\"" Oct 30 19:52:15.386253 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:15.386203 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f6c7b487f-g84tm"] Oct 30 19:52:15.464459 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:15.464434 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c4416f2d-d31c-4a91-b910-66ef37e203b5-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f6c7b487f-g84tm\" (UID: \"c4416f2d-d31c-4a91-b910-66ef37e203b5\") " pod="opendatahub/opendatahub-operator-controller-manager-6f6c7b487f-g84tm" Oct 30 19:52:15.464576 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:15.464468 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4416f2d-d31c-4a91-b910-66ef37e203b5-tmp\") pod \"opendatahub-operator-controller-manager-6f6c7b487f-g84tm\" (UID: \"c4416f2d-d31c-4a91-b910-66ef37e203b5\") " pod="opendatahub/opendatahub-operator-controller-manager-6f6c7b487f-g84tm" Oct 30 19:52:15.464576 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:15.464494 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c4416f2d-d31c-4a91-b910-66ef37e203b5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f6c7b487f-g84tm\" (UID: \"c4416f2d-d31c-4a91-b910-66ef37e203b5\") " pod="opendatahub/opendatahub-operator-controller-manager-6f6c7b487f-g84tm" Oct 30 19:52:15.464676 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:15.464605 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q5p7\" (UniqueName: \"kubernetes.io/projected/c4416f2d-d31c-4a91-b910-66ef37e203b5-kube-api-access-8q5p7\") pod \"opendatahub-operator-controller-manager-6f6c7b487f-g84tm\" (UID: \"c4416f2d-d31c-4a91-b910-66ef37e203b5\") " pod="opendatahub/opendatahub-operator-controller-manager-6f6c7b487f-g84tm" Oct 30 19:52:15.464724 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:15.464676 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c4416f2d-d31c-4a91-b910-66ef37e203b5-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f6c7b487f-g84tm\" (UID: \"c4416f2d-d31c-4a91-b910-66ef37e203b5\") " pod="opendatahub/opendatahub-operator-controller-manager-6f6c7b487f-g84tm" Oct 30 19:52:15.464724 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:15.464695 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c4416f2d-d31c-4a91-b910-66ef37e203b5-manifests\") pod \"opendatahub-operator-controller-manager-6f6c7b487f-g84tm\" (UID: \"c4416f2d-d31c-4a91-b910-66ef37e203b5\") " pod="opendatahub/opendatahub-operator-controller-manager-6f6c7b487f-g84tm" Oct 30 19:52:15.565093 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:15.565067 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c4416f2d-d31c-4a91-b910-66ef37e203b5-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f6c7b487f-g84tm\" (UID: \"c4416f2d-d31c-4a91-b910-66ef37e203b5\") " pod="opendatahub/opendatahub-operator-controller-manager-6f6c7b487f-g84tm" Oct 30 19:52:15.565093 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:15.565096 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c4416f2d-d31c-4a91-b910-66ef37e203b5-manifests\") pod \"opendatahub-operator-controller-manager-6f6c7b487f-g84tm\" (UID: \"c4416f2d-d31c-4a91-b910-66ef37e203b5\") " pod="opendatahub/opendatahub-operator-controller-manager-6f6c7b487f-g84tm" Oct 30 19:52:15.565269 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:15.565120 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c4416f2d-d31c-4a91-b910-66ef37e203b5-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f6c7b487f-g84tm\" (UID: \"c4416f2d-d31c-4a91-b910-66ef37e203b5\") " pod="opendatahub/opendatahub-operator-controller-manager-6f6c7b487f-g84tm" Oct 30 19:52:15.565269 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:15.565152 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4416f2d-d31c-4a91-b910-66ef37e203b5-tmp\") pod \"opendatahub-operator-controller-manager-6f6c7b487f-g84tm\" (UID: \"c4416f2d-d31c-4a91-b910-66ef37e203b5\") " pod="opendatahub/opendatahub-operator-controller-manager-6f6c7b487f-g84tm" Oct 30 19:52:15.565269 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:15.565185 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c4416f2d-d31c-4a91-b910-66ef37e203b5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f6c7b487f-g84tm\" (UID: \"c4416f2d-d31c-4a91-b910-66ef37e203b5\") " pod="opendatahub/opendatahub-operator-controller-manager-6f6c7b487f-g84tm" Oct 30 19:52:15.565269 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:15.565230 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q5p7\" (UniqueName: \"kubernetes.io/projected/c4416f2d-d31c-4a91-b910-66ef37e203b5-kube-api-access-8q5p7\") pod \"opendatahub-operator-controller-manager-6f6c7b487f-g84tm\" (UID: \"c4416f2d-d31c-4a91-b910-66ef37e203b5\") " pod="opendatahub/opendatahub-operator-controller-manager-6f6c7b487f-g84tm" Oct 30 19:52:15.565528 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:15.565507 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c4416f2d-d31c-4a91-b910-66ef37e203b5-manifests\") pod \"opendatahub-operator-controller-manager-6f6c7b487f-g84tm\" (UID: \"c4416f2d-d31c-4a91-b910-66ef37e203b5\") " pod="opendatahub/opendatahub-operator-controller-manager-6f6c7b487f-g84tm" Oct 30 19:52:15.567420 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:15.567392 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4416f2d-d31c-4a91-b910-66ef37e203b5-tmp\") pod \"opendatahub-operator-controller-manager-6f6c7b487f-g84tm\" (UID: \"c4416f2d-d31c-4a91-b910-66ef37e203b5\") " pod="opendatahub/opendatahub-operator-controller-manager-6f6c7b487f-g84tm" Oct 30 19:52:15.567701 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:15.567686 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c4416f2d-d31c-4a91-b910-66ef37e203b5-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f6c7b487f-g84tm\" (UID: \"c4416f2d-d31c-4a91-b910-66ef37e203b5\") " pod="opendatahub/opendatahub-operator-controller-manager-6f6c7b487f-g84tm" Oct 30 19:52:15.567747 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:15.567700 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c4416f2d-d31c-4a91-b910-66ef37e203b5-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f6c7b487f-g84tm\" (UID: \"c4416f2d-d31c-4a91-b910-66ef37e203b5\") " pod="opendatahub/opendatahub-operator-controller-manager-6f6c7b487f-g84tm" Oct 30 19:52:15.567747 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:15.567725 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c4416f2d-d31c-4a91-b910-66ef37e203b5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f6c7b487f-g84tm\" (UID: \"c4416f2d-d31c-4a91-b910-66ef37e203b5\") " pod="opendatahub/opendatahub-operator-controller-manager-6f6c7b487f-g84tm" Oct 30 19:52:15.598644 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:15.598574 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q5p7\" (UniqueName: \"kubernetes.io/projected/c4416f2d-d31c-4a91-b910-66ef37e203b5-kube-api-access-8q5p7\") pod \"opendatahub-operator-controller-manager-6f6c7b487f-g84tm\" (UID: \"c4416f2d-d31c-4a91-b910-66ef37e203b5\") " pod="opendatahub/opendatahub-operator-controller-manager-6f6c7b487f-g84tm" Oct 30 19:52:15.660454 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:15.660431 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f6c7b487f-g84tm" Oct 30 19:52:16.023884 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:16.023850 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f6c7b487f-g84tm"] Oct 30 19:52:16.039208 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:52:16.039180 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4416f2d_d31c_4a91_b910_66ef37e203b5.slice/crio-7f2719d31ced7de4bba78623c3b7ea70640d4858ecbaa150beadd51d6388ebbc WatchSource:0}: Error finding container 7f2719d31ced7de4bba78623c3b7ea70640d4858ecbaa150beadd51d6388ebbc: Status 404 returned error can't find the container with id 7f2719d31ced7de4bba78623c3b7ea70640d4858ecbaa150beadd51d6388ebbc Oct 30 19:52:16.112461 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:16.112433 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f6c7b487f-g84tm" event={"ID":"c4416f2d-d31c-4a91-b910-66ef37e203b5","Type":"ContainerStarted","Data":"7f2719d31ced7de4bba78623c3b7ea70640d4858ecbaa150beadd51d6388ebbc"} Oct 30 19:52:18.747737 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:18.747678 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qdj4h_da850825-59ab-480f-9047-afd7d2bdc19f/console-operator/1.log" Oct 30 19:52:18.749603 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:18.749581 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qdj4h_da850825-59ab-480f-9047-afd7d2bdc19f/console-operator/1.log" Oct 30 19:52:18.752394 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:18.752375 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xpjpn_688d079a-932d-4f63-97da-5cd6222a03e1/ovn-acl-logging/0.log" Oct 30 19:52:18.754062 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:18.754043 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xpjpn_688d079a-932d-4f63-97da-5cd6222a03e1/ovn-acl-logging/0.log" Oct 30 19:52:19.128985 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:19.128952 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f6c7b487f-g84tm" event={"ID":"c4416f2d-d31c-4a91-b910-66ef37e203b5","Type":"ContainerStarted","Data":"f63fd2dd8fd2d6437845c6d3bf34a2d7b809afa3398272d08a6ef8056e6fa155"} Oct 30 19:52:20.134981 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:20.134949 5088 generic.go:356] "Generic (PLEG): container finished" podID="c4416f2d-d31c-4a91-b910-66ef37e203b5" containerID="f63fd2dd8fd2d6437845c6d3bf34a2d7b809afa3398272d08a6ef8056e6fa155" exitCode=0 Oct 30 19:52:20.135437 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:20.135040 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f6c7b487f-g84tm" event={"ID":"c4416f2d-d31c-4a91-b910-66ef37e203b5","Type":"ContainerDied","Data":"f63fd2dd8fd2d6437845c6d3bf34a2d7b809afa3398272d08a6ef8056e6fa155"} Oct 30 19:52:21.140650 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:21.140605 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f6c7b487f-g84tm" event={"ID":"c4416f2d-d31c-4a91-b910-66ef37e203b5","Type":"ContainerStarted","Data":"70de913fd37ca88339860eaf662134e7c1e44995ecfced735fa565edb2e4688e"} Oct 30 19:52:21.141027 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:21.140765 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6f6c7b487f-g84tm" Oct 30 19:52:21.257673 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:21.257601 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6f6c7b487f-g84tm" podStartSLOduration=2.841688528 podStartE2EDuration="6.257587546s" totalImagesPullingTime="3.002908985s" totalInitContainerRuntime="412.990033ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:21.256992445 +0000 UTC m=+302.669706235" watchObservedRunningTime="2026-10-30 19:52:21.257587546 +0000 UTC m=+302.670301324" Oct 30 19:52:22.435135 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:52:22.435089 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 19:52:32.146967 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:32.146889 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6f6c7b487f-g84tm" Oct 30 19:52:52.435824 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:52:52.435784 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 19:52:55.801018 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:55.800985 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6f5c96996f-wsql6"] Oct 30 19:52:55.807444 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:55.807423 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6f5c96996f-wsql6" Oct 30 19:52:55.827207 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:55.827180 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6f5c96996f-wsql6"] Oct 30 19:52:55.839785 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:55.839751 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 19:52:55.839785 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:55.839750 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-jxhc6\"" Oct 30 19:52:55.839785 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:55.839753 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 19:52:55.890042 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:55.890010 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mm5l\" (UniqueName: \"kubernetes.io/projected/e4e0fd61-e6e4-4d87-a607-ad397cc39697-kube-api-access-2mm5l\") pod \"kube-auth-proxy-6f5c96996f-wsql6\" (UID: \"e4e0fd61-e6e4-4d87-a607-ad397cc39697\") " pod="openshift-ingress/kube-auth-proxy-6f5c96996f-wsql6" Oct 30 19:52:55.890189 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:55.890049 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4e0fd61-e6e4-4d87-a607-ad397cc39697-tls-certs\") pod \"kube-auth-proxy-6f5c96996f-wsql6\" (UID: \"e4e0fd61-e6e4-4d87-a607-ad397cc39697\") " pod="openshift-ingress/kube-auth-proxy-6f5c96996f-wsql6" Oct 30 19:52:55.890189 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:55.890101 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4e0fd61-e6e4-4d87-a607-ad397cc39697-tmp\") pod \"kube-auth-proxy-6f5c96996f-wsql6\" (UID: \"e4e0fd61-e6e4-4d87-a607-ad397cc39697\") " pod="openshift-ingress/kube-auth-proxy-6f5c96996f-wsql6" Oct 30 19:52:55.990750 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:55.990709 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mm5l\" (UniqueName: \"kubernetes.io/projected/e4e0fd61-e6e4-4d87-a607-ad397cc39697-kube-api-access-2mm5l\") pod \"kube-auth-proxy-6f5c96996f-wsql6\" (UID: \"e4e0fd61-e6e4-4d87-a607-ad397cc39697\") " pod="openshift-ingress/kube-auth-proxy-6f5c96996f-wsql6" Oct 30 19:52:55.990750 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:55.990756 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4e0fd61-e6e4-4d87-a607-ad397cc39697-tls-certs\") pod \"kube-auth-proxy-6f5c96996f-wsql6\" (UID: \"e4e0fd61-e6e4-4d87-a607-ad397cc39697\") " pod="openshift-ingress/kube-auth-proxy-6f5c96996f-wsql6" Oct 30 19:52:55.990952 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:55.990809 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4e0fd61-e6e4-4d87-a607-ad397cc39697-tmp\") pod \"kube-auth-proxy-6f5c96996f-wsql6\" (UID: \"e4e0fd61-e6e4-4d87-a607-ad397cc39697\") " pod="openshift-ingress/kube-auth-proxy-6f5c96996f-wsql6" Oct 30 19:52:55.993090 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:55.993062 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4e0fd61-e6e4-4d87-a607-ad397cc39697-tmp\") pod \"kube-auth-proxy-6f5c96996f-wsql6\" (UID: \"e4e0fd61-e6e4-4d87-a607-ad397cc39697\") " pod="openshift-ingress/kube-auth-proxy-6f5c96996f-wsql6" Oct 30 19:52:55.993236 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:55.993216 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4e0fd61-e6e4-4d87-a607-ad397cc39697-tls-certs\") pod \"kube-auth-proxy-6f5c96996f-wsql6\" (UID: \"e4e0fd61-e6e4-4d87-a607-ad397cc39697\") " pod="openshift-ingress/kube-auth-proxy-6f5c96996f-wsql6" Oct 30 19:52:56.017643 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:56.017601 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mm5l\" (UniqueName: \"kubernetes.io/projected/e4e0fd61-e6e4-4d87-a607-ad397cc39697-kube-api-access-2mm5l\") pod \"kube-auth-proxy-6f5c96996f-wsql6\" (UID: \"e4e0fd61-e6e4-4d87-a607-ad397cc39697\") " pod="openshift-ingress/kube-auth-proxy-6f5c96996f-wsql6" Oct 30 19:52:56.119469 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:56.119375 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6f5c96996f-wsql6" Oct 30 19:52:56.427749 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:56.427557 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6f5c96996f-wsql6"] Oct 30 19:52:56.430404 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:52:56.430376 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4e0fd61_e6e4_4d87_a607_ad397cc39697.slice/crio-34650c7d18c5f31c8ab73c1ed9b073237599d779b138c44d9837807379857e1f WatchSource:0}: Error finding container 34650c7d18c5f31c8ab73c1ed9b073237599d779b138c44d9837807379857e1f: Status 404 returned error can't find the container with id 34650c7d18c5f31c8ab73c1ed9b073237599d779b138c44d9837807379857e1f Oct 30 19:52:56.432283 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:56.432266 5088 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:52:57.283127 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:57.283090 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6f5c96996f-wsql6" event={"ID":"e4e0fd61-e6e4-4d87-a607-ad397cc39697","Type":"ContainerStarted","Data":"34650c7d18c5f31c8ab73c1ed9b073237599d779b138c44d9837807379857e1f"} Oct 30 19:52:58.049245 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:58.049211 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-vsnwj"] Oct 30 19:52:58.057470 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:58.057445 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vsnwj" Oct 30 19:52:58.075546 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:58.075526 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 19:52:58.075703 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:58.075525 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 19:52:58.078021 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:58.077985 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-r79wj\"" Oct 30 19:52:58.093640 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:58.091401 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-vsnwj"] Oct 30 19:52:58.093640 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:58.091525 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 19:52:58.111077 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:58.111047 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d155893f-7afa-49d3-b3cf-67086d9d3bdd-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-vsnwj\" (UID: \"d155893f-7afa-49d3-b3cf-67086d9d3bdd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vsnwj" Oct 30 19:52:58.111208 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:58.111082 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bn6q\" (UniqueName: \"kubernetes.io/projected/d155893f-7afa-49d3-b3cf-67086d9d3bdd-kube-api-access-6bn6q\") pod \"istiod-openshift-gateway-7876bfcb74-vsnwj\" (UID: \"d155893f-7afa-49d3-b3cf-67086d9d3bdd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vsnwj" Oct 30 19:52:58.111208 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:58.111105 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d155893f-7afa-49d3-b3cf-67086d9d3bdd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-vsnwj\" (UID: \"d155893f-7afa-49d3-b3cf-67086d9d3bdd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vsnwj" Oct 30 19:52:58.111286 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:58.111210 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d155893f-7afa-49d3-b3cf-67086d9d3bdd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-vsnwj\" (UID: \"d155893f-7afa-49d3-b3cf-67086d9d3bdd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vsnwj" Oct 30 19:52:58.111286 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:58.111232 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d155893f-7afa-49d3-b3cf-67086d9d3bdd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-vsnwj\" (UID: \"d155893f-7afa-49d3-b3cf-67086d9d3bdd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vsnwj" Oct 30 19:52:58.111286 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:58.111254 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d155893f-7afa-49d3-b3cf-67086d9d3bdd-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-vsnwj\" (UID: \"d155893f-7afa-49d3-b3cf-67086d9d3bdd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vsnwj" Oct 30 19:52:58.111286 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:58.111271 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d155893f-7afa-49d3-b3cf-67086d9d3bdd-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-vsnwj\" (UID: \"d155893f-7afa-49d3-b3cf-67086d9d3bdd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vsnwj" Oct 30 19:52:58.175967 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:58.175934 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 19:52:58.212667 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:58.212632 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d155893f-7afa-49d3-b3cf-67086d9d3bdd-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-vsnwj\" (UID: \"d155893f-7afa-49d3-b3cf-67086d9d3bdd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vsnwj" Oct 30 19:52:58.212667 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:58.212679 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bn6q\" (UniqueName: \"kubernetes.io/projected/d155893f-7afa-49d3-b3cf-67086d9d3bdd-kube-api-access-6bn6q\") pod \"istiod-openshift-gateway-7876bfcb74-vsnwj\" (UID: \"d155893f-7afa-49d3-b3cf-67086d9d3bdd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vsnwj" Oct 30 19:52:58.212892 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:58.212714 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d155893f-7afa-49d3-b3cf-67086d9d3bdd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-vsnwj\" (UID: \"d155893f-7afa-49d3-b3cf-67086d9d3bdd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vsnwj" Oct 30 19:52:58.212892 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:58.212743 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d155893f-7afa-49d3-b3cf-67086d9d3bdd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-vsnwj\" (UID: \"d155893f-7afa-49d3-b3cf-67086d9d3bdd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vsnwj" Oct 30 19:52:58.212892 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:58.212767 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d155893f-7afa-49d3-b3cf-67086d9d3bdd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-vsnwj\" (UID: \"d155893f-7afa-49d3-b3cf-67086d9d3bdd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vsnwj" Oct 30 19:52:58.212892 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:58.212801 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d155893f-7afa-49d3-b3cf-67086d9d3bdd-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-vsnwj\" (UID: \"d155893f-7afa-49d3-b3cf-67086d9d3bdd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vsnwj" Oct 30 19:52:58.212892 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:58.212825 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d155893f-7afa-49d3-b3cf-67086d9d3bdd-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-vsnwj\" (UID: \"d155893f-7afa-49d3-b3cf-67086d9d3bdd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vsnwj" Oct 30 19:52:58.213686 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:58.213645 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d155893f-7afa-49d3-b3cf-67086d9d3bdd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-vsnwj\" (UID: \"d155893f-7afa-49d3-b3cf-67086d9d3bdd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vsnwj" Oct 30 19:52:58.215416 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:58.215368 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d155893f-7afa-49d3-b3cf-67086d9d3bdd-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-vsnwj\" (UID: \"d155893f-7afa-49d3-b3cf-67086d9d3bdd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vsnwj" Oct 30 19:52:58.215535 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:58.215441 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d155893f-7afa-49d3-b3cf-67086d9d3bdd-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-vsnwj\" (UID: \"d155893f-7afa-49d3-b3cf-67086d9d3bdd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vsnwj" Oct 30 19:52:58.215678 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:58.215647 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d155893f-7afa-49d3-b3cf-67086d9d3bdd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-vsnwj\" (UID: \"d155893f-7afa-49d3-b3cf-67086d9d3bdd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vsnwj" Oct 30 19:52:58.215796 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:58.215713 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d155893f-7afa-49d3-b3cf-67086d9d3bdd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-vsnwj\" (UID: \"d155893f-7afa-49d3-b3cf-67086d9d3bdd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vsnwj" Oct 30 19:52:58.238964 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:58.238930 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d155893f-7afa-49d3-b3cf-67086d9d3bdd-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-vsnwj\" (UID: \"d155893f-7afa-49d3-b3cf-67086d9d3bdd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vsnwj" Oct 30 19:52:58.239454 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:58.239431 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bn6q\" (UniqueName: \"kubernetes.io/projected/d155893f-7afa-49d3-b3cf-67086d9d3bdd-kube-api-access-6bn6q\") pod \"istiod-openshift-gateway-7876bfcb74-vsnwj\" (UID: \"d155893f-7afa-49d3-b3cf-67086d9d3bdd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vsnwj" Oct 30 19:52:58.371369 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:58.371286 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vsnwj" Oct 30 19:52:58.693215 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:58.693128 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-vsnwj"] Oct 30 19:52:59.293168 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:59.293134 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6f5c96996f-wsql6" event={"ID":"e4e0fd61-e6e4-4d87-a607-ad397cc39697","Type":"ContainerStarted","Data":"375ae40934044dc6e4a792c7330ec85e4528b93ae0d6c54197b066b92fff7e00"} Oct 30 19:52:59.294310 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:59.294283 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vsnwj" event={"ID":"d155893f-7afa-49d3-b3cf-67086d9d3bdd","Type":"ContainerStarted","Data":"3a174bb483414eb23a0f09c5ad784c9bee061dbcce5ece490acad3b6d5ff7d04"} Oct 30 19:52:59.341128 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:52:59.340529 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6f5c96996f-wsql6" podStartSLOduration=1.9741559309999999 podStartE2EDuration="4.340513353s" totalImagesPullingTime="2.366357422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:59.339243018 +0000 UTC m=+340.751956795" watchObservedRunningTime="2026-10-30 19:52:59.340513353 +0000 UTC m=+340.753227131" Oct 30 19:53:01.038581 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:01.038547 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:53:01.038982 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:01.038634 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:53:01.304684 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:01.304577 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vsnwj" event={"ID":"d155893f-7afa-49d3-b3cf-67086d9d3bdd","Type":"ContainerStarted","Data":"ca43b5f9a1849a226ce8df25f0e4959c381e181af61916f1e6b5f52618f08e3f"} Oct 30 19:53:01.304843 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:01.304756 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vsnwj" Oct 30 19:53:01.381343 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:01.381294 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vsnwj" podStartSLOduration=1.047003986 podStartE2EDuration="3.381279363s" totalImagesPullingTime="2.334275377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:01.366807556 +0000 UTC m=+342.779521336" watchObservedRunningTime="2026-10-30 19:53:01.381279363 +0000 UTC m=+342.793993135" Oct 30 19:53:02.310471 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:02.310441 5088 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-vsnwj container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:53:02.310927 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:02.310497 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vsnwj" podUID="d155893f-7afa-49d3-b3cf-67086d9d3bdd" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:53:02.742269 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:02.742229 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6"] Oct 30 19:53:02.745546 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:02.745525 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6" Oct 30 19:53:02.779083 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:02.779057 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6"] Oct 30 19:53:02.807404 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:02.807373 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-xqk7z\"" Oct 30 19:53:02.849821 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:02.849789 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6\" (UID: \"825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6" Oct 30 19:53:02.849821 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:02.849822 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x22f2\" (UniqueName: \"kubernetes.io/projected/825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1-kube-api-access-x22f2\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6\" (UID: \"825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6" Oct 30 19:53:02.849992 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:02.849842 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6\" (UID: \"825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6" Oct 30 19:53:02.849992 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:02.849862 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6\" (UID: \"825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6" Oct 30 19:53:02.849992 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:02.849923 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6\" (UID: \"825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6" Oct 30 19:53:02.849992 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:02.849980 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6\" (UID: \"825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6" Oct 30 19:53:02.850715 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:02.850038 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6\" (UID: \"825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6" Oct 30 19:53:02.850715 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:02.850088 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6\" (UID: \"825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6" Oct 30 19:53:02.850715 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:02.850117 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6\" (UID: \"825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6" Oct 30 19:53:02.951417 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:02.951385 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6\" (UID: \"825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6" Oct 30 19:53:02.951417 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:02.951420 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x22f2\" (UniqueName: \"kubernetes.io/projected/825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1-kube-api-access-x22f2\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6\" (UID: \"825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6" Oct 30 19:53:02.951637 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:02.951442 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6\" (UID: \"825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6" Oct 30 19:53:02.951637 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:02.951479 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6\" (UID: \"825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6" Oct 30 19:53:02.951637 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:02.951497 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6\" (UID: \"825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6" Oct 30 19:53:02.951637 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:02.951517 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6\" (UID: \"825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6" Oct 30 19:53:02.951637 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:02.951539 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6\" (UID: \"825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6" Oct 30 19:53:02.951637 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:02.951581 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6\" (UID: \"825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6" Oct 30 19:53:02.951930 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:02.951687 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6\" (UID: \"825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6" Oct 30 19:53:02.951930 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:02.951877 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6\" (UID: \"825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6" Oct 30 19:53:02.952051 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:02.951940 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6\" (UID: \"825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6" Oct 30 19:53:02.952116 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:02.952038 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6\" (UID: \"825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6" Oct 30 19:53:02.952177 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:02.952156 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6\" (UID: \"825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6" Oct 30 19:53:02.952275 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:02.952252 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6\" (UID: \"825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6" Oct 30 19:53:02.953995 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:02.953975 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6\" (UID: \"825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6" Oct 30 19:53:02.954084 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:02.954056 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6\" (UID: \"825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6" Oct 30 19:53:02.977141 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:02.977107 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6\" (UID: \"825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6" Oct 30 19:53:02.977246 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:02.977224 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x22f2\" (UniqueName: \"kubernetes.io/projected/825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1-kube-api-access-x22f2\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6\" (UID: \"825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6" Oct 30 19:53:03.058814 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:03.058731 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6" Oct 30 19:53:03.379911 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:03.379885 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6"] Oct 30 19:53:03.382157 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:53:03.382134 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod825ed6e8_c40b_423a_97f2_b9b2dcc6cdd1.slice/crio-42556f73ec394dbdb21f4c7381201c425a6b0c06f39e4bc5861262c3264a87fb WatchSource:0}: Error finding container 42556f73ec394dbdb21f4c7381201c425a6b0c06f39e4bc5861262c3264a87fb: Status 404 returned error can't find the container with id 42556f73ec394dbdb21f4c7381201c425a6b0c06f39e4bc5861262c3264a87fb Oct 30 19:53:04.319265 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:04.319228 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6" event={"ID":"825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1","Type":"ContainerStarted","Data":"42556f73ec394dbdb21f4c7381201c425a6b0c06f39e4bc5861262c3264a87fb"} Oct 30 19:53:05.310006 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:05.309969 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vsnwj" Oct 30 19:53:06.133874 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:06.133844 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:53:06.133976 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:06.133911 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:53:06.133976 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:06.133942 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:53:06.330644 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:06.330591 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6" event={"ID":"825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1","Type":"ContainerStarted","Data":"69218028bcaa6268f20344af875ce9c6807cc6816e36966c1197418781fbfaec"} Oct 30 19:53:06.390424 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:06.390337 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6" podStartSLOduration=1.6413452689999999 podStartE2EDuration="4.390325056s" totalImagesPullingTime="2.748979787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:06.389339576 +0000 UTC m=+347.802053354" watchObservedRunningTime="2026-10-30 19:53:06.390325056 +0000 UTC m=+347.803038901" Oct 30 19:53:07.059593 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:07.059558 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6" Oct 30 19:53:07.064143 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:07.064116 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6" Oct 30 19:53:07.334778 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:07.334699 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6" Oct 30 19:53:07.335849 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:53:07.335831 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6" Oct 30 19:53:22.437142 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:53:22.437088 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 19:53:52.435859 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:53:52.435818 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 19:54:11.634503 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:11.634471 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l9g8p"] Oct 30 19:54:11.639140 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:11.639124 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l9g8p" Oct 30 19:54:11.672293 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:11.672266 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l9g8p"] Oct 30 19:54:11.699519 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:11.699487 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ed843517-0e6d-44b9-a6f9-4c6e686bad29-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l9g8p\" (UID: \"ed843517-0e6d-44b9-a6f9-4c6e686bad29\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l9g8p" Oct 30 19:54:11.699663 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:11.699628 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ed843517-0e6d-44b9-a6f9-4c6e686bad29-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l9g8p\" (UID: \"ed843517-0e6d-44b9-a6f9-4c6e686bad29\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l9g8p" Oct 30 19:54:11.699767 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:11.699736 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ed843517-0e6d-44b9-a6f9-4c6e686bad29-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l9g8p\" (UID: \"ed843517-0e6d-44b9-a6f9-4c6e686bad29\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l9g8p" Oct 30 19:54:11.699864 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:11.699831 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ed843517-0e6d-44b9-a6f9-4c6e686bad29-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l9g8p\" (UID: \"ed843517-0e6d-44b9-a6f9-4c6e686bad29\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l9g8p" Oct 30 19:54:11.699917 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:11.699898 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed843517-0e6d-44b9-a6f9-4c6e686bad29-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l9g8p\" (UID: \"ed843517-0e6d-44b9-a6f9-4c6e686bad29\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l9g8p" Oct 30 19:54:11.699958 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:11.699922 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed843517-0e6d-44b9-a6f9-4c6e686bad29-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l9g8p\" (UID: \"ed843517-0e6d-44b9-a6f9-4c6e686bad29\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l9g8p" Oct 30 19:54:11.699958 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:11.699950 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp5pm\" (UniqueName: \"kubernetes.io/projected/ed843517-0e6d-44b9-a6f9-4c6e686bad29-kube-api-access-jp5pm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l9g8p\" (UID: \"ed843517-0e6d-44b9-a6f9-4c6e686bad29\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l9g8p" Oct 30 19:54:11.800590 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:11.800557 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed843517-0e6d-44b9-a6f9-4c6e686bad29-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l9g8p\" (UID: \"ed843517-0e6d-44b9-a6f9-4c6e686bad29\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l9g8p" Oct 30 19:54:11.800789 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:11.800595 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp5pm\" (UniqueName: \"kubernetes.io/projected/ed843517-0e6d-44b9-a6f9-4c6e686bad29-kube-api-access-jp5pm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l9g8p\" (UID: \"ed843517-0e6d-44b9-a6f9-4c6e686bad29\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l9g8p" Oct 30 19:54:11.800789 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:11.800649 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ed843517-0e6d-44b9-a6f9-4c6e686bad29-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l9g8p\" (UID: \"ed843517-0e6d-44b9-a6f9-4c6e686bad29\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l9g8p" Oct 30 19:54:11.800789 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:11.800715 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ed843517-0e6d-44b9-a6f9-4c6e686bad29-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l9g8p\" (UID: \"ed843517-0e6d-44b9-a6f9-4c6e686bad29\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l9g8p" Oct 30 19:54:11.800789 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:11.800762 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ed843517-0e6d-44b9-a6f9-4c6e686bad29-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l9g8p\" (UID: \"ed843517-0e6d-44b9-a6f9-4c6e686bad29\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l9g8p" Oct 30 19:54:11.801027 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:11.800822 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ed843517-0e6d-44b9-a6f9-4c6e686bad29-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l9g8p\" (UID: \"ed843517-0e6d-44b9-a6f9-4c6e686bad29\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l9g8p" Oct 30 19:54:11.801027 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:11.800856 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed843517-0e6d-44b9-a6f9-4c6e686bad29-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l9g8p\" (UID: \"ed843517-0e6d-44b9-a6f9-4c6e686bad29\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l9g8p" Oct 30 19:54:11.801772 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:11.801748 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ed843517-0e6d-44b9-a6f9-4c6e686bad29-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l9g8p\" (UID: \"ed843517-0e6d-44b9-a6f9-4c6e686bad29\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l9g8p" Oct 30 19:54:11.803547 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:11.803519 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ed843517-0e6d-44b9-a6f9-4c6e686bad29-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l9g8p\" (UID: \"ed843517-0e6d-44b9-a6f9-4c6e686bad29\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l9g8p" Oct 30 19:54:11.803672 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:11.803551 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed843517-0e6d-44b9-a6f9-4c6e686bad29-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l9g8p\" (UID: \"ed843517-0e6d-44b9-a6f9-4c6e686bad29\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l9g8p" Oct 30 19:54:11.804119 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:11.804097 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ed843517-0e6d-44b9-a6f9-4c6e686bad29-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l9g8p\" (UID: \"ed843517-0e6d-44b9-a6f9-4c6e686bad29\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l9g8p" Oct 30 19:54:11.804650 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:11.804622 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ed843517-0e6d-44b9-a6f9-4c6e686bad29-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l9g8p\" (UID: \"ed843517-0e6d-44b9-a6f9-4c6e686bad29\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l9g8p" Oct 30 19:54:11.828441 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:11.828421 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed843517-0e6d-44b9-a6f9-4c6e686bad29-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l9g8p\" (UID: \"ed843517-0e6d-44b9-a6f9-4c6e686bad29\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l9g8p" Oct 30 19:54:11.828796 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:11.828776 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp5pm\" (UniqueName: \"kubernetes.io/projected/ed843517-0e6d-44b9-a6f9-4c6e686bad29-kube-api-access-jp5pm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l9g8p\" (UID: \"ed843517-0e6d-44b9-a6f9-4c6e686bad29\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l9g8p" Oct 30 19:54:11.950774 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:11.950692 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l9g8p" Oct 30 19:54:12.317121 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:12.317097 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l9g8p"] Oct 30 19:54:12.318270 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:54:12.318244 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded843517_0e6d_44b9_a6f9_4c6e686bad29.slice/crio-441f24f89ab5ede910939ad5c4284474947072367421f6557647d24a0e31da00 WatchSource:0}: Error finding container 441f24f89ab5ede910939ad5c4284474947072367421f6557647d24a0e31da00: Status 404 returned error can't find the container with id 441f24f89ab5ede910939ad5c4284474947072367421f6557647d24a0e31da00 Oct 30 19:54:12.320158 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:12.320130 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:54:12.320208 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:12.320199 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:54:12.599698 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:12.599603 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l9g8p" event={"ID":"ed843517-0e6d-44b9-a6f9-4c6e686bad29","Type":"ContainerStarted","Data":"a61a35e278671e7f1958632ff232fd00ea673f09b69ecc4613e486b2698b2074"} Oct 30 19:54:12.599698 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:12.599665 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l9g8p" event={"ID":"ed843517-0e6d-44b9-a6f9-4c6e686bad29","Type":"ContainerStarted","Data":"441f24f89ab5ede910939ad5c4284474947072367421f6557647d24a0e31da00"} Oct 30 19:54:12.599908 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:12.599801 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l9g8p" Oct 30 19:54:12.694907 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:12.694856 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l9g8p" podStartSLOduration=1.694842841 podStartE2EDuration="1.694842841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:12.694161091 +0000 UTC m=+414.106874892" watchObservedRunningTime="2026-10-30 19:54:12.694842841 +0000 UTC m=+414.107556618" Oct 30 19:54:13.605599 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:13.605569 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l9g8p" Oct 30 19:54:13.751445 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:13.751414 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-vsnwj"] Oct 30 19:54:13.751927 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:13.751602 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vsnwj" podUID="d155893f-7afa-49d3-b3cf-67086d9d3bdd" containerName="discovery" containerID="cri-o://ca43b5f9a1849a226ce8df25f0e4959c381e181af61916f1e6b5f52618f08e3f" gracePeriod=30 Oct 30 19:54:13.984416 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:13.984391 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vsnwj" Oct 30 19:54:14.017645 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:14.017621 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d155893f-7afa-49d3-b3cf-67086d9d3bdd-cacerts\" (UniqueName: \"kubernetes.io/secret/d155893f-7afa-49d3-b3cf-67086d9d3bdd-cacerts\") pod \"d155893f-7afa-49d3-b3cf-67086d9d3bdd\" (UID: \"d155893f-7afa-49d3-b3cf-67086d9d3bdd\") " Oct 30 19:54:14.017730 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:14.017702 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d155893f-7afa-49d3-b3cf-67086d9d3bdd-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d155893f-7afa-49d3-b3cf-67086d9d3bdd-istio-kubeconfig\") pod \"d155893f-7afa-49d3-b3cf-67086d9d3bdd\" (UID: \"d155893f-7afa-49d3-b3cf-67086d9d3bdd\") " Oct 30 19:54:14.017771 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:14.017736 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d155893f-7afa-49d3-b3cf-67086d9d3bdd-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d155893f-7afa-49d3-b3cf-67086d9d3bdd-istio-csr-ca-configmap\") pod \"d155893f-7afa-49d3-b3cf-67086d9d3bdd\" (UID: \"d155893f-7afa-49d3-b3cf-67086d9d3bdd\") " Oct 30 19:54:14.017771 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:14.017761 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d155893f-7afa-49d3-b3cf-67086d9d3bdd-istio-token\" (UniqueName: \"kubernetes.io/projected/d155893f-7afa-49d3-b3cf-67086d9d3bdd-istio-token\") pod \"d155893f-7afa-49d3-b3cf-67086d9d3bdd\" (UID: \"d155893f-7afa-49d3-b3cf-67086d9d3bdd\") " Oct 30 19:54:14.017977 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:14.017898 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d155893f-7afa-49d3-b3cf-67086d9d3bdd-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d155893f-7afa-49d3-b3cf-67086d9d3bdd-local-certs\") pod \"d155893f-7afa-49d3-b3cf-67086d9d3bdd\" (UID: \"d155893f-7afa-49d3-b3cf-67086d9d3bdd\") " Oct 30 19:54:14.018088 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:14.017984 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d155893f-7afa-49d3-b3cf-67086d9d3bdd-kube-api-access-6bn6q\" (UniqueName: \"kubernetes.io/projected/d155893f-7afa-49d3-b3cf-67086d9d3bdd-kube-api-access-6bn6q\") pod \"d155893f-7afa-49d3-b3cf-67086d9d3bdd\" (UID: \"d155893f-7afa-49d3-b3cf-67086d9d3bdd\") " Oct 30 19:54:14.018088 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:14.018036 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d155893f-7afa-49d3-b3cf-67086d9d3bdd-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d155893f-7afa-49d3-b3cf-67086d9d3bdd-istio-csr-dns-cert\") pod \"d155893f-7afa-49d3-b3cf-67086d9d3bdd\" (UID: \"d155893f-7afa-49d3-b3cf-67086d9d3bdd\") " Oct 30 19:54:14.018199 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:14.018105 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d155893f-7afa-49d3-b3cf-67086d9d3bdd-istio-csr-ca-configmap" pod "d155893f-7afa-49d3-b3cf-67086d9d3bdd" (UID: "d155893f-7afa-49d3-b3cf-67086d9d3bdd"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:14.018317 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:14.018301 5088 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d155893f-7afa-49d3-b3cf-67086d9d3bdd-istio-csr-ca-configmap\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 19:54:14.020001 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:14.019976 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d155893f-7afa-49d3-b3cf-67086d9d3bdd-istio-token" pod "d155893f-7afa-49d3-b3cf-67086d9d3bdd" (UID: "d155893f-7afa-49d3-b3cf-67086d9d3bdd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:14.020172 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:14.020145 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d155893f-7afa-49d3-b3cf-67086d9d3bdd-cacerts" pod "d155893f-7afa-49d3-b3cf-67086d9d3bdd" (UID: "d155893f-7afa-49d3-b3cf-67086d9d3bdd"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:54:14.020172 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:14.020160 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d155893f-7afa-49d3-b3cf-67086d9d3bdd-istio-kubeconfig" pod "d155893f-7afa-49d3-b3cf-67086d9d3bdd" (UID: "d155893f-7afa-49d3-b3cf-67086d9d3bdd"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:54:14.020308 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:14.020199 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d155893f-7afa-49d3-b3cf-67086d9d3bdd-local-certs" pod "d155893f-7afa-49d3-b3cf-67086d9d3bdd" (UID: "d155893f-7afa-49d3-b3cf-67086d9d3bdd"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:14.020468 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:14.020448 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d155893f-7afa-49d3-b3cf-67086d9d3bdd-istio-csr-dns-cert" pod "d155893f-7afa-49d3-b3cf-67086d9d3bdd" (UID: "d155893f-7afa-49d3-b3cf-67086d9d3bdd"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:54:14.020519 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:14.020468 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d155893f-7afa-49d3-b3cf-67086d9d3bdd-kube-api-access-6bn6q" pod "d155893f-7afa-49d3-b3cf-67086d9d3bdd" (UID: "d155893f-7afa-49d3-b3cf-67086d9d3bdd"). InnerVolumeSpecName "kube-api-access-6bn6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:14.119321 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:14.119247 5088 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d155893f-7afa-49d3-b3cf-67086d9d3bdd-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 19:54:14.119321 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:14.119272 5088 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d155893f-7afa-49d3-b3cf-67086d9d3bdd-local-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 19:54:14.119321 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:14.119281 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bn6q\" (UniqueName: \"kubernetes.io/projected/d155893f-7afa-49d3-b3cf-67086d9d3bdd-kube-api-access-6bn6q\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 19:54:14.119321 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:14.119291 5088 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d155893f-7afa-49d3-b3cf-67086d9d3bdd-istio-csr-dns-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 19:54:14.119321 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:14.119301 5088 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d155893f-7afa-49d3-b3cf-67086d9d3bdd-cacerts\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 19:54:14.119321 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:14.119311 5088 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d155893f-7afa-49d3-b3cf-67086d9d3bdd-istio-kubeconfig\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 19:54:14.609532 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:14.609499 5088 generic.go:356] "Generic (PLEG): container finished" podID="d155893f-7afa-49d3-b3cf-67086d9d3bdd" containerID="ca43b5f9a1849a226ce8df25f0e4959c381e181af61916f1e6b5f52618f08e3f" exitCode=0 Oct 30 19:54:14.609735 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:14.609548 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vsnwj" Oct 30 19:54:14.609735 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:14.609584 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vsnwj" event={"ID":"d155893f-7afa-49d3-b3cf-67086d9d3bdd","Type":"ContainerDied","Data":"ca43b5f9a1849a226ce8df25f0e4959c381e181af61916f1e6b5f52618f08e3f"} Oct 30 19:54:14.609735 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:14.609655 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vsnwj" event={"ID":"d155893f-7afa-49d3-b3cf-67086d9d3bdd","Type":"ContainerDied","Data":"3a174bb483414eb23a0f09c5ad784c9bee061dbcce5ece490acad3b6d5ff7d04"} Oct 30 19:54:14.609735 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:14.609682 5088 scope.go:122] "RemoveContainer" containerID="ca43b5f9a1849a226ce8df25f0e4959c381e181af61916f1e6b5f52618f08e3f" Oct 30 19:54:14.618025 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:14.618009 5088 scope.go:122] "RemoveContainer" containerID="ca43b5f9a1849a226ce8df25f0e4959c381e181af61916f1e6b5f52618f08e3f" Oct 30 19:54:14.618260 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:54:14.618243 5088 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca43b5f9a1849a226ce8df25f0e4959c381e181af61916f1e6b5f52618f08e3f\": container with ID starting with ca43b5f9a1849a226ce8df25f0e4959c381e181af61916f1e6b5f52618f08e3f not found: ID does not exist" containerID="ca43b5f9a1849a226ce8df25f0e4959c381e181af61916f1e6b5f52618f08e3f" Oct 30 19:54:14.618310 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:14.618268 5088 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca43b5f9a1849a226ce8df25f0e4959c381e181af61916f1e6b5f52618f08e3f"} err="failed to get container status \"ca43b5f9a1849a226ce8df25f0e4959c381e181af61916f1e6b5f52618f08e3f\": rpc error: code = NotFound desc = could not find container \"ca43b5f9a1849a226ce8df25f0e4959c381e181af61916f1e6b5f52618f08e3f\": container with ID starting with ca43b5f9a1849a226ce8df25f0e4959c381e181af61916f1e6b5f52618f08e3f not found: ID does not exist" Oct 30 19:54:14.671519 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:14.671495 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-vsnwj"] Oct 30 19:54:14.676480 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:14.676457 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-vsnwj"] Oct 30 19:54:14.741397 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:54:14.741366 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d155893f-7afa-49d3-b3cf-67086d9d3bdd" path="/var/lib/kubelet/pods/d155893f-7afa-49d3-b3cf-67086d9d3bdd/volumes" Oct 30 19:54:22.436124 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:54:22.436085 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 19:54:52.436861 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:54:52.436822 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 19:55:22.436448 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:55:22.436412 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 19:55:23.196262 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:23.196223 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-ssrhr"] Oct 30 19:55:23.196698 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:23.196677 5088 cpu_manager.go:414] "containerMap: removing container" podUID="d155893f-7afa-49d3-b3cf-67086d9d3bdd" containerName="discovery" Oct 30 19:55:23.196804 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:23.196699 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d155893f-7afa-49d3-b3cf-67086d9d3bdd" containerName="discovery" Oct 30 19:55:23.196867 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:23.196830 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="d155893f-7afa-49d3-b3cf-67086d9d3bdd" containerName="discovery" Oct 30 19:55:23.199902 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:23.199880 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-ssrhr" Oct 30 19:55:23.232283 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:23.232249 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 19:55:23.232283 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:23.232251 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 19:55:23.232749 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:23.232732 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-6r4rf\"" Oct 30 19:55:23.233690 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:23.233672 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-ssrhr"] Oct 30 19:55:23.297413 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:23.297377 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w689\" (UniqueName: \"kubernetes.io/projected/551014b2-4dfe-4df4-9ff1-87a27bdf79ce-kube-api-access-7w689\") pod \"limitador-operator-controller-manager-856dd584c5-ssrhr\" (UID: \"551014b2-4dfe-4df4-9ff1-87a27bdf79ce\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-ssrhr" Oct 30 19:55:23.398214 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:23.398177 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w689\" (UniqueName: \"kubernetes.io/projected/551014b2-4dfe-4df4-9ff1-87a27bdf79ce-kube-api-access-7w689\") pod \"limitador-operator-controller-manager-856dd584c5-ssrhr\" (UID: \"551014b2-4dfe-4df4-9ff1-87a27bdf79ce\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-ssrhr" Oct 30 19:55:23.427396 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:23.427370 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w689\" (UniqueName: \"kubernetes.io/projected/551014b2-4dfe-4df4-9ff1-87a27bdf79ce-kube-api-access-7w689\") pod \"limitador-operator-controller-manager-856dd584c5-ssrhr\" (UID: \"551014b2-4dfe-4df4-9ff1-87a27bdf79ce\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-ssrhr" Oct 30 19:55:23.512203 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:23.512172 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-ssrhr" Oct 30 19:55:23.818671 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:23.818646 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-ssrhr"] Oct 30 19:55:23.821302 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:55:23.821275 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod551014b2_4dfe_4df4_9ff1_87a27bdf79ce.slice/crio-fb4d49072d7d2702ac6bf2a40f29913bdd1988b4042a8e25fc2f63d7bae6d2da WatchSource:0}: Error finding container fb4d49072d7d2702ac6bf2a40f29913bdd1988b4042a8e25fc2f63d7bae6d2da: Status 404 returned error can't find the container with id fb4d49072d7d2702ac6bf2a40f29913bdd1988b4042a8e25fc2f63d7bae6d2da Oct 30 19:55:23.892924 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:23.892892 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-ssrhr" event={"ID":"551014b2-4dfe-4df4-9ff1-87a27bdf79ce","Type":"ContainerStarted","Data":"fb4d49072d7d2702ac6bf2a40f29913bdd1988b4042a8e25fc2f63d7bae6d2da"} Oct 30 19:55:25.471963 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:25.471919 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-fvxph"] Oct 30 19:55:25.480685 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:25.480658 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-fvxph" Oct 30 19:55:25.516753 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:25.516700 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jkfc\" (UniqueName: \"kubernetes.io/projected/335ef769-6b60-4648-bff4-ed5d0d1bbb2e-kube-api-access-6jkfc\") pod \"dns-operator-controller-manager-65bfd79598-fvxph\" (UID: \"335ef769-6b60-4648-bff4-ed5d0d1bbb2e\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-fvxph" Oct 30 19:55:25.517006 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:25.516981 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 19:55:25.517091 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:25.517038 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-tvpks\"" Oct 30 19:55:25.518537 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:25.518513 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-fvxph"] Oct 30 19:55:25.617469 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:25.617428 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jkfc\" (UniqueName: \"kubernetes.io/projected/335ef769-6b60-4648-bff4-ed5d0d1bbb2e-kube-api-access-6jkfc\") pod \"dns-operator-controller-manager-65bfd79598-fvxph\" (UID: \"335ef769-6b60-4648-bff4-ed5d0d1bbb2e\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-fvxph" Oct 30 19:55:25.651606 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:25.651580 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jkfc\" (UniqueName: \"kubernetes.io/projected/335ef769-6b60-4648-bff4-ed5d0d1bbb2e-kube-api-access-6jkfc\") pod \"dns-operator-controller-manager-65bfd79598-fvxph\" (UID: \"335ef769-6b60-4648-bff4-ed5d0d1bbb2e\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-fvxph" Oct 30 19:55:25.795212 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:25.795177 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-fvxph" Oct 30 19:55:26.104543 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:26.104506 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-fvxph"] Oct 30 19:55:26.121635 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:55:26.121585 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod335ef769_6b60_4648_bff4_ed5d0d1bbb2e.slice/crio-9faf116556dddd3b2717e2c346e29156c91f98b86f7ee566d196dbaf773f9248 WatchSource:0}: Error finding container 9faf116556dddd3b2717e2c346e29156c91f98b86f7ee566d196dbaf773f9248: Status 404 returned error can't find the container with id 9faf116556dddd3b2717e2c346e29156c91f98b86f7ee566d196dbaf773f9248 Oct 30 19:55:26.909657 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:26.909605 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-fvxph" event={"ID":"335ef769-6b60-4648-bff4-ed5d0d1bbb2e","Type":"ContainerStarted","Data":"9faf116556dddd3b2717e2c346e29156c91f98b86f7ee566d196dbaf773f9248"} Oct 30 19:55:26.911378 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:26.911345 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-ssrhr" event={"ID":"551014b2-4dfe-4df4-9ff1-87a27bdf79ce","Type":"ContainerStarted","Data":"0320399dde54add15262a7a848d58bf02949e9753ae531d251604eb809342394"} Oct 30 19:55:26.911518 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:26.911499 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-ssrhr" Oct 30 19:55:26.970280 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:26.970213 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-ssrhr" podStartSLOduration=1.65652182 podStartE2EDuration="3.970194225s" totalImagesPullingTime="2.313672405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:26.96774152 +0000 UTC m=+488.380455300" watchObservedRunningTime="2026-10-30 19:55:26.970194225 +0000 UTC m=+488.382908003" Oct 30 19:55:27.917387 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:27.917349 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-fvxph" event={"ID":"335ef769-6b60-4648-bff4-ed5d0d1bbb2e","Type":"ContainerStarted","Data":"5ee2ab357621c23df6c83d13c7c67e47000847e3cdf6e254788aa1cbffe2ec3d"} Oct 30 19:55:27.917880 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:27.917536 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-fvxph" Oct 30 19:55:27.973030 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:27.972979 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-fvxph" podStartSLOduration=1.631420433 podStartE2EDuration="2.972965722s" totalImagesPullingTime="1.341545289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:27.971215335 +0000 UTC m=+489.383929132" watchObservedRunningTime="2026-10-30 19:55:27.972965722 +0000 UTC m=+489.385679496" Oct 30 19:55:29.259356 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:29.259324 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-rq4t2"] Oct 30 19:55:29.263235 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:29.263216 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-rq4t2" Oct 30 19:55:29.296323 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:29.296293 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-gxr5r\"" Oct 30 19:55:29.297830 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:29.297803 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-rq4t2"] Oct 30 19:55:29.351221 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:29.351187 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5s68\" (UniqueName: \"kubernetes.io/projected/5a7ba990-22c5-4549-80a9-3a877255b7dd-kube-api-access-h5s68\") pod \"authorino-operator-68d6dd76dc-rq4t2\" (UID: \"5a7ba990-22c5-4549-80a9-3a877255b7dd\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-rq4t2" Oct 30 19:55:29.452259 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:29.452223 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5s68\" (UniqueName: \"kubernetes.io/projected/5a7ba990-22c5-4549-80a9-3a877255b7dd-kube-api-access-h5s68\") pod \"authorino-operator-68d6dd76dc-rq4t2\" (UID: \"5a7ba990-22c5-4549-80a9-3a877255b7dd\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-rq4t2" Oct 30 19:55:29.480247 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:29.480215 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5s68\" (UniqueName: \"kubernetes.io/projected/5a7ba990-22c5-4549-80a9-3a877255b7dd-kube-api-access-h5s68\") pod \"authorino-operator-68d6dd76dc-rq4t2\" (UID: \"5a7ba990-22c5-4549-80a9-3a877255b7dd\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-rq4t2" Oct 30 19:55:29.576288 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:29.576200 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-rq4t2" Oct 30 19:55:29.879460 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:29.879406 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-rq4t2"] Oct 30 19:55:29.882159 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:55:29.882123 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a7ba990_22c5_4549_80a9_3a877255b7dd.slice/crio-3f41818e9800cfa47b240f193623378045fdda9a1de0844cad2d8d2b9c5dee1a WatchSource:0}: Error finding container 3f41818e9800cfa47b240f193623378045fdda9a1de0844cad2d8d2b9c5dee1a: Status 404 returned error can't find the container with id 3f41818e9800cfa47b240f193623378045fdda9a1de0844cad2d8d2b9c5dee1a Oct 30 19:55:29.926451 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:29.926417 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-rq4t2" event={"ID":"5a7ba990-22c5-4549-80a9-3a877255b7dd","Type":"ContainerStarted","Data":"3f41818e9800cfa47b240f193623378045fdda9a1de0844cad2d8d2b9c5dee1a"} Oct 30 19:55:31.939133 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:31.939090 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-rq4t2" event={"ID":"5a7ba990-22c5-4549-80a9-3a877255b7dd","Type":"ContainerStarted","Data":"a4f286a31163e89080c32546e6a5644cc9d5b7c1c9984f499a93a5d011231646"} Oct 30 19:55:31.939572 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:31.939226 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-rq4t2" Oct 30 19:55:32.001377 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:32.001317 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d6dd76dc-rq4t2" podStartSLOduration=1.591202879 podStartE2EDuration="3.001304289s" totalImagesPullingTime="1.41010141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:31.999902821 +0000 UTC m=+493.412616618" watchObservedRunningTime="2026-10-30 19:55:32.001304289 +0000 UTC m=+493.414018061" Oct 30 19:55:37.920509 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:37.920477 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-ssrhr" Oct 30 19:55:38.924253 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:38.924223 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-fvxph" Oct 30 19:55:42.945768 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:42.945737 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-rq4t2" Oct 30 19:55:52.437191 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:55:52.437151 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 19:55:53.606857 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:53.606816 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-nl84d"] Oct 30 19:55:53.610795 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:53.610768 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-nl84d" Oct 30 19:55:53.630660 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:53.630628 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-nl84d"] Oct 30 19:55:53.641964 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:53.641936 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-5pgqn\"" Oct 30 19:55:53.773043 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:53.773004 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6083dece-de09-4439-9949-300a11dca700-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-nl84d\" (UID: \"6083dece-de09-4439-9949-300a11dca700\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-nl84d" Oct 30 19:55:53.773225 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:53.773064 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvfxj\" (UniqueName: \"kubernetes.io/projected/6083dece-de09-4439-9949-300a11dca700-kube-api-access-jvfxj\") pod \"kuadrant-operator-controller-manager-956956bc4-nl84d\" (UID: \"6083dece-de09-4439-9949-300a11dca700\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-nl84d" Oct 30 19:55:53.874151 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:53.874059 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6083dece-de09-4439-9949-300a11dca700-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-nl84d\" (UID: \"6083dece-de09-4439-9949-300a11dca700\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-nl84d" Oct 30 19:55:53.874151 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:53.874110 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvfxj\" (UniqueName: \"kubernetes.io/projected/6083dece-de09-4439-9949-300a11dca700-kube-api-access-jvfxj\") pod \"kuadrant-operator-controller-manager-956956bc4-nl84d\" (UID: \"6083dece-de09-4439-9949-300a11dca700\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-nl84d" Oct 30 19:55:53.874474 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:53.874454 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6083dece-de09-4439-9949-300a11dca700-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-nl84d\" (UID: \"6083dece-de09-4439-9949-300a11dca700\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-nl84d" Oct 30 19:55:53.900448 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:53.900412 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvfxj\" (UniqueName: \"kubernetes.io/projected/6083dece-de09-4439-9949-300a11dca700-kube-api-access-jvfxj\") pod \"kuadrant-operator-controller-manager-956956bc4-nl84d\" (UID: \"6083dece-de09-4439-9949-300a11dca700\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-nl84d" Oct 30 19:55:53.923324 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:53.923291 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-nl84d" Oct 30 19:55:54.254854 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:54.254826 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-nl84d"] Oct 30 19:55:54.270102 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:55:54.270076 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6083dece_de09_4439_9949_300a11dca700.slice/crio-b0ee7bd6ec34fc0838feec1691ef554165289f4d85790fd528178e4a7d9ff2d3 WatchSource:0}: Error finding container b0ee7bd6ec34fc0838feec1691ef554165289f4d85790fd528178e4a7d9ff2d3: Status 404 returned error can't find the container with id b0ee7bd6ec34fc0838feec1691ef554165289f4d85790fd528178e4a7d9ff2d3 Oct 30 19:55:55.040689 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:55.040644 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-nl84d" event={"ID":"6083dece-de09-4439-9949-300a11dca700","Type":"ContainerStarted","Data":"b0ee7bd6ec34fc0838feec1691ef554165289f4d85790fd528178e4a7d9ff2d3"} Oct 30 19:55:55.440999 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:55.440840 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-nl84d"] Oct 30 19:55:55.440999 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:55.440916 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-nl84d"] Oct 30 19:55:55.462475 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:55.462436 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-ssrhr"] Oct 30 19:55:55.462740 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:55.462711 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-ssrhr" podUID="551014b2-4dfe-4df4-9ff1-87a27bdf79ce" containerName="manager" containerID="cri-o://0320399dde54add15262a7a848d58bf02949e9753ae531d251604eb809342394" gracePeriod=2 Oct 30 19:55:55.473790 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:55.473749 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-ssrhr"] Oct 30 19:55:55.477902 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:55:55.477872 5088 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-ssrhr\" is forbidden: User \"system:node:ip-10-0-132-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-173.ec2.internal' and this object" podUID="551014b2-4dfe-4df4-9ff1-87a27bdf79ce" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-ssrhr" Oct 30 19:55:55.493340 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:55.493305 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-m67x6"] Oct 30 19:55:55.494115 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:55.493865 5088 cpu_manager.go:414] "containerMap: removing container" podUID="551014b2-4dfe-4df4-9ff1-87a27bdf79ce" containerName="manager" Oct 30 19:55:55.494115 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:55.493888 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="551014b2-4dfe-4df4-9ff1-87a27bdf79ce" containerName="manager" Oct 30 19:55:55.494115 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:55.494028 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="551014b2-4dfe-4df4-9ff1-87a27bdf79ce" containerName="manager" Oct 30 19:55:55.498758 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:55.498727 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-m67x6" Oct 30 19:55:55.525957 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:55.525652 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-m67x6"] Oct 30 19:55:55.566671 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:55:55.566602 5088 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-ssrhr\" is forbidden: User \"system:node:ip-10-0-132-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-173.ec2.internal' and this object" podUID="551014b2-4dfe-4df4-9ff1-87a27bdf79ce" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-ssrhr" Oct 30 19:55:55.589151 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:55.589110 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdnrl\" (UniqueName: \"kubernetes.io/projected/99574888-155b-41ad-a913-04105d65b7a0-kube-api-access-pdnrl\") pod \"limitador-operator-controller-manager-856dd584c5-m67x6\" (UID: \"99574888-155b-41ad-a913-04105d65b7a0\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-m67x6" Oct 30 19:55:55.689792 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:55.689761 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdnrl\" (UniqueName: \"kubernetes.io/projected/99574888-155b-41ad-a913-04105d65b7a0-kube-api-access-pdnrl\") pod \"limitador-operator-controller-manager-856dd584c5-m67x6\" (UID: \"99574888-155b-41ad-a913-04105d65b7a0\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-m67x6" Oct 30 19:55:55.719939 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:55.719911 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdnrl\" (UniqueName: \"kubernetes.io/projected/99574888-155b-41ad-a913-04105d65b7a0-kube-api-access-pdnrl\") pod \"limitador-operator-controller-manager-856dd584c5-m67x6\" (UID: \"99574888-155b-41ad-a913-04105d65b7a0\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-m67x6" Oct 30 19:55:55.730486 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:55.730466 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-ssrhr" Oct 30 19:55:55.745374 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:55:55.745343 5088 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-ssrhr\" is forbidden: User \"system:node:ip-10-0-132-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-173.ec2.internal' and this object" podUID="551014b2-4dfe-4df4-9ff1-87a27bdf79ce" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-ssrhr" Oct 30 19:55:55.839662 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:55.839603 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-m67x6" Oct 30 19:55:55.891848 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:55.891811 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/551014b2-4dfe-4df4-9ff1-87a27bdf79ce-kube-api-access-7w689\" (UniqueName: \"kubernetes.io/projected/551014b2-4dfe-4df4-9ff1-87a27bdf79ce-kube-api-access-7w689\") pod \"551014b2-4dfe-4df4-9ff1-87a27bdf79ce\" (UID: \"551014b2-4dfe-4df4-9ff1-87a27bdf79ce\") " Oct 30 19:55:55.894222 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:55.894190 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/551014b2-4dfe-4df4-9ff1-87a27bdf79ce-kube-api-access-7w689" pod "551014b2-4dfe-4df4-9ff1-87a27bdf79ce" (UID: "551014b2-4dfe-4df4-9ff1-87a27bdf79ce"). InnerVolumeSpecName "kube-api-access-7w689". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:55.992524 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:55.992484 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w689\" (UniqueName: \"kubernetes.io/projected/551014b2-4dfe-4df4-9ff1-87a27bdf79ce-kube-api-access-7w689\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 19:55:56.047913 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:56.047872 5088 generic.go:356] "Generic (PLEG): container finished" podID="551014b2-4dfe-4df4-9ff1-87a27bdf79ce" containerID="0320399dde54add15262a7a848d58bf02949e9753ae531d251604eb809342394" exitCode=0 Oct 30 19:55:56.048379 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:56.047943 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-ssrhr" Oct 30 19:55:56.048379 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:56.047953 5088 scope.go:122] "RemoveContainer" containerID="0320399dde54add15262a7a848d58bf02949e9753ae531d251604eb809342394" Oct 30 19:55:56.060835 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:56.060811 5088 scope.go:122] "RemoveContainer" containerID="0320399dde54add15262a7a848d58bf02949e9753ae531d251604eb809342394" Oct 30 19:55:56.061264 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:55:56.061241 5088 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0320399dde54add15262a7a848d58bf02949e9753ae531d251604eb809342394\": container with ID starting with 0320399dde54add15262a7a848d58bf02949e9753ae531d251604eb809342394 not found: ID does not exist" containerID="0320399dde54add15262a7a848d58bf02949e9753ae531d251604eb809342394" Oct 30 19:55:56.061350 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:56.061275 5088 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0320399dde54add15262a7a848d58bf02949e9753ae531d251604eb809342394"} err="failed to get container status \"0320399dde54add15262a7a848d58bf02949e9753ae531d251604eb809342394\": rpc error: code = NotFound desc = could not find container \"0320399dde54add15262a7a848d58bf02949e9753ae531d251604eb809342394\": container with ID starting with 0320399dde54add15262a7a848d58bf02949e9753ae531d251604eb809342394 not found: ID does not exist" Oct 30 19:55:56.063452 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:55:56.063425 5088 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-ssrhr\" is forbidden: User \"system:node:ip-10-0-132-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-173.ec2.internal' and this object" podUID="551014b2-4dfe-4df4-9ff1-87a27bdf79ce" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-ssrhr" Oct 30 19:55:56.078582 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:55:56.078544 5088 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-ssrhr\" is forbidden: User \"system:node:ip-10-0-132-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-173.ec2.internal' and this object" podUID="551014b2-4dfe-4df4-9ff1-87a27bdf79ce" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-ssrhr" Oct 30 19:55:56.178965 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:56.178919 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-m67x6"] Oct 30 19:55:56.182969 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:55:56.182939 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99574888_155b_41ad_a913_04105d65b7a0.slice/crio-3420cc0386f3dfe059c654b65d24b65c083a602a1e92d935c0b3355036845425 WatchSource:0}: Error finding container 3420cc0386f3dfe059c654b65d24b65c083a602a1e92d935c0b3355036845425: Status 404 returned error can't find the container with id 3420cc0386f3dfe059c654b65d24b65c083a602a1e92d935c0b3355036845425 Oct 30 19:55:56.738980 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:56.738951 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="551014b2-4dfe-4df4-9ff1-87a27bdf79ce" path="/var/lib/kubelet/pods/551014b2-4dfe-4df4-9ff1-87a27bdf79ce/volumes" Oct 30 19:55:57.055842 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:57.055759 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-m67x6" event={"ID":"99574888-155b-41ad-a913-04105d65b7a0","Type":"ContainerStarted","Data":"b5fe0a3f40a47a8773ca631269cb6b65331b4535e2e87c7d5a1bf9607350eff2"} Oct 30 19:55:57.055842 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:57.055814 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-m67x6" event={"ID":"99574888-155b-41ad-a913-04105d65b7a0","Type":"ContainerStarted","Data":"3420cc0386f3dfe059c654b65d24b65c083a602a1e92d935c0b3355036845425"} Oct 30 19:55:57.056325 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:57.055856 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-m67x6" Oct 30 19:55:57.107509 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:57.107457 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-m67x6" podStartSLOduration=2.107442758 podStartE2EDuration="2.107442758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:57.107219999 +0000 UTC m=+518.519933788" watchObservedRunningTime="2026-10-30 19:55:57.107442758 +0000 UTC m=+518.520156535" Oct 30 19:55:58.065109 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:58.065066 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-nl84d" podUID="6083dece-de09-4439-9949-300a11dca700" containerName="manager" containerID="cri-o://7cf55acf202bdd27c3320099830df0726b80c8dcc8db6876123ef9f9f80d8322" gracePeriod=2 Oct 30 19:55:58.314483 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:58.314456 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-nl84d" Oct 30 19:55:58.329479 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:55:58.329414 5088 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-956956bc4-nl84d\" is forbidden: User \"system:node:ip-10-0-132-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-173.ec2.internal' and this object" podUID="6083dece-de09-4439-9949-300a11dca700" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-nl84d" Oct 30 19:55:58.414731 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:58.414692 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6083dece-de09-4439-9949-300a11dca700-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6083dece-de09-4439-9949-300a11dca700-extensions-socket-volume\") pod \"6083dece-de09-4439-9949-300a11dca700\" (UID: \"6083dece-de09-4439-9949-300a11dca700\") " Oct 30 19:55:58.414886 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:58.414752 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6083dece-de09-4439-9949-300a11dca700-kube-api-access-jvfxj\" (UniqueName: \"kubernetes.io/projected/6083dece-de09-4439-9949-300a11dca700-kube-api-access-jvfxj\") pod \"6083dece-de09-4439-9949-300a11dca700\" (UID: \"6083dece-de09-4439-9949-300a11dca700\") " Oct 30 19:55:58.414986 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:58.414962 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6083dece-de09-4439-9949-300a11dca700-extensions-socket-volume" pod "6083dece-de09-4439-9949-300a11dca700" (UID: "6083dece-de09-4439-9949-300a11dca700"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:58.415071 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:58.415054 5088 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6083dece-de09-4439-9949-300a11dca700-extensions-socket-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 19:55:58.416806 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:58.416786 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6083dece-de09-4439-9949-300a11dca700-kube-api-access-jvfxj" pod "6083dece-de09-4439-9949-300a11dca700" (UID: "6083dece-de09-4439-9949-300a11dca700"). InnerVolumeSpecName "kube-api-access-jvfxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:58.516325 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:58.516272 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvfxj\" (UniqueName: \"kubernetes.io/projected/6083dece-de09-4439-9949-300a11dca700-kube-api-access-jvfxj\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 19:55:58.737453 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:58.737424 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6083dece-de09-4439-9949-300a11dca700" path="/var/lib/kubelet/pods/6083dece-de09-4439-9949-300a11dca700/volumes" Oct 30 19:55:58.752755 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:55:58.752726 5088 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-956956bc4-nl84d\" is forbidden: User \"system:node:ip-10-0-132-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-173.ec2.internal' and this object" podUID="6083dece-de09-4439-9949-300a11dca700" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-nl84d" Oct 30 19:55:59.070306 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:59.070215 5088 generic.go:356] "Generic (PLEG): container finished" podID="6083dece-de09-4439-9949-300a11dca700" containerID="7cf55acf202bdd27c3320099830df0726b80c8dcc8db6876123ef9f9f80d8322" exitCode=2 Oct 30 19:55:59.070306 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:59.070271 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-nl84d" Oct 30 19:55:59.070814 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:59.070310 5088 scope.go:122] "RemoveContainer" containerID="7cf55acf202bdd27c3320099830df0726b80c8dcc8db6876123ef9f9f80d8322" Oct 30 19:55:59.078550 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:59.078531 5088 scope.go:122] "RemoveContainer" containerID="7cf55acf202bdd27c3320099830df0726b80c8dcc8db6876123ef9f9f80d8322" Oct 30 19:55:59.078864 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:55:59.078843 5088 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cf55acf202bdd27c3320099830df0726b80c8dcc8db6876123ef9f9f80d8322\": container with ID starting with 7cf55acf202bdd27c3320099830df0726b80c8dcc8db6876123ef9f9f80d8322 not found: ID does not exist" containerID="7cf55acf202bdd27c3320099830df0726b80c8dcc8db6876123ef9f9f80d8322" Oct 30 19:55:59.078916 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:55:59.078872 5088 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cf55acf202bdd27c3320099830df0726b80c8dcc8db6876123ef9f9f80d8322"} err="failed to get container status \"7cf55acf202bdd27c3320099830df0726b80c8dcc8db6876123ef9f9f80d8322\": rpc error: code = NotFound desc = could not find container \"7cf55acf202bdd27c3320099830df0726b80c8dcc8db6876123ef9f9f80d8322\": container with ID starting with 7cf55acf202bdd27c3320099830df0726b80c8dcc8db6876123ef9f9f80d8322 not found: ID does not exist" Oct 30 19:55:59.087520 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:55:59.087488 5088 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-956956bc4-nl84d\" is forbidden: User \"system:node:ip-10-0-132-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-173.ec2.internal' and this object" podUID="6083dece-de09-4439-9949-300a11dca700" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-nl84d" Oct 30 19:56:08.067697 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:08.067668 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-m67x6" Oct 30 19:56:22.436966 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:56:22.436929 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 19:56:38.084837 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:38.084800 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk"] Oct 30 19:56:38.085597 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:38.085574 5088 cpu_manager.go:414] "containerMap: removing container" podUID="6083dece-de09-4439-9949-300a11dca700" containerName="manager" Oct 30 19:56:38.085688 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:38.085601 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6083dece-de09-4439-9949-300a11dca700" containerName="manager" Oct 30 19:56:38.085727 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:38.085711 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="6083dece-de09-4439-9949-300a11dca700" containerName="manager" Oct 30 19:56:38.089242 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:38.089220 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" Oct 30 19:56:38.109839 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:38.109817 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-wbdhc\"" Oct 30 19:56:38.121350 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:38.121322 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk"] Oct 30 19:56:38.165673 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:38.165635 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b460a5d-51f8-4694-8dd6-316f788bf8fc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bfxfk\" (UID: \"4b460a5d-51f8-4694-8dd6-316f788bf8fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" Oct 30 19:56:38.165673 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:38.165675 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b460a5d-51f8-4694-8dd6-316f788bf8fc-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bfxfk\" (UID: \"4b460a5d-51f8-4694-8dd6-316f788bf8fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" Oct 30 19:56:38.165893 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:38.165801 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b460a5d-51f8-4694-8dd6-316f788bf8fc-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bfxfk\" (UID: \"4b460a5d-51f8-4694-8dd6-316f788bf8fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" Oct 30 19:56:38.165893 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:38.165848 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b460a5d-51f8-4694-8dd6-316f788bf8fc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bfxfk\" (UID: \"4b460a5d-51f8-4694-8dd6-316f788bf8fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" Oct 30 19:56:38.165969 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:38.165914 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b460a5d-51f8-4694-8dd6-316f788bf8fc-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bfxfk\" (UID: \"4b460a5d-51f8-4694-8dd6-316f788bf8fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" Oct 30 19:56:38.165969 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:38.165938 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85lvh\" (UniqueName: \"kubernetes.io/projected/4b460a5d-51f8-4694-8dd6-316f788bf8fc-kube-api-access-85lvh\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bfxfk\" (UID: \"4b460a5d-51f8-4694-8dd6-316f788bf8fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" Oct 30 19:56:38.166050 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:38.166035 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b460a5d-51f8-4694-8dd6-316f788bf8fc-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bfxfk\" (UID: \"4b460a5d-51f8-4694-8dd6-316f788bf8fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" Oct 30 19:56:38.166101 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:38.166058 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b460a5d-51f8-4694-8dd6-316f788bf8fc-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bfxfk\" (UID: \"4b460a5d-51f8-4694-8dd6-316f788bf8fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" Oct 30 19:56:38.166160 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:38.166117 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b460a5d-51f8-4694-8dd6-316f788bf8fc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bfxfk\" (UID: \"4b460a5d-51f8-4694-8dd6-316f788bf8fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" Oct 30 19:56:38.267112 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:38.267072 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b460a5d-51f8-4694-8dd6-316f788bf8fc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bfxfk\" (UID: \"4b460a5d-51f8-4694-8dd6-316f788bf8fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" Oct 30 19:56:38.267112 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:38.267114 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b460a5d-51f8-4694-8dd6-316f788bf8fc-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bfxfk\" (UID: \"4b460a5d-51f8-4694-8dd6-316f788bf8fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" Oct 30 19:56:38.267341 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:38.267146 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b460a5d-51f8-4694-8dd6-316f788bf8fc-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bfxfk\" (UID: \"4b460a5d-51f8-4694-8dd6-316f788bf8fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" Oct 30 19:56:38.267341 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:38.267168 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b460a5d-51f8-4694-8dd6-316f788bf8fc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bfxfk\" (UID: \"4b460a5d-51f8-4694-8dd6-316f788bf8fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" Oct 30 19:56:38.267341 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:38.267189 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b460a5d-51f8-4694-8dd6-316f788bf8fc-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bfxfk\" (UID: \"4b460a5d-51f8-4694-8dd6-316f788bf8fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" Oct 30 19:56:38.267341 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:38.267205 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85lvh\" (UniqueName: \"kubernetes.io/projected/4b460a5d-51f8-4694-8dd6-316f788bf8fc-kube-api-access-85lvh\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bfxfk\" (UID: \"4b460a5d-51f8-4694-8dd6-316f788bf8fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" Oct 30 19:56:38.267548 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:38.267353 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b460a5d-51f8-4694-8dd6-316f788bf8fc-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bfxfk\" (UID: \"4b460a5d-51f8-4694-8dd6-316f788bf8fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" Oct 30 19:56:38.267548 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:38.267390 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b460a5d-51f8-4694-8dd6-316f788bf8fc-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bfxfk\" (UID: \"4b460a5d-51f8-4694-8dd6-316f788bf8fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" Oct 30 19:56:38.267548 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:38.267413 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b460a5d-51f8-4694-8dd6-316f788bf8fc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bfxfk\" (UID: \"4b460a5d-51f8-4694-8dd6-316f788bf8fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" Oct 30 19:56:38.267726 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:38.267540 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b460a5d-51f8-4694-8dd6-316f788bf8fc-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bfxfk\" (UID: \"4b460a5d-51f8-4694-8dd6-316f788bf8fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" Oct 30 19:56:38.267726 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:38.267685 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b460a5d-51f8-4694-8dd6-316f788bf8fc-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bfxfk\" (UID: \"4b460a5d-51f8-4694-8dd6-316f788bf8fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" Oct 30 19:56:38.267800 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:38.267718 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b460a5d-51f8-4694-8dd6-316f788bf8fc-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bfxfk\" (UID: \"4b460a5d-51f8-4694-8dd6-316f788bf8fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" Oct 30 19:56:38.267906 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:38.267887 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b460a5d-51f8-4694-8dd6-316f788bf8fc-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bfxfk\" (UID: \"4b460a5d-51f8-4694-8dd6-316f788bf8fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" Oct 30 19:56:38.267942 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:38.267909 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b460a5d-51f8-4694-8dd6-316f788bf8fc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bfxfk\" (UID: \"4b460a5d-51f8-4694-8dd6-316f788bf8fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" Oct 30 19:56:38.269523 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:38.269500 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b460a5d-51f8-4694-8dd6-316f788bf8fc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bfxfk\" (UID: \"4b460a5d-51f8-4694-8dd6-316f788bf8fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" Oct 30 19:56:38.269889 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:38.269870 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b460a5d-51f8-4694-8dd6-316f788bf8fc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bfxfk\" (UID: \"4b460a5d-51f8-4694-8dd6-316f788bf8fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" Oct 30 19:56:38.293288 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:38.293261 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b460a5d-51f8-4694-8dd6-316f788bf8fc-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bfxfk\" (UID: \"4b460a5d-51f8-4694-8dd6-316f788bf8fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" Oct 30 19:56:38.293456 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:38.293435 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85lvh\" (UniqueName: \"kubernetes.io/projected/4b460a5d-51f8-4694-8dd6-316f788bf8fc-kube-api-access-85lvh\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bfxfk\" (UID: \"4b460a5d-51f8-4694-8dd6-316f788bf8fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" Oct 30 19:56:38.403445 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:38.403343 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" Oct 30 19:56:38.722657 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:38.722509 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk"] Oct 30 19:56:38.725375 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:56:38.725351 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b460a5d_51f8_4694_8dd6_316f788bf8fc.slice/crio-ba77fba8a238f34bd2d0c69f3ceea4d2c41899e092e873741e15d1357e37d5a7 WatchSource:0}: Error finding container ba77fba8a238f34bd2d0c69f3ceea4d2c41899e092e873741e15d1357e37d5a7: Status 404 returned error can't find the container with id ba77fba8a238f34bd2d0c69f3ceea4d2c41899e092e873741e15d1357e37d5a7 Oct 30 19:56:38.727336 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:38.727307 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:56:38.727384 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:38.727371 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:56:38.727417 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:38.727400 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:56:39.247580 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:39.247546 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" event={"ID":"4b460a5d-51f8-4694-8dd6-316f788bf8fc","Type":"ContainerStarted","Data":"7c6da6c315d4fbe78e6e0aafcd52c3208967b18c66b774c500a33cc3e1385857"} Oct 30 19:56:39.248030 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:39.247599 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" event={"ID":"4b460a5d-51f8-4694-8dd6-316f788bf8fc","Type":"ContainerStarted","Data":"ba77fba8a238f34bd2d0c69f3ceea4d2c41899e092e873741e15d1357e37d5a7"} Oct 30 19:56:39.311673 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:39.311602 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" podStartSLOduration=1.311589082 podStartE2EDuration="1.311589082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:39.310902284 +0000 UTC m=+560.723616061" watchObservedRunningTime="2026-10-30 19:56:39.311589082 +0000 UTC m=+560.724302859" Oct 30 19:56:39.404314 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:39.404281 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" Oct 30 19:56:39.409084 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:39.409056 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" Oct 30 19:56:40.252056 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:40.252021 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" Oct 30 19:56:40.252994 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:40.252972 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" Oct 30 19:56:44.540796 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:44.540762 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 19:56:44.544467 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:44.544448 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-pdkrz" Oct 30 19:56:44.563248 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:44.563220 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 19:56:44.575370 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:44.575345 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-ssn2n\"" Oct 30 19:56:44.577783 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:44.577751 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 19:56:44.622520 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:44.622482 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a2a6ff3a-e8bf-4256-b860-89065d5116f7-config-file\") pod \"limitador-limitador-7c9b87d574-pdkrz\" (UID: \"a2a6ff3a-e8bf-4256-b860-89065d5116f7\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-pdkrz" Oct 30 19:56:44.622706 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:44.622606 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlcdn\" (UniqueName: \"kubernetes.io/projected/a2a6ff3a-e8bf-4256-b860-89065d5116f7-kube-api-access-qlcdn\") pod \"limitador-limitador-7c9b87d574-pdkrz\" (UID: \"a2a6ff3a-e8bf-4256-b860-89065d5116f7\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-pdkrz" Oct 30 19:56:44.644906 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:44.644868 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 19:56:44.723932 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:44.723890 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a2a6ff3a-e8bf-4256-b860-89065d5116f7-config-file\") pod \"limitador-limitador-7c9b87d574-pdkrz\" (UID: \"a2a6ff3a-e8bf-4256-b860-89065d5116f7\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-pdkrz" Oct 30 19:56:44.724128 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:44.724004 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlcdn\" (UniqueName: \"kubernetes.io/projected/a2a6ff3a-e8bf-4256-b860-89065d5116f7-kube-api-access-qlcdn\") pod \"limitador-limitador-7c9b87d574-pdkrz\" (UID: \"a2a6ff3a-e8bf-4256-b860-89065d5116f7\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-pdkrz" Oct 30 19:56:44.724597 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:44.724569 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a2a6ff3a-e8bf-4256-b860-89065d5116f7-config-file\") pod \"limitador-limitador-7c9b87d574-pdkrz\" (UID: \"a2a6ff3a-e8bf-4256-b860-89065d5116f7\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-pdkrz" Oct 30 19:56:44.752409 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:44.752375 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlcdn\" (UniqueName: \"kubernetes.io/projected/a2a6ff3a-e8bf-4256-b860-89065d5116f7-kube-api-access-qlcdn\") pod \"limitador-limitador-7c9b87d574-pdkrz\" (UID: \"a2a6ff3a-e8bf-4256-b860-89065d5116f7\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-pdkrz" Oct 30 19:56:44.856897 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:44.856799 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-pdkrz" Oct 30 19:56:45.156992 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:45.156885 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 19:56:45.159215 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:56:45.159180 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2a6ff3a_e8bf_4256_b860_89065d5116f7.slice/crio-6fc51574ef03ae2ef63dea259d0113508843f9fc1ebeba22daf9e4cdd69ac865 WatchSource:0}: Error finding container 6fc51574ef03ae2ef63dea259d0113508843f9fc1ebeba22daf9e4cdd69ac865: Status 404 returned error can't find the container with id 6fc51574ef03ae2ef63dea259d0113508843f9fc1ebeba22daf9e4cdd69ac865 Oct 30 19:56:45.274937 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:45.274899 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-pdkrz" event={"ID":"a2a6ff3a-e8bf-4256-b860-89065d5116f7","Type":"ContainerStarted","Data":"6fc51574ef03ae2ef63dea259d0113508843f9fc1ebeba22daf9e4cdd69ac865"} Oct 30 19:56:49.294231 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:49.294188 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-pdkrz" event={"ID":"a2a6ff3a-e8bf-4256-b860-89065d5116f7","Type":"ContainerStarted","Data":"f54fc6ad101b5f47f09518f8c2af64e75e42c6cd4149a518b9abc3ebd7f986a9"} Oct 30 19:56:49.294605 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:49.294317 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-pdkrz" Oct 30 19:56:49.339911 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:56:49.339865 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-pdkrz" podStartSLOduration=1.983070847 podStartE2EDuration="5.339852333s" totalImagesPullingTime="3.356781486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:49.339375233 +0000 UTC m=+570.752089031" watchObservedRunningTime="2026-10-30 19:56:49.339852333 +0000 UTC m=+570.752566110" Oct 30 19:56:52.436894 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:56:52.436790 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 19:57:00.299678 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:57:00.299641 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-pdkrz" Oct 30 19:57:18.778131 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:57:18.778102 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qdj4h_da850825-59ab-480f-9047-afd7d2bdc19f/console-operator/1.log" Oct 30 19:57:18.778585 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:57:18.778565 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qdj4h_da850825-59ab-480f-9047-afd7d2bdc19f/console-operator/1.log" Oct 30 19:57:18.782879 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:57:18.782853 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xpjpn_688d079a-932d-4f63-97da-5cd6222a03e1/ovn-acl-logging/0.log" Oct 30 19:57:18.783792 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:57:18.783768 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xpjpn_688d079a-932d-4f63-97da-5cd6222a03e1/ovn-acl-logging/0.log" Oct 30 19:57:22.435820 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:57:22.435772 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 19:57:52.437373 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:57:52.437334 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 19:58:05.455882 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:58:05.455843 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 19:58:15.674717 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:58:15.674680 5088 generic.go:356] "Generic (PLEG): container finished" podID="4b460a5d-51f8-4694-8dd6-316f788bf8fc" containerID="7c6da6c315d4fbe78e6e0aafcd52c3208967b18c66b774c500a33cc3e1385857" exitCode=0 Oct 30 19:58:15.675138 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:58:15.674754 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" event={"ID":"4b460a5d-51f8-4694-8dd6-316f788bf8fc","Type":"ContainerDied","Data":"7c6da6c315d4fbe78e6e0aafcd52c3208967b18c66b774c500a33cc3e1385857"} Oct 30 19:58:15.675138 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:58:15.675092 5088 scope.go:122] "RemoveContainer" containerID="7c6da6c315d4fbe78e6e0aafcd52c3208967b18c66b774c500a33cc3e1385857" Oct 30 19:58:15.676103 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:58:15.676080 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:58:15.676151 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:58:15.676142 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:58:15.676185 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:58:15.676177 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:58:16.680890 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:58:16.680854 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" event={"ID":"4b460a5d-51f8-4694-8dd6-316f788bf8fc","Type":"ContainerStarted","Data":"ff36f4b4c2d32f61ab6fcb408b8ba3df106255924be41311fa85f232ee372896"} Oct 30 19:58:17.406603 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:58:17.405970 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" Oct 30 19:58:17.406603 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:58:17.406092 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-bfxfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 30 19:58:17.406603 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:58:17.406137 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" podUID="4b460a5d-51f8-4694-8dd6-316f788bf8fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 30 19:58:18.404140 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:58:18.404103 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-bfxfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 30 19:58:18.404773 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:58:18.404742 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" podUID="4b460a5d-51f8-4694-8dd6-316f788bf8fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 30 19:58:19.405637 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:58:19.405019 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-bfxfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 30 19:58:19.405637 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:58:19.405091 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" podUID="4b460a5d-51f8-4694-8dd6-316f788bf8fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 30 19:58:20.404060 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:58:20.404021 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-bfxfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 30 19:58:20.404390 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:58:20.404348 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" podUID="4b460a5d-51f8-4694-8dd6-316f788bf8fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 30 19:58:21.409034 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:58:21.408991 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-bfxfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 30 19:58:21.409448 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:58:21.409057 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" podUID="4b460a5d-51f8-4694-8dd6-316f788bf8fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 30 19:58:22.407713 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:58:22.407684 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" Oct 30 19:58:22.408097 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:58:22.408074 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" Oct 30 19:58:22.408704 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:58:22.408686 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bfxfk" Oct 30 19:58:22.435967 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:58:22.435938 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 19:58:52.437179 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:58:52.437098 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 19:59:04.083863 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:04.083830 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 19:59:07.394969 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:07.394933 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2"] Oct 30 19:59:07.398514 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:07.398490 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 19:59:07.416660 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:07.416638 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 19:59:07.418370 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:07.418351 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xlfkq\"" Oct 30 19:59:07.430364 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:07.430339 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 19:59:07.430490 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:07.430440 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 19:59:07.433771 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:07.433747 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2"] Oct 30 19:59:07.535844 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:07.535811 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee73a969-6d25-490c-b943-2c9771cfa738-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2\" (UID: \"ee73a969-6d25-490c-b943-2c9771cfa738\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 19:59:07.535844 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:07.535846 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee73a969-6d25-490c-b943-2c9771cfa738-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2\" (UID: \"ee73a969-6d25-490c-b943-2c9771cfa738\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 19:59:07.536065 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:07.535874 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee73a969-6d25-490c-b943-2c9771cfa738-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2\" (UID: \"ee73a969-6d25-490c-b943-2c9771cfa738\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 19:59:07.536065 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:07.536011 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee73a969-6d25-490c-b943-2c9771cfa738-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2\" (UID: \"ee73a969-6d25-490c-b943-2c9771cfa738\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 19:59:07.536173 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:07.536076 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee73a969-6d25-490c-b943-2c9771cfa738-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2\" (UID: \"ee73a969-6d25-490c-b943-2c9771cfa738\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 19:59:07.536173 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:07.536136 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjsl7\" (UniqueName: \"kubernetes.io/projected/ee73a969-6d25-490c-b943-2c9771cfa738-kube-api-access-fjsl7\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2\" (UID: \"ee73a969-6d25-490c-b943-2c9771cfa738\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 19:59:07.636564 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:07.636525 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee73a969-6d25-490c-b943-2c9771cfa738-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2\" (UID: \"ee73a969-6d25-490c-b943-2c9771cfa738\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 19:59:07.636564 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:07.636569 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjsl7\" (UniqueName: \"kubernetes.io/projected/ee73a969-6d25-490c-b943-2c9771cfa738-kube-api-access-fjsl7\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2\" (UID: \"ee73a969-6d25-490c-b943-2c9771cfa738\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 19:59:07.636848 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:07.636605 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee73a969-6d25-490c-b943-2c9771cfa738-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2\" (UID: \"ee73a969-6d25-490c-b943-2c9771cfa738\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 19:59:07.636848 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:07.636648 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee73a969-6d25-490c-b943-2c9771cfa738-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2\" (UID: \"ee73a969-6d25-490c-b943-2c9771cfa738\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 19:59:07.636848 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:07.636687 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee73a969-6d25-490c-b943-2c9771cfa738-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2\" (UID: \"ee73a969-6d25-490c-b943-2c9771cfa738\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 19:59:07.637011 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:07.636869 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee73a969-6d25-490c-b943-2c9771cfa738-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2\" (UID: \"ee73a969-6d25-490c-b943-2c9771cfa738\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 19:59:07.637084 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:07.637036 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee73a969-6d25-490c-b943-2c9771cfa738-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2\" (UID: \"ee73a969-6d25-490c-b943-2c9771cfa738\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 19:59:07.637141 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:07.637086 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee73a969-6d25-490c-b943-2c9771cfa738-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2\" (UID: \"ee73a969-6d25-490c-b943-2c9771cfa738\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 19:59:07.637278 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:07.637256 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee73a969-6d25-490c-b943-2c9771cfa738-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2\" (UID: \"ee73a969-6d25-490c-b943-2c9771cfa738\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 19:59:07.638992 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:07.638970 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee73a969-6d25-490c-b943-2c9771cfa738-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2\" (UID: \"ee73a969-6d25-490c-b943-2c9771cfa738\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 19:59:07.639169 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:07.639152 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee73a969-6d25-490c-b943-2c9771cfa738-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2\" (UID: \"ee73a969-6d25-490c-b943-2c9771cfa738\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 19:59:07.663563 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:07.663499 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjsl7\" (UniqueName: \"kubernetes.io/projected/ee73a969-6d25-490c-b943-2c9771cfa738-kube-api-access-fjsl7\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2\" (UID: \"ee73a969-6d25-490c-b943-2c9771cfa738\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 19:59:07.729435 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:07.729408 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 19:59:08.031949 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:08.031808 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2"] Oct 30 19:59:08.034753 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:59:08.034724 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee73a969_6d25_490c_b943_2c9771cfa738.slice/crio-19361440ce4450041237ed18d38db4cd29ca2e0fdcb8781021084be1b9d75b2f WatchSource:0}: Error finding container 19361440ce4450041237ed18d38db4cd29ca2e0fdcb8781021084be1b9d75b2f: Status 404 returned error can't find the container with id 19361440ce4450041237ed18d38db4cd29ca2e0fdcb8781021084be1b9d75b2f Oct 30 19:59:08.036146 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:08.036130 5088 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:59:08.779435 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:08.779407 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 19:59:08.929458 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:08.929421 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" event={"ID":"ee73a969-6d25-490c-b943-2c9771cfa738","Type":"ContainerStarted","Data":"19361440ce4450041237ed18d38db4cd29ca2e0fdcb8781021084be1b9d75b2f"} Oct 30 19:59:09.439661 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:09.439271 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9"] Oct 30 19:59:09.444050 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:09.444027 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9" Oct 30 19:59:09.462500 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:09.462475 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9"] Oct 30 19:59:09.474487 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:09.474464 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 30 19:59:09.555343 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:09.555314 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fa4e381-a575-4bf9-a65d-33bdc3093587-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9\" (UID: \"1fa4e381-a575-4bf9-a65d-33bdc3093587\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9" Oct 30 19:59:09.555515 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:09.555353 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fa4e381-a575-4bf9-a65d-33bdc3093587-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9\" (UID: \"1fa4e381-a575-4bf9-a65d-33bdc3093587\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9" Oct 30 19:59:09.555515 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:09.555374 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fa4e381-a575-4bf9-a65d-33bdc3093587-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9\" (UID: \"1fa4e381-a575-4bf9-a65d-33bdc3093587\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9" Oct 30 19:59:09.555646 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:09.555533 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fa4e381-a575-4bf9-a65d-33bdc3093587-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9\" (UID: \"1fa4e381-a575-4bf9-a65d-33bdc3093587\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9" Oct 30 19:59:09.555646 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:09.555584 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fa4e381-a575-4bf9-a65d-33bdc3093587-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9\" (UID: \"1fa4e381-a575-4bf9-a65d-33bdc3093587\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9" Oct 30 19:59:09.555721 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:09.555696 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5wsv\" (UniqueName: \"kubernetes.io/projected/1fa4e381-a575-4bf9-a65d-33bdc3093587-kube-api-access-p5wsv\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9\" (UID: \"1fa4e381-a575-4bf9-a65d-33bdc3093587\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9" Oct 30 19:59:09.656629 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:09.656584 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fa4e381-a575-4bf9-a65d-33bdc3093587-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9\" (UID: \"1fa4e381-a575-4bf9-a65d-33bdc3093587\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9" Oct 30 19:59:09.656837 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:09.656666 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5wsv\" (UniqueName: \"kubernetes.io/projected/1fa4e381-a575-4bf9-a65d-33bdc3093587-kube-api-access-p5wsv\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9\" (UID: \"1fa4e381-a575-4bf9-a65d-33bdc3093587\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9" Oct 30 19:59:09.656837 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:09.656706 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fa4e381-a575-4bf9-a65d-33bdc3093587-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9\" (UID: \"1fa4e381-a575-4bf9-a65d-33bdc3093587\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9" Oct 30 19:59:09.656837 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:09.656728 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fa4e381-a575-4bf9-a65d-33bdc3093587-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9\" (UID: \"1fa4e381-a575-4bf9-a65d-33bdc3093587\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9" Oct 30 19:59:09.656837 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:09.656746 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fa4e381-a575-4bf9-a65d-33bdc3093587-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9\" (UID: \"1fa4e381-a575-4bf9-a65d-33bdc3093587\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9" Oct 30 19:59:09.657072 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:09.656854 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fa4e381-a575-4bf9-a65d-33bdc3093587-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9\" (UID: \"1fa4e381-a575-4bf9-a65d-33bdc3093587\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9" Oct 30 19:59:09.657144 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:09.657082 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fa4e381-a575-4bf9-a65d-33bdc3093587-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9\" (UID: \"1fa4e381-a575-4bf9-a65d-33bdc3093587\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9" Oct 30 19:59:09.657144 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:09.657107 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fa4e381-a575-4bf9-a65d-33bdc3093587-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9\" (UID: \"1fa4e381-a575-4bf9-a65d-33bdc3093587\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9" Oct 30 19:59:09.657267 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:09.657245 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fa4e381-a575-4bf9-a65d-33bdc3093587-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9\" (UID: \"1fa4e381-a575-4bf9-a65d-33bdc3093587\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9" Oct 30 19:59:09.658904 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:09.658883 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fa4e381-a575-4bf9-a65d-33bdc3093587-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9\" (UID: \"1fa4e381-a575-4bf9-a65d-33bdc3093587\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9" Oct 30 19:59:09.659150 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:09.659132 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fa4e381-a575-4bf9-a65d-33bdc3093587-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9\" (UID: \"1fa4e381-a575-4bf9-a65d-33bdc3093587\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9" Oct 30 19:59:09.683447 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:09.683417 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5wsv\" (UniqueName: \"kubernetes.io/projected/1fa4e381-a575-4bf9-a65d-33bdc3093587-kube-api-access-p5wsv\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9\" (UID: \"1fa4e381-a575-4bf9-a65d-33bdc3093587\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9" Oct 30 19:59:09.761184 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:09.761155 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9" Oct 30 19:59:09.936980 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:09.936943 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" event={"ID":"ee73a969-6d25-490c-b943-2c9771cfa738","Type":"ContainerStarted","Data":"a07b94c2e8378d9579fb9792c5ecc4e72666ef5b6302621c5c9d905e2ce1c124"} Oct 30 19:59:10.010771 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:10.010720 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podStartSLOduration=1.9722504760000001 podStartE2EDuration="3.010704299s" totalImagesPullingTime="1.038453823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:10.009009042 +0000 UTC m=+711.421722841" watchObservedRunningTime="2026-10-30 19:59:10.010704299 +0000 UTC m=+711.423418079" Oct 30 19:59:10.105408 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:10.105385 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9"] Oct 30 19:59:10.107517 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:59:10.107491 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fa4e381_a575_4bf9_a65d_33bdc3093587.slice/crio-b3b0f45e7769b129bb53e50546fe217e929954500cb7e296e07a1c77d1fe6257 WatchSource:0}: Error finding container b3b0f45e7769b129bb53e50546fe217e929954500cb7e296e07a1c77d1fe6257: Status 404 returned error can't find the container with id b3b0f45e7769b129bb53e50546fe217e929954500cb7e296e07a1c77d1fe6257 Oct 30 19:59:10.730079 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:10.730045 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 19:59:10.943350 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:10.943316 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9" event={"ID":"1fa4e381-a575-4bf9-a65d-33bdc3093587","Type":"ContainerStarted","Data":"962decab064cee149617bd16b2aef6e45bbb0146eb8a51743cc54e7f4ba3124b"} Oct 30 19:59:10.943748 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:10.943372 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9" event={"ID":"1fa4e381-a575-4bf9-a65d-33bdc3093587","Type":"ContainerStarted","Data":"b3b0f45e7769b129bb53e50546fe217e929954500cb7e296e07a1c77d1fe6257"} Oct 30 19:59:10.991173 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:10.991127 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9" podStartSLOduration=1.806774205 podStartE2EDuration="1.991114925s" totalImagesPullingTime="184.34072ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:10.990000803 +0000 UTC m=+712.402714582" watchObservedRunningTime="2026-10-30 19:59:10.991114925 +0000 UTC m=+712.403828704" Oct 30 19:59:11.762099 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:11.762059 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9" Oct 30 19:59:14.289180 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:14.289145 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-wtlm7"] Oct 30 19:59:14.298061 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:14.298037 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wtlm7" Oct 30 19:59:14.331929 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:14.331907 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 30 19:59:14.334477 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:14.334450 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-wtlm7"] Oct 30 19:59:14.395842 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:14.395815 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/419ffdb0-c223-4426-872a-50e3073812e4-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wtlm7\" (UID: \"419ffdb0-c223-4426-872a-50e3073812e4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wtlm7" Oct 30 19:59:14.395972 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:14.395872 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/419ffdb0-c223-4426-872a-50e3073812e4-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wtlm7\" (UID: \"419ffdb0-c223-4426-872a-50e3073812e4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wtlm7" Oct 30 19:59:14.395972 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:14.395966 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8mbr\" (UniqueName: \"kubernetes.io/projected/419ffdb0-c223-4426-872a-50e3073812e4-kube-api-access-l8mbr\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wtlm7\" (UID: \"419ffdb0-c223-4426-872a-50e3073812e4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wtlm7" Oct 30 19:59:14.396082 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:14.395988 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/419ffdb0-c223-4426-872a-50e3073812e4-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wtlm7\" (UID: \"419ffdb0-c223-4426-872a-50e3073812e4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wtlm7" Oct 30 19:59:14.396082 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:14.396003 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/419ffdb0-c223-4426-872a-50e3073812e4-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wtlm7\" (UID: \"419ffdb0-c223-4426-872a-50e3073812e4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wtlm7" Oct 30 19:59:14.396082 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:14.396024 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/419ffdb0-c223-4426-872a-50e3073812e4-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wtlm7\" (UID: \"419ffdb0-c223-4426-872a-50e3073812e4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wtlm7" Oct 30 19:59:14.497328 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:14.497295 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/419ffdb0-c223-4426-872a-50e3073812e4-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wtlm7\" (UID: \"419ffdb0-c223-4426-872a-50e3073812e4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wtlm7" Oct 30 19:59:14.497507 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:14.497350 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8mbr\" (UniqueName: \"kubernetes.io/projected/419ffdb0-c223-4426-872a-50e3073812e4-kube-api-access-l8mbr\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wtlm7\" (UID: \"419ffdb0-c223-4426-872a-50e3073812e4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wtlm7" Oct 30 19:59:14.497507 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:14.497462 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/419ffdb0-c223-4426-872a-50e3073812e4-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wtlm7\" (UID: \"419ffdb0-c223-4426-872a-50e3073812e4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wtlm7" Oct 30 19:59:14.497507 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:14.497497 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/419ffdb0-c223-4426-872a-50e3073812e4-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wtlm7\" (UID: \"419ffdb0-c223-4426-872a-50e3073812e4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wtlm7" Oct 30 19:59:14.497507 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:14.497523 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/419ffdb0-c223-4426-872a-50e3073812e4-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wtlm7\" (UID: \"419ffdb0-c223-4426-872a-50e3073812e4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wtlm7" Oct 30 19:59:14.497852 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:14.497568 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/419ffdb0-c223-4426-872a-50e3073812e4-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wtlm7\" (UID: \"419ffdb0-c223-4426-872a-50e3073812e4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wtlm7" Oct 30 19:59:14.497852 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:14.497791 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/419ffdb0-c223-4426-872a-50e3073812e4-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wtlm7\" (UID: \"419ffdb0-c223-4426-872a-50e3073812e4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wtlm7" Oct 30 19:59:14.497958 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:14.497881 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/419ffdb0-c223-4426-872a-50e3073812e4-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wtlm7\" (UID: \"419ffdb0-c223-4426-872a-50e3073812e4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wtlm7" Oct 30 19:59:14.497998 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:14.497984 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/419ffdb0-c223-4426-872a-50e3073812e4-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wtlm7\" (UID: \"419ffdb0-c223-4426-872a-50e3073812e4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wtlm7" Oct 30 19:59:14.499793 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:14.499767 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/419ffdb0-c223-4426-872a-50e3073812e4-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wtlm7\" (UID: \"419ffdb0-c223-4426-872a-50e3073812e4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wtlm7" Oct 30 19:59:14.500047 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:14.500031 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/419ffdb0-c223-4426-872a-50e3073812e4-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wtlm7\" (UID: \"419ffdb0-c223-4426-872a-50e3073812e4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wtlm7" Oct 30 19:59:14.532859 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:14.532837 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8mbr\" (UniqueName: \"kubernetes.io/projected/419ffdb0-c223-4426-872a-50e3073812e4-kube-api-access-l8mbr\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wtlm7\" (UID: \"419ffdb0-c223-4426-872a-50e3073812e4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wtlm7" Oct 30 19:59:14.570254 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:14.570199 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 19:59:14.611789 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:14.611765 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wtlm7" Oct 30 19:59:14.912763 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:14.912733 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-wtlm7"] Oct 30 19:59:14.914926 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:59:14.914899 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod419ffdb0_c223_4426_872a_50e3073812e4.slice/crio-d9c346388019a14dc45ada63264a8c271679b685ccd29d937fea288b3188e093 WatchSource:0}: Error finding container d9c346388019a14dc45ada63264a8c271679b685ccd29d937fea288b3188e093: Status 404 returned error can't find the container with id d9c346388019a14dc45ada63264a8c271679b685ccd29d937fea288b3188e093 Oct 30 19:59:14.964565 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:14.964512 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wtlm7" event={"ID":"419ffdb0-c223-4426-872a-50e3073812e4","Type":"ContainerStarted","Data":"d9c346388019a14dc45ada63264a8c271679b685ccd29d937fea288b3188e093"} Oct 30 19:59:15.970869 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:15.970826 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wtlm7" event={"ID":"419ffdb0-c223-4426-872a-50e3073812e4","Type":"ContainerStarted","Data":"e84f2ad8807c6f224f40f0e9477daf48ae474a102eb5737564d2c531eb20c081"} Oct 30 19:59:16.020495 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:16.020442 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wtlm7" podStartSLOduration=1.83342958 podStartE2EDuration="2.020429806s" totalImagesPullingTime="187.000226ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:16.01886317 +0000 UTC m=+717.431576949" watchObservedRunningTime="2026-10-30 19:59:16.020429806 +0000 UTC m=+717.433143584" Oct 30 19:59:16.612865 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:16.612834 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wtlm7" Oct 30 19:59:17.535664 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:17.535628 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-f4snn"] Oct 30 19:59:17.540077 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:17.540051 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f4snn" Oct 30 19:59:17.559177 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:17.559156 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 19:59:17.576790 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:17.576766 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-f4snn"] Oct 30 19:59:17.623929 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:17.623907 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ec64166-d2fc-47f6-96d1-48ea0ee1b70c-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f4snn\" (UID: \"5ec64166-d2fc-47f6-96d1-48ea0ee1b70c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f4snn" Oct 30 19:59:17.624051 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:17.623935 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ec64166-d2fc-47f6-96d1-48ea0ee1b70c-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f4snn\" (UID: \"5ec64166-d2fc-47f6-96d1-48ea0ee1b70c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f4snn" Oct 30 19:59:17.624051 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:17.623961 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ec64166-d2fc-47f6-96d1-48ea0ee1b70c-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f4snn\" (UID: \"5ec64166-d2fc-47f6-96d1-48ea0ee1b70c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f4snn" Oct 30 19:59:17.624051 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:17.624032 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rd6k\" (UniqueName: \"kubernetes.io/projected/5ec64166-d2fc-47f6-96d1-48ea0ee1b70c-kube-api-access-5rd6k\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f4snn\" (UID: \"5ec64166-d2fc-47f6-96d1-48ea0ee1b70c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f4snn" Oct 30 19:59:17.624169 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:17.624108 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ec64166-d2fc-47f6-96d1-48ea0ee1b70c-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f4snn\" (UID: \"5ec64166-d2fc-47f6-96d1-48ea0ee1b70c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f4snn" Oct 30 19:59:17.624169 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:17.624161 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ec64166-d2fc-47f6-96d1-48ea0ee1b70c-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f4snn\" (UID: \"5ec64166-d2fc-47f6-96d1-48ea0ee1b70c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f4snn" Oct 30 19:59:17.725027 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:17.724993 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ec64166-d2fc-47f6-96d1-48ea0ee1b70c-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f4snn\" (UID: \"5ec64166-d2fc-47f6-96d1-48ea0ee1b70c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f4snn" Oct 30 19:59:17.725027 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:17.725028 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ec64166-d2fc-47f6-96d1-48ea0ee1b70c-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f4snn\" (UID: \"5ec64166-d2fc-47f6-96d1-48ea0ee1b70c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f4snn" Oct 30 19:59:17.725261 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:17.725065 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ec64166-d2fc-47f6-96d1-48ea0ee1b70c-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f4snn\" (UID: \"5ec64166-d2fc-47f6-96d1-48ea0ee1b70c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f4snn" Oct 30 19:59:17.725261 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:17.725090 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rd6k\" (UniqueName: \"kubernetes.io/projected/5ec64166-d2fc-47f6-96d1-48ea0ee1b70c-kube-api-access-5rd6k\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f4snn\" (UID: \"5ec64166-d2fc-47f6-96d1-48ea0ee1b70c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f4snn" Oct 30 19:59:17.725261 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:17.725111 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ec64166-d2fc-47f6-96d1-48ea0ee1b70c-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f4snn\" (UID: \"5ec64166-d2fc-47f6-96d1-48ea0ee1b70c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f4snn" Oct 30 19:59:17.725261 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:17.725157 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ec64166-d2fc-47f6-96d1-48ea0ee1b70c-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f4snn\" (UID: \"5ec64166-d2fc-47f6-96d1-48ea0ee1b70c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f4snn" Oct 30 19:59:17.725462 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:17.725430 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ec64166-d2fc-47f6-96d1-48ea0ee1b70c-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f4snn\" (UID: \"5ec64166-d2fc-47f6-96d1-48ea0ee1b70c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f4snn" Oct 30 19:59:17.725549 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:17.725527 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ec64166-d2fc-47f6-96d1-48ea0ee1b70c-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f4snn\" (UID: \"5ec64166-d2fc-47f6-96d1-48ea0ee1b70c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f4snn" Oct 30 19:59:17.725649 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:17.725604 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ec64166-d2fc-47f6-96d1-48ea0ee1b70c-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f4snn\" (UID: \"5ec64166-d2fc-47f6-96d1-48ea0ee1b70c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f4snn" Oct 30 19:59:17.727345 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:17.727314 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ec64166-d2fc-47f6-96d1-48ea0ee1b70c-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f4snn\" (UID: \"5ec64166-d2fc-47f6-96d1-48ea0ee1b70c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f4snn" Oct 30 19:59:17.727662 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:17.727599 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ec64166-d2fc-47f6-96d1-48ea0ee1b70c-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f4snn\" (UID: \"5ec64166-d2fc-47f6-96d1-48ea0ee1b70c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f4snn" Oct 30 19:59:17.730424 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:17.730404 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 19:59:17.743212 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:17.743191 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 19:59:17.751684 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:17.751663 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rd6k\" (UniqueName: \"kubernetes.io/projected/5ec64166-d2fc-47f6-96d1-48ea0ee1b70c-kube-api-access-5rd6k\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f4snn\" (UID: \"5ec64166-d2fc-47f6-96d1-48ea0ee1b70c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f4snn" Oct 30 19:59:17.854623 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:17.854538 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f4snn" Oct 30 19:59:17.990788 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:17.990759 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 19:59:18.168118 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:18.168090 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-f4snn"] Oct 30 19:59:18.170066 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:59:18.170039 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ec64166_d2fc_47f6_96d1_48ea0ee1b70c.slice/crio-a7681b6dd5ca13781e8375e60c701768612e28ae1e6f4dd7dcae91129d2db74b WatchSource:0}: Error finding container a7681b6dd5ca13781e8375e60c701768612e28ae1e6f4dd7dcae91129d2db74b: Status 404 returned error can't find the container with id a7681b6dd5ca13781e8375e60c701768612e28ae1e6f4dd7dcae91129d2db74b Oct 30 19:59:18.985012 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:18.984958 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f4snn" event={"ID":"5ec64166-d2fc-47f6-96d1-48ea0ee1b70c","Type":"ContainerStarted","Data":"64e6ce0410c9888d00fe7af2771da8b1893d684ac7a466bdaec81335122e5429"} Oct 30 19:59:18.985493 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:18.985029 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f4snn" event={"ID":"5ec64166-d2fc-47f6-96d1-48ea0ee1b70c","Type":"ContainerStarted","Data":"a7681b6dd5ca13781e8375e60c701768612e28ae1e6f4dd7dcae91129d2db74b"} Oct 30 19:59:19.034719 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:19.034659 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f4snn" podStartSLOduration=1.8453243860000001 podStartE2EDuration="2.034641234s" totalImagesPullingTime="189.316848ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:19.032479386 +0000 UTC m=+720.445193178" watchObservedRunningTime="2026-10-30 19:59:19.034641234 +0000 UTC m=+720.447355013" Oct 30 19:59:19.762021 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:19.761987 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9" Oct 30 19:59:19.774521 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:19.774493 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9" Oct 30 19:59:19.854935 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:19.854907 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f4snn" Oct 30 19:59:20.001173 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:20.001148 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9" Oct 30 19:59:20.586850 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:20.586822 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 19:59:22.437553 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:59:22.437516 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 19:59:24.323647 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:24.323600 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9"] Oct 30 19:59:24.328428 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:24.328409 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" Oct 30 19:59:24.361185 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:24.361160 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9"] Oct 30 19:59:24.380971 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:24.380941 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce0a1543-79bc-440d-ad18-c045ac745786-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-c2tb9\" (UID: \"ce0a1543-79bc-440d-ad18-c045ac745786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" Oct 30 19:59:24.381107 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:24.380985 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce0a1543-79bc-440d-ad18-c045ac745786-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-c2tb9\" (UID: \"ce0a1543-79bc-440d-ad18-c045ac745786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" Oct 30 19:59:24.381107 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:24.381082 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce0a1543-79bc-440d-ad18-c045ac745786-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-c2tb9\" (UID: \"ce0a1543-79bc-440d-ad18-c045ac745786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" Oct 30 19:59:24.381209 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:24.381117 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce0a1543-79bc-440d-ad18-c045ac745786-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-c2tb9\" (UID: \"ce0a1543-79bc-440d-ad18-c045ac745786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" Oct 30 19:59:24.381251 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:24.381203 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce0a1543-79bc-440d-ad18-c045ac745786-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-c2tb9\" (UID: \"ce0a1543-79bc-440d-ad18-c045ac745786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" Oct 30 19:59:24.381251 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:24.381240 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g99qh\" (UniqueName: \"kubernetes.io/projected/ce0a1543-79bc-440d-ad18-c045ac745786-kube-api-access-g99qh\") pod \"maas-default-gateway-openshift-default-7f5667f86c-c2tb9\" (UID: \"ce0a1543-79bc-440d-ad18-c045ac745786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" Oct 30 19:59:24.381341 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:24.381276 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce0a1543-79bc-440d-ad18-c045ac745786-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-c2tb9\" (UID: \"ce0a1543-79bc-440d-ad18-c045ac745786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" Oct 30 19:59:24.381341 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:24.381323 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce0a1543-79bc-440d-ad18-c045ac745786-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-c2tb9\" (UID: \"ce0a1543-79bc-440d-ad18-c045ac745786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" Oct 30 19:59:24.381459 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:24.381446 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce0a1543-79bc-440d-ad18-c045ac745786-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-c2tb9\" (UID: \"ce0a1543-79bc-440d-ad18-c045ac745786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" Oct 30 19:59:24.482351 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:24.482296 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce0a1543-79bc-440d-ad18-c045ac745786-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-c2tb9\" (UID: \"ce0a1543-79bc-440d-ad18-c045ac745786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" Oct 30 19:59:24.482487 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:24.482352 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce0a1543-79bc-440d-ad18-c045ac745786-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-c2tb9\" (UID: \"ce0a1543-79bc-440d-ad18-c045ac745786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" Oct 30 19:59:24.482487 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:24.482388 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce0a1543-79bc-440d-ad18-c045ac745786-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-c2tb9\" (UID: \"ce0a1543-79bc-440d-ad18-c045ac745786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" Oct 30 19:59:24.482487 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:24.482426 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce0a1543-79bc-440d-ad18-c045ac745786-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-c2tb9\" (UID: \"ce0a1543-79bc-440d-ad18-c045ac745786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" Oct 30 19:59:24.482487 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:24.482450 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce0a1543-79bc-440d-ad18-c045ac745786-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-c2tb9\" (UID: \"ce0a1543-79bc-440d-ad18-c045ac745786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" Oct 30 19:59:24.482740 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:24.482489 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce0a1543-79bc-440d-ad18-c045ac745786-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-c2tb9\" (UID: \"ce0a1543-79bc-440d-ad18-c045ac745786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" Oct 30 19:59:24.482740 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:24.482533 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g99qh\" (UniqueName: \"kubernetes.io/projected/ce0a1543-79bc-440d-ad18-c045ac745786-kube-api-access-g99qh\") pod \"maas-default-gateway-openshift-default-7f5667f86c-c2tb9\" (UID: \"ce0a1543-79bc-440d-ad18-c045ac745786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" Oct 30 19:59:24.482740 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:24.482586 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce0a1543-79bc-440d-ad18-c045ac745786-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-c2tb9\" (UID: \"ce0a1543-79bc-440d-ad18-c045ac745786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" Oct 30 19:59:24.482740 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:24.482650 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce0a1543-79bc-440d-ad18-c045ac745786-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-c2tb9\" (UID: \"ce0a1543-79bc-440d-ad18-c045ac745786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" Oct 30 19:59:24.482909 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:24.482860 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce0a1543-79bc-440d-ad18-c045ac745786-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-c2tb9\" (UID: \"ce0a1543-79bc-440d-ad18-c045ac745786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" Oct 30 19:59:24.482909 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:24.482893 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce0a1543-79bc-440d-ad18-c045ac745786-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-c2tb9\" (UID: \"ce0a1543-79bc-440d-ad18-c045ac745786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" Oct 30 19:59:24.483110 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:24.483086 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce0a1543-79bc-440d-ad18-c045ac745786-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-c2tb9\" (UID: \"ce0a1543-79bc-440d-ad18-c045ac745786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" Oct 30 19:59:24.483235 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:24.483139 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce0a1543-79bc-440d-ad18-c045ac745786-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-c2tb9\" (UID: \"ce0a1543-79bc-440d-ad18-c045ac745786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" Oct 30 19:59:24.483478 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:24.483453 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce0a1543-79bc-440d-ad18-c045ac745786-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-c2tb9\" (UID: \"ce0a1543-79bc-440d-ad18-c045ac745786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" Oct 30 19:59:24.484784 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:24.484766 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce0a1543-79bc-440d-ad18-c045ac745786-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-c2tb9\" (UID: \"ce0a1543-79bc-440d-ad18-c045ac745786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" Oct 30 19:59:24.484980 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:24.484966 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce0a1543-79bc-440d-ad18-c045ac745786-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-c2tb9\" (UID: \"ce0a1543-79bc-440d-ad18-c045ac745786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" Oct 30 19:59:24.508860 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:24.508829 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce0a1543-79bc-440d-ad18-c045ac745786-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-c2tb9\" (UID: \"ce0a1543-79bc-440d-ad18-c045ac745786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" Oct 30 19:59:24.508860 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:24.508844 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g99qh\" (UniqueName: \"kubernetes.io/projected/ce0a1543-79bc-440d-ad18-c045ac745786-kube-api-access-g99qh\") pod \"maas-default-gateway-openshift-default-7f5667f86c-c2tb9\" (UID: \"ce0a1543-79bc-440d-ad18-c045ac745786\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" Oct 30 19:59:24.611964 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:24.611891 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wtlm7" Oct 30 19:59:24.625125 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:24.625102 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wtlm7" Oct 30 19:59:24.641582 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:24.641541 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" Oct 30 19:59:24.962014 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:24.961981 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9"] Oct 30 19:59:24.963861 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:59:24.963832 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce0a1543_79bc_440d_ad18_c045ac745786.slice/crio-b39cda757011ae93c86ea2dc4112cbf5f4c94f97105dece3a0cd513cb000a6c1 WatchSource:0}: Error finding container b39cda757011ae93c86ea2dc4112cbf5f4c94f97105dece3a0cd513cb000a6c1: Status 404 returned error can't find the container with id b39cda757011ae93c86ea2dc4112cbf5f4c94f97105dece3a0cd513cb000a6c1 Oct 30 19:59:24.965977 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:24.965949 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:59:24.966030 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:24.966011 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:59:24.966064 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:24.966038 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:59:25.018195 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:25.018168 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" event={"ID":"ce0a1543-79bc-440d-ad18-c045ac745786","Type":"ContainerStarted","Data":"b39cda757011ae93c86ea2dc4112cbf5f4c94f97105dece3a0cd513cb000a6c1"} Oct 30 19:59:25.029891 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:25.029782 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wtlm7" Oct 30 19:59:26.025209 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:26.025177 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" event={"ID":"ce0a1543-79bc-440d-ad18-c045ac745786","Type":"ContainerStarted","Data":"7df42333ece3dc05bd12c6571f204791532ed04817f98383dcd2cdb84ce00c44"} Oct 30 19:59:26.111877 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:26.111806 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" podStartSLOduration=2.111784336 podStartE2EDuration="2.111784336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:26.110432154 +0000 UTC m=+727.523145934" watchObservedRunningTime="2026-10-30 19:59:26.111784336 +0000 UTC m=+727.524498119" Oct 30 19:59:26.641889 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:26.641855 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" Oct 30 19:59:26.643499 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:26.643468 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-c2tb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 30 19:59:26.643646 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:26.643516 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" podUID="ce0a1543-79bc-440d-ad18-c045ac745786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 30 19:59:27.642649 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:27.642595 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-c2tb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 30 19:59:27.643028 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:27.642667 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" podUID="ce0a1543-79bc-440d-ad18-c045ac745786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 30 19:59:27.855052 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:27.855021 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f4snn" Oct 30 19:59:27.867553 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:27.867524 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f4snn" Oct 30 19:59:28.045550 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:28.045519 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f4snn" Oct 30 19:59:28.642771 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:28.642730 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-c2tb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 30 19:59:28.643161 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:28.642797 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" podUID="ce0a1543-79bc-440d-ad18-c045ac745786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 30 19:59:29.642112 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:29.642078 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-c2tb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 30 19:59:29.642285 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:29.642130 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" podUID="ce0a1543-79bc-440d-ad18-c045ac745786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 30 19:59:30.642585 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:30.642553 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-c2tb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 30 19:59:30.642972 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:30.642607 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" podUID="ce0a1543-79bc-440d-ad18-c045ac745786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 30 19:59:31.642761 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:31.642720 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-c2tb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 30 19:59:31.643272 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:31.642785 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" podUID="ce0a1543-79bc-440d-ad18-c045ac745786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 30 19:59:33.280428 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:33.280388 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" Oct 30 19:59:33.280951 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:33.280928 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" Oct 30 19:59:33.282018 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:33.281993 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" Oct 30 19:59:39.398834 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:39.398803 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s"] Oct 30 19:59:39.405348 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:39.405324 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" Oct 30 19:59:39.434104 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:39.434075 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s"] Oct 30 19:59:39.548740 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:39.548699 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00264202-1595-4616-8796-2ecd7415c577-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-59p8s\" (UID: \"00264202-1595-4616-8796-2ecd7415c577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" Oct 30 19:59:39.548930 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:39.548749 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn6rq\" (UniqueName: \"kubernetes.io/projected/00264202-1595-4616-8796-2ecd7415c577-kube-api-access-gn6rq\") pod \"maas-default-gateway-openshift-default-7f5667f86c-59p8s\" (UID: \"00264202-1595-4616-8796-2ecd7415c577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" Oct 30 19:59:39.548930 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:39.548862 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00264202-1595-4616-8796-2ecd7415c577-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-59p8s\" (UID: \"00264202-1595-4616-8796-2ecd7415c577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" Oct 30 19:59:39.549059 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:39.548977 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00264202-1595-4616-8796-2ecd7415c577-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-59p8s\" (UID: \"00264202-1595-4616-8796-2ecd7415c577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" Oct 30 19:59:39.549059 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:39.549038 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00264202-1595-4616-8796-2ecd7415c577-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-59p8s\" (UID: \"00264202-1595-4616-8796-2ecd7415c577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" Oct 30 19:59:39.549150 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:39.549100 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00264202-1595-4616-8796-2ecd7415c577-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-59p8s\" (UID: \"00264202-1595-4616-8796-2ecd7415c577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" Oct 30 19:59:39.549193 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:39.549158 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00264202-1595-4616-8796-2ecd7415c577-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-59p8s\" (UID: \"00264202-1595-4616-8796-2ecd7415c577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" Oct 30 19:59:39.549193 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:39.549184 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00264202-1595-4616-8796-2ecd7415c577-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-59p8s\" (UID: \"00264202-1595-4616-8796-2ecd7415c577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" Oct 30 19:59:39.549284 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:39.549200 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00264202-1595-4616-8796-2ecd7415c577-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-59p8s\" (UID: \"00264202-1595-4616-8796-2ecd7415c577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" Oct 30 19:59:39.649813 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:39.649719 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00264202-1595-4616-8796-2ecd7415c577-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-59p8s\" (UID: \"00264202-1595-4616-8796-2ecd7415c577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" Oct 30 19:59:39.649813 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:39.649760 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00264202-1595-4616-8796-2ecd7415c577-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-59p8s\" (UID: \"00264202-1595-4616-8796-2ecd7415c577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" Oct 30 19:59:39.649813 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:39.649792 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00264202-1595-4616-8796-2ecd7415c577-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-59p8s\" (UID: \"00264202-1595-4616-8796-2ecd7415c577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" Oct 30 19:59:39.650099 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:39.649814 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00264202-1595-4616-8796-2ecd7415c577-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-59p8s\" (UID: \"00264202-1595-4616-8796-2ecd7415c577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" Oct 30 19:59:39.650099 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:39.649840 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00264202-1595-4616-8796-2ecd7415c577-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-59p8s\" (UID: \"00264202-1595-4616-8796-2ecd7415c577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" Oct 30 19:59:39.650099 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:39.649884 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00264202-1595-4616-8796-2ecd7415c577-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-59p8s\" (UID: \"00264202-1595-4616-8796-2ecd7415c577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" Oct 30 19:59:39.650099 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:39.649913 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn6rq\" (UniqueName: \"kubernetes.io/projected/00264202-1595-4616-8796-2ecd7415c577-kube-api-access-gn6rq\") pod \"maas-default-gateway-openshift-default-7f5667f86c-59p8s\" (UID: \"00264202-1595-4616-8796-2ecd7415c577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" Oct 30 19:59:39.650099 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:39.649953 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00264202-1595-4616-8796-2ecd7415c577-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-59p8s\" (UID: \"00264202-1595-4616-8796-2ecd7415c577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" Oct 30 19:59:39.650099 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:39.650031 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00264202-1595-4616-8796-2ecd7415c577-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-59p8s\" (UID: \"00264202-1595-4616-8796-2ecd7415c577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" Oct 30 19:59:39.650418 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:39.650189 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00264202-1595-4616-8796-2ecd7415c577-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-59p8s\" (UID: \"00264202-1595-4616-8796-2ecd7415c577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" Oct 30 19:59:39.650418 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:39.650229 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00264202-1595-4616-8796-2ecd7415c577-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-59p8s\" (UID: \"00264202-1595-4616-8796-2ecd7415c577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" Oct 30 19:59:39.650515 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:39.650488 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00264202-1595-4616-8796-2ecd7415c577-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-59p8s\" (UID: \"00264202-1595-4616-8796-2ecd7415c577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" Oct 30 19:59:39.650549 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:39.650507 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00264202-1595-4616-8796-2ecd7415c577-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-59p8s\" (UID: \"00264202-1595-4616-8796-2ecd7415c577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" Oct 30 19:59:39.650729 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:39.650705 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00264202-1595-4616-8796-2ecd7415c577-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-59p8s\" (UID: \"00264202-1595-4616-8796-2ecd7415c577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" Oct 30 19:59:39.652288 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:39.652260 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00264202-1595-4616-8796-2ecd7415c577-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-59p8s\" (UID: \"00264202-1595-4616-8796-2ecd7415c577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" Oct 30 19:59:39.652468 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:39.652447 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00264202-1595-4616-8796-2ecd7415c577-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-59p8s\" (UID: \"00264202-1595-4616-8796-2ecd7415c577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" Oct 30 19:59:39.676347 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:39.676303 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00264202-1595-4616-8796-2ecd7415c577-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-59p8s\" (UID: \"00264202-1595-4616-8796-2ecd7415c577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" Oct 30 19:59:39.676487 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:39.676359 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn6rq\" (UniqueName: \"kubernetes.io/projected/00264202-1595-4616-8796-2ecd7415c577-kube-api-access-gn6rq\") pod \"maas-default-gateway-openshift-default-7f5667f86c-59p8s\" (UID: \"00264202-1595-4616-8796-2ecd7415c577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" Oct 30 19:59:39.722486 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:39.722449 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" Oct 30 19:59:40.047943 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:40.047917 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s"] Oct 30 19:59:40.051698 ip-10-0-132-173 kubenswrapper[5088]: W1030 19:59:40.051669 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00264202_1595_4616_8796_2ecd7415c577.slice/crio-42eac92ea231f22e879fdbca88921a9ec3d6af992c0704e4cdb7ee6ccd57c3af WatchSource:0}: Error finding container 42eac92ea231f22e879fdbca88921a9ec3d6af992c0704e4cdb7ee6ccd57c3af: Status 404 returned error can't find the container with id 42eac92ea231f22e879fdbca88921a9ec3d6af992c0704e4cdb7ee6ccd57c3af Oct 30 19:59:40.053098 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:40.053065 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:59:40.053184 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:40.053124 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:59:40.053184 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:40.053151 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:59:40.093100 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:40.093074 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" event={"ID":"00264202-1595-4616-8796-2ecd7415c577","Type":"ContainerStarted","Data":"42eac92ea231f22e879fdbca88921a9ec3d6af992c0704e4cdb7ee6ccd57c3af"} Oct 30 19:59:41.114266 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:41.114219 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" event={"ID":"00264202-1595-4616-8796-2ecd7415c577","Type":"ContainerStarted","Data":"5e32dd83a9a502d3feb8fbe6ae10467c0acbc30fa91cd27137549b4abbe249df"} Oct 30 19:59:41.210594 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:41.210533 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" podStartSLOduration=2.210518012 podStartE2EDuration="2.210518012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:41.179083658 +0000 UTC m=+742.591797431" watchObservedRunningTime="2026-10-30 19:59:41.210518012 +0000 UTC m=+742.623231791" Oct 30 19:59:41.723441 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:41.723405 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" Oct 30 19:59:41.724988 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:41.724967 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-59p8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 19:59:41.725059 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:41.725012 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" podUID="00264202-1595-4616-8796-2ecd7415c577" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 19:59:42.723807 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:42.723768 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-59p8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 19:59:42.724221 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:42.723827 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" podUID="00264202-1595-4616-8796-2ecd7415c577" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 19:59:43.723167 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:43.723132 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-59p8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 19:59:43.723329 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:43.723187 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" podUID="00264202-1595-4616-8796-2ecd7415c577" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 19:59:44.723685 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:44.723653 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-59p8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 19:59:44.724092 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:44.723704 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" podUID="00264202-1595-4616-8796-2ecd7415c577" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 19:59:45.723493 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:45.723457 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-59p8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 19:59:45.723687 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:45.723516 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" podUID="00264202-1595-4616-8796-2ecd7415c577" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 19:59:46.723572 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:46.723530 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-59p8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 19:59:46.724008 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:46.723599 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" podUID="00264202-1595-4616-8796-2ecd7415c577" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 19:59:48.290936 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:48.290907 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" Oct 30 19:59:48.292718 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:48.291524 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" Oct 30 19:59:48.293031 ip-10-0-132-173 kubenswrapper[5088]: I1030 19:59:48.293015 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" Oct 30 19:59:52.437593 ip-10-0-132-173 kubenswrapper[5088]: E1030 19:59:52.437545 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:00:00.170403 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:00.170370 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-r548n"] Oct 30 20:00:00.175531 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:00.175512 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-r548n" Oct 30 20:00:00.193459 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:00.193435 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:00:00.193548 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:00.193521 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:00:00.206381 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:00.206353 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:00:00.208947 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:00.208921 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-r548n"] Oct 30 20:00:00.268206 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:00.268175 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lj2l\" (UniqueName: \"kubernetes.io/projected/d8d37bd4-1c8c-4ecb-a173-954916afdc34-kube-api-access-9lj2l\") pod \"maas-api-key-cleanup-29889840-r548n\" (UID: \"d8d37bd4-1c8c-4ecb-a173-954916afdc34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-r548n" Oct 30 20:00:00.268384 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:00.268224 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8d37bd4-1c8c-4ecb-a173-954916afdc34-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889840-r548n\" (UID: \"d8d37bd4-1c8c-4ecb-a173-954916afdc34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-r548n" Oct 30 20:00:00.369504 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:00.369472 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8d37bd4-1c8c-4ecb-a173-954916afdc34-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889840-r548n\" (UID: \"d8d37bd4-1c8c-4ecb-a173-954916afdc34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-r548n" Oct 30 20:00:00.369712 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:00.369566 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lj2l\" (UniqueName: \"kubernetes.io/projected/d8d37bd4-1c8c-4ecb-a173-954916afdc34-kube-api-access-9lj2l\") pod \"maas-api-key-cleanup-29889840-r548n\" (UID: \"d8d37bd4-1c8c-4ecb-a173-954916afdc34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-r548n" Oct 30 20:00:00.370211 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:00.370187 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8d37bd4-1c8c-4ecb-a173-954916afdc34-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889840-r548n\" (UID: \"d8d37bd4-1c8c-4ecb-a173-954916afdc34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-r548n" Oct 30 20:00:00.395555 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:00.395525 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lj2l\" (UniqueName: \"kubernetes.io/projected/d8d37bd4-1c8c-4ecb-a173-954916afdc34-kube-api-access-9lj2l\") pod \"maas-api-key-cleanup-29889840-r548n\" (UID: \"d8d37bd4-1c8c-4ecb-a173-954916afdc34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-r548n" Oct 30 20:00:00.488905 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:00.488877 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-r548n" Oct 30 20:00:00.790665 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:00.790640 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-r548n"] Oct 30 20:00:00.792638 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:00:00.792597 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8d37bd4_1c8c_4ecb_a173_954916afdc34.slice/crio-a9b1aae30f272233e2cf904df5a870ffc91533dfb1c272bf73786f30ed33dceb WatchSource:0}: Error finding container a9b1aae30f272233e2cf904df5a870ffc91533dfb1c272bf73786f30ed33dceb: Status 404 returned error can't find the container with id a9b1aae30f272233e2cf904df5a870ffc91533dfb1c272bf73786f30ed33dceb Oct 30 20:00:01.214839 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:01.214735 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-r548n" event={"ID":"d8d37bd4-1c8c-4ecb-a173-954916afdc34","Type":"ContainerStarted","Data":"a9b1aae30f272233e2cf904df5a870ffc91533dfb1c272bf73786f30ed33dceb"} Oct 30 20:00:04.232186 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:04.232152 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-r548n" event={"ID":"d8d37bd4-1c8c-4ecb-a173-954916afdc34","Type":"ContainerStarted","Data":"8d5a5a7aa773cd0196235f3aa316d2ca4bafa4de9b5de93e7e41e7947c083d36"} Oct 30 20:00:04.278576 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:04.278518 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-r548n" podStartSLOduration=1.2081064 podStartE2EDuration="4.278504813s" totalImagesPullingTime="3.070398413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:04.27644188 +0000 UTC m=+765.689155669" watchObservedRunningTime="2026-10-30 20:00:04.278504813 +0000 UTC m=+765.691218593" Oct 30 20:00:05.238068 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:05.238021 5088 generic.go:356] "Generic (PLEG): container finished" podID="d8d37bd4-1c8c-4ecb-a173-954916afdc34" containerID="8d5a5a7aa773cd0196235f3aa316d2ca4bafa4de9b5de93e7e41e7947c083d36" exitCode=60 Oct 30 20:00:05.238489 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:05.238074 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-r548n" event={"ID":"d8d37bd4-1c8c-4ecb-a173-954916afdc34","Type":"ContainerDied","Data":"8d5a5a7aa773cd0196235f3aa316d2ca4bafa4de9b5de93e7e41e7947c083d36"} Oct 30 20:00:05.238489 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:05.238330 5088 scope.go:122] "RemoveContainer" containerID="8d5a5a7aa773cd0196235f3aa316d2ca4bafa4de9b5de93e7e41e7947c083d36" Oct 30 20:00:06.245277 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:06.245191 5088 generic.go:356] "Generic (PLEG): container finished" podID="d8d37bd4-1c8c-4ecb-a173-954916afdc34" containerID="d39408d04fffb604f1d8b3659a6fa7b9d2573b538989dc0281d6872648a2359b" exitCode=60 Oct 30 20:00:06.245742 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:06.245270 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-r548n" event={"ID":"d8d37bd4-1c8c-4ecb-a173-954916afdc34","Type":"ContainerDied","Data":"d39408d04fffb604f1d8b3659a6fa7b9d2573b538989dc0281d6872648a2359b"} Oct 30 20:00:06.245742 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:06.245321 5088 scope.go:122] "RemoveContainer" containerID="8d5a5a7aa773cd0196235f3aa316d2ca4bafa4de9b5de93e7e41e7947c083d36" Oct 30 20:00:06.245742 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:06.245598 5088 scope.go:122] "RemoveContainer" containerID="d39408d04fffb604f1d8b3659a6fa7b9d2573b538989dc0281d6872648a2359b" Oct 30 20:00:06.245911 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:00:06.245888 5088 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889840-r548n_odh-ai-gateway-infra(d8d37bd4-1c8c-4ecb-a173-954916afdc34)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-r548n" podUID="d8d37bd4-1c8c-4ecb-a173-954916afdc34" Oct 30 20:00:07.251388 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:07.251348 5088 scope.go:122] "RemoveContainer" containerID="d39408d04fffb604f1d8b3659a6fa7b9d2573b538989dc0281d6872648a2359b" Oct 30 20:00:07.251801 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:00:07.251557 5088 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889840-r548n_odh-ai-gateway-infra(d8d37bd4-1c8c-4ecb-a173-954916afdc34)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-r548n" podUID="d8d37bd4-1c8c-4ecb-a173-954916afdc34" Oct 30 20:00:18.735119 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:18.735090 5088 scope.go:122] "RemoveContainer" containerID="d39408d04fffb604f1d8b3659a6fa7b9d2573b538989dc0281d6872648a2359b" Oct 30 20:00:19.309024 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:19.308945 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-r548n" event={"ID":"d8d37bd4-1c8c-4ecb-a173-954916afdc34","Type":"ContainerStarted","Data":"7781fb60d3c86ffdfd6a0007ec3f2d6657911c01c93b1836d9c0187d0e015940"} Oct 30 20:00:20.315331 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:20.315298 5088 generic.go:356] "Generic (PLEG): container finished" podID="d8d37bd4-1c8c-4ecb-a173-954916afdc34" containerID="7781fb60d3c86ffdfd6a0007ec3f2d6657911c01c93b1836d9c0187d0e015940" exitCode=60 Oct 30 20:00:20.315778 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:20.315344 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-r548n" event={"ID":"d8d37bd4-1c8c-4ecb-a173-954916afdc34","Type":"ContainerDied","Data":"7781fb60d3c86ffdfd6a0007ec3f2d6657911c01c93b1836d9c0187d0e015940"} Oct 30 20:00:20.315778 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:20.315398 5088 scope.go:122] "RemoveContainer" containerID="d39408d04fffb604f1d8b3659a6fa7b9d2573b538989dc0281d6872648a2359b" Oct 30 20:00:20.315877 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:20.315802 5088 scope.go:122] "RemoveContainer" containerID="7781fb60d3c86ffdfd6a0007ec3f2d6657911c01c93b1836d9c0187d0e015940" Oct 30 20:00:20.316069 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:00:20.316049 5088 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889840-r548n_odh-ai-gateway-infra(d8d37bd4-1c8c-4ecb-a173-954916afdc34)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-r548n" podUID="d8d37bd4-1c8c-4ecb-a173-954916afdc34" Oct 30 20:00:20.372457 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:20.372429 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-r548n"] Oct 30 20:00:21.458245 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:21.458221 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-r548n" Oct 30 20:00:21.560888 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:21.560858 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8d37bd4-1c8c-4ecb-a173-954916afdc34-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8d37bd4-1c8c-4ecb-a173-954916afdc34-maas-api-service-ca\") pod \"d8d37bd4-1c8c-4ecb-a173-954916afdc34\" (UID: \"d8d37bd4-1c8c-4ecb-a173-954916afdc34\") " Oct 30 20:00:21.561080 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:21.560950 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8d37bd4-1c8c-4ecb-a173-954916afdc34-kube-api-access-9lj2l\" (UniqueName: \"kubernetes.io/projected/d8d37bd4-1c8c-4ecb-a173-954916afdc34-kube-api-access-9lj2l\") pod \"d8d37bd4-1c8c-4ecb-a173-954916afdc34\" (UID: \"d8d37bd4-1c8c-4ecb-a173-954916afdc34\") " Oct 30 20:00:21.561201 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:21.561177 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8d37bd4-1c8c-4ecb-a173-954916afdc34-maas-api-service-ca" pod "d8d37bd4-1c8c-4ecb-a173-954916afdc34" (UID: "d8d37bd4-1c8c-4ecb-a173-954916afdc34"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:21.562950 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:21.562926 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8d37bd4-1c8c-4ecb-a173-954916afdc34-kube-api-access-9lj2l" pod "d8d37bd4-1c8c-4ecb-a173-954916afdc34" (UID: "d8d37bd4-1c8c-4ecb-a173-954916afdc34"). InnerVolumeSpecName "kube-api-access-9lj2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:21.662418 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:21.662355 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8d37bd4-1c8c-4ecb-a173-954916afdc34-maas-api-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:00:21.662418 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:21.662379 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lj2l\" (UniqueName: \"kubernetes.io/projected/d8d37bd4-1c8c-4ecb-a173-954916afdc34-kube-api-access-9lj2l\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:00:22.326564 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:22.326536 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-r548n" Oct 30 20:00:22.326754 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:22.326567 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-r548n" event={"ID":"d8d37bd4-1c8c-4ecb-a173-954916afdc34","Type":"ContainerDied","Data":"a9b1aae30f272233e2cf904df5a870ffc91533dfb1c272bf73786f30ed33dceb"} Oct 30 20:00:22.326754 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:22.326645 5088 scope.go:122] "RemoveContainer" containerID="7781fb60d3c86ffdfd6a0007ec3f2d6657911c01c93b1836d9c0187d0e015940" Oct 30 20:00:22.384520 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:22.384496 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-r548n"] Oct 30 20:00:22.398198 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:22.398175 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-r548n"] Oct 30 20:00:22.437039 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:00:22.437010 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:00:22.737854 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:22.737824 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8d37bd4-1c8c-4ecb-a173-954916afdc34" path="/var/lib/kubelet/pods/d8d37bd4-1c8c-4ecb-a173-954916afdc34/volumes" Oct 30 20:00:24.000351 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.000320 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z"] Oct 30 20:00:24.000936 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.000921 5088 cpu_manager.go:414] "containerMap: removing container" podUID="d8d37bd4-1c8c-4ecb-a173-954916afdc34" containerName="cleanup" Oct 30 20:00:24.000994 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.000939 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8d37bd4-1c8c-4ecb-a173-954916afdc34" containerName="cleanup" Oct 30 20:00:24.000994 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.000969 5088 cpu_manager.go:414] "containerMap: removing container" podUID="d8d37bd4-1c8c-4ecb-a173-954916afdc34" containerName="cleanup" Oct 30 20:00:24.000994 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.000976 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8d37bd4-1c8c-4ecb-a173-954916afdc34" containerName="cleanup" Oct 30 20:00:24.001100 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.001081 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8d37bd4-1c8c-4ecb-a173-954916afdc34" containerName="cleanup" Oct 30 20:00:24.001100 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.001095 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8d37bd4-1c8c-4ecb-a173-954916afdc34" containerName="cleanup" Oct 30 20:00:24.006007 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.005984 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" Oct 30 20:00:24.038441 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.038419 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-626870-openshift-default-dockercfg-8x78h\"" Oct 30 20:00:24.040382 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.040362 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z"] Oct 30 20:00:24.089111 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.089086 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4b44c6e-891a-4073-9192-035aeaad10a4-workload-certs\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z\" (UID: \"f4b44c6e-891a-4073-9192-035aeaad10a4\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" Oct 30 20:00:24.089231 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.089122 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4b44c6e-891a-4073-9192-035aeaad10a4-istio-podinfo\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z\" (UID: \"f4b44c6e-891a-4073-9192-035aeaad10a4\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" Oct 30 20:00:24.089231 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.089144 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4b44c6e-891a-4073-9192-035aeaad10a4-istio-token\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z\" (UID: \"f4b44c6e-891a-4073-9192-035aeaad10a4\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" Oct 30 20:00:24.089323 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.089251 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4b44c6e-891a-4073-9192-035aeaad10a4-istio-data\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z\" (UID: \"f4b44c6e-891a-4073-9192-035aeaad10a4\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" Oct 30 20:00:24.089323 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.089305 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4b44c6e-891a-4073-9192-035aeaad10a4-istiod-ca-cert\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z\" (UID: \"f4b44c6e-891a-4073-9192-035aeaad10a4\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" Oct 30 20:00:24.089419 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.089395 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b44c6e-891a-4073-9192-035aeaad10a4-workload-socket\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z\" (UID: \"f4b44c6e-891a-4073-9192-035aeaad10a4\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" Oct 30 20:00:24.089495 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.089454 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b44c6e-891a-4073-9192-035aeaad10a4-credential-socket\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z\" (UID: \"f4b44c6e-891a-4073-9192-035aeaad10a4\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" Oct 30 20:00:24.089527 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.089515 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4b44c6e-891a-4073-9192-035aeaad10a4-istio-envoy\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z\" (UID: \"f4b44c6e-891a-4073-9192-035aeaad10a4\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" Oct 30 20:00:24.089647 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.089601 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bhbp\" (UniqueName: \"kubernetes.io/projected/f4b44c6e-891a-4073-9192-035aeaad10a4-kube-api-access-2bhbp\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z\" (UID: \"f4b44c6e-891a-4073-9192-035aeaad10a4\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" Oct 30 20:00:24.133256 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.133223 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98"] Oct 30 20:00:24.133867 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.133847 5088 cpu_manager.go:414] "containerMap: removing container" podUID="d8d37bd4-1c8c-4ecb-a173-954916afdc34" containerName="cleanup" Oct 30 20:00:24.134038 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.134018 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8d37bd4-1c8c-4ecb-a173-954916afdc34" containerName="cleanup" Oct 30 20:00:24.134181 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.134169 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8d37bd4-1c8c-4ecb-a173-954916afdc34" containerName="cleanup" Oct 30 20:00:24.137646 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.137602 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" Oct 30 20:00:24.155651 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.155605 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-efbac3-openshift-default-dockercfg-wbss2\"" Oct 30 20:00:24.162202 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.162177 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98"] Oct 30 20:00:24.190565 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.190535 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4b44c6e-891a-4073-9192-035aeaad10a4-istiod-ca-cert\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z\" (UID: \"f4b44c6e-891a-4073-9192-035aeaad10a4\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" Oct 30 20:00:24.190720 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.190582 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b44c6e-891a-4073-9192-035aeaad10a4-workload-socket\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z\" (UID: \"f4b44c6e-891a-4073-9192-035aeaad10a4\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" Oct 30 20:00:24.190720 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.190664 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b44c6e-891a-4073-9192-035aeaad10a4-credential-socket\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z\" (UID: \"f4b44c6e-891a-4073-9192-035aeaad10a4\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" Oct 30 20:00:24.190841 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.190729 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4b44c6e-891a-4073-9192-035aeaad10a4-istio-envoy\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z\" (UID: \"f4b44c6e-891a-4073-9192-035aeaad10a4\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" Oct 30 20:00:24.190841 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.190759 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bhbp\" (UniqueName: \"kubernetes.io/projected/f4b44c6e-891a-4073-9192-035aeaad10a4-kube-api-access-2bhbp\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z\" (UID: \"f4b44c6e-891a-4073-9192-035aeaad10a4\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" Oct 30 20:00:24.190841 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.190801 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4b44c6e-891a-4073-9192-035aeaad10a4-workload-certs\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z\" (UID: \"f4b44c6e-891a-4073-9192-035aeaad10a4\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" Oct 30 20:00:24.190841 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.190834 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4b44c6e-891a-4073-9192-035aeaad10a4-istio-podinfo\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z\" (UID: \"f4b44c6e-891a-4073-9192-035aeaad10a4\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" Oct 30 20:00:24.191046 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.190861 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4b44c6e-891a-4073-9192-035aeaad10a4-istio-token\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z\" (UID: \"f4b44c6e-891a-4073-9192-035aeaad10a4\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" Oct 30 20:00:24.191046 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.190922 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4b44c6e-891a-4073-9192-035aeaad10a4-istio-data\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z\" (UID: \"f4b44c6e-891a-4073-9192-035aeaad10a4\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" Oct 30 20:00:24.191346 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.191323 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4b44c6e-891a-4073-9192-035aeaad10a4-istio-data\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z\" (UID: \"f4b44c6e-891a-4073-9192-035aeaad10a4\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" Oct 30 20:00:24.191440 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.191345 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4b44c6e-891a-4073-9192-035aeaad10a4-istiod-ca-cert\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z\" (UID: \"f4b44c6e-891a-4073-9192-035aeaad10a4\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" Oct 30 20:00:24.191440 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.191362 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b44c6e-891a-4073-9192-035aeaad10a4-credential-socket\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z\" (UID: \"f4b44c6e-891a-4073-9192-035aeaad10a4\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" Oct 30 20:00:24.191643 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.191602 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4b44c6e-891a-4073-9192-035aeaad10a4-workload-certs\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z\" (UID: \"f4b44c6e-891a-4073-9192-035aeaad10a4\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" Oct 30 20:00:24.191924 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.191904 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b44c6e-891a-4073-9192-035aeaad10a4-workload-socket\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z\" (UID: \"f4b44c6e-891a-4073-9192-035aeaad10a4\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" Oct 30 20:00:24.193630 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.193586 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4b44c6e-891a-4073-9192-035aeaad10a4-istio-envoy\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z\" (UID: \"f4b44c6e-891a-4073-9192-035aeaad10a4\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" Oct 30 20:00:24.194315 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.194292 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4b44c6e-891a-4073-9192-035aeaad10a4-istio-podinfo\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z\" (UID: \"f4b44c6e-891a-4073-9192-035aeaad10a4\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" Oct 30 20:00:24.217518 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.217486 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4b44c6e-891a-4073-9192-035aeaad10a4-istio-token\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z\" (UID: \"f4b44c6e-891a-4073-9192-035aeaad10a4\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" Oct 30 20:00:24.217990 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.217968 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bhbp\" (UniqueName: \"kubernetes.io/projected/f4b44c6e-891a-4073-9192-035aeaad10a4-kube-api-access-2bhbp\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z\" (UID: \"f4b44c6e-891a-4073-9192-035aeaad10a4\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" Oct 30 20:00:24.291917 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.291837 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce337cd9-bd40-432b-998d-4357f13b46d1-istio-envoy\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98\" (UID: \"ce337cd9-bd40-432b-998d-4357f13b46d1\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" Oct 30 20:00:24.291917 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.291888 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce337cd9-bd40-432b-998d-4357f13b46d1-istiod-ca-cert\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98\" (UID: \"ce337cd9-bd40-432b-998d-4357f13b46d1\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" Oct 30 20:00:24.292087 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.292025 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce337cd9-bd40-432b-998d-4357f13b46d1-istio-token\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98\" (UID: \"ce337cd9-bd40-432b-998d-4357f13b46d1\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" Oct 30 20:00:24.292087 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.292071 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhzwd\" (UniqueName: \"kubernetes.io/projected/ce337cd9-bd40-432b-998d-4357f13b46d1-kube-api-access-dhzwd\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98\" (UID: \"ce337cd9-bd40-432b-998d-4357f13b46d1\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" Oct 30 20:00:24.292157 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.292116 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce337cd9-bd40-432b-998d-4357f13b46d1-workload-socket\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98\" (UID: \"ce337cd9-bd40-432b-998d-4357f13b46d1\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" Oct 30 20:00:24.292204 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.292191 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce337cd9-bd40-432b-998d-4357f13b46d1-istio-data\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98\" (UID: \"ce337cd9-bd40-432b-998d-4357f13b46d1\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" Oct 30 20:00:24.292261 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.292249 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce337cd9-bd40-432b-998d-4357f13b46d1-workload-certs\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98\" (UID: \"ce337cd9-bd40-432b-998d-4357f13b46d1\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" Oct 30 20:00:24.292293 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.292277 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce337cd9-bd40-432b-998d-4357f13b46d1-credential-socket\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98\" (UID: \"ce337cd9-bd40-432b-998d-4357f13b46d1\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" Oct 30 20:00:24.292328 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.292295 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce337cd9-bd40-432b-998d-4357f13b46d1-istio-podinfo\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98\" (UID: \"ce337cd9-bd40-432b-998d-4357f13b46d1\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" Oct 30 20:00:24.319444 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.319415 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" Oct 30 20:00:24.393393 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.393359 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce337cd9-bd40-432b-998d-4357f13b46d1-workload-certs\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98\" (UID: \"ce337cd9-bd40-432b-998d-4357f13b46d1\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" Oct 30 20:00:24.393547 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.393406 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce337cd9-bd40-432b-998d-4357f13b46d1-credential-socket\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98\" (UID: \"ce337cd9-bd40-432b-998d-4357f13b46d1\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" Oct 30 20:00:24.393547 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.393513 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce337cd9-bd40-432b-998d-4357f13b46d1-istio-podinfo\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98\" (UID: \"ce337cd9-bd40-432b-998d-4357f13b46d1\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" Oct 30 20:00:24.393695 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.393635 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce337cd9-bd40-432b-998d-4357f13b46d1-istio-envoy\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98\" (UID: \"ce337cd9-bd40-432b-998d-4357f13b46d1\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" Oct 30 20:00:24.393695 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.393669 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce337cd9-bd40-432b-998d-4357f13b46d1-istiod-ca-cert\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98\" (UID: \"ce337cd9-bd40-432b-998d-4357f13b46d1\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" Oct 30 20:00:24.393866 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.393846 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce337cd9-bd40-432b-998d-4357f13b46d1-workload-certs\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98\" (UID: \"ce337cd9-bd40-432b-998d-4357f13b46d1\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" Oct 30 20:00:24.393929 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.393861 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce337cd9-bd40-432b-998d-4357f13b46d1-credential-socket\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98\" (UID: \"ce337cd9-bd40-432b-998d-4357f13b46d1\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" Oct 30 20:00:24.394534 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.394108 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce337cd9-bd40-432b-998d-4357f13b46d1-istio-token\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98\" (UID: \"ce337cd9-bd40-432b-998d-4357f13b46d1\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" Oct 30 20:00:24.394534 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.394185 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhzwd\" (UniqueName: \"kubernetes.io/projected/ce337cd9-bd40-432b-998d-4357f13b46d1-kube-api-access-dhzwd\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98\" (UID: \"ce337cd9-bd40-432b-998d-4357f13b46d1\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" Oct 30 20:00:24.394534 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.394216 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce337cd9-bd40-432b-998d-4357f13b46d1-workload-socket\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98\" (UID: \"ce337cd9-bd40-432b-998d-4357f13b46d1\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" Oct 30 20:00:24.394534 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.394315 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce337cd9-bd40-432b-998d-4357f13b46d1-istio-data\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98\" (UID: \"ce337cd9-bd40-432b-998d-4357f13b46d1\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" Oct 30 20:00:24.394534 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.394461 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce337cd9-bd40-432b-998d-4357f13b46d1-istiod-ca-cert\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98\" (UID: \"ce337cd9-bd40-432b-998d-4357f13b46d1\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" Oct 30 20:00:24.394955 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.394558 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce337cd9-bd40-432b-998d-4357f13b46d1-workload-socket\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98\" (UID: \"ce337cd9-bd40-432b-998d-4357f13b46d1\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" Oct 30 20:00:24.394955 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.394656 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce337cd9-bd40-432b-998d-4357f13b46d1-istio-data\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98\" (UID: \"ce337cd9-bd40-432b-998d-4357f13b46d1\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" Oct 30 20:00:24.397100 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.397077 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce337cd9-bd40-432b-998d-4357f13b46d1-istio-envoy\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98\" (UID: \"ce337cd9-bd40-432b-998d-4357f13b46d1\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" Oct 30 20:00:24.399309 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.399286 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce337cd9-bd40-432b-998d-4357f13b46d1-istio-podinfo\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98\" (UID: \"ce337cd9-bd40-432b-998d-4357f13b46d1\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" Oct 30 20:00:24.427859 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.427830 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce337cd9-bd40-432b-998d-4357f13b46d1-istio-token\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98\" (UID: \"ce337cd9-bd40-432b-998d-4357f13b46d1\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" Oct 30 20:00:24.428471 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.428449 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhzwd\" (UniqueName: \"kubernetes.io/projected/ce337cd9-bd40-432b-998d-4357f13b46d1-kube-api-access-dhzwd\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98\" (UID: \"ce337cd9-bd40-432b-998d-4357f13b46d1\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" Oct 30 20:00:24.452986 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.452960 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" Oct 30 20:00:24.577500 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.577416 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s"] Oct 30 20:00:24.582835 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.582809 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" Oct 30 20:00:24.603747 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.603719 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-912f24-openshift-default-dockercfg-pzcfl\"" Oct 30 20:00:24.605524 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.605501 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s"] Oct 30 20:00:24.658565 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.658538 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z"] Oct 30 20:00:24.660011 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.659975 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:00:24.660130 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.660061 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:00:24.660130 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.660106 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:00:24.697718 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.697685 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05a46855-a143-4c6d-8c87-404dc271bdb3-istio-data\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s\" (UID: \"05a46855-a143-4c6d-8c87-404dc271bdb3\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" Oct 30 20:00:24.697828 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.697783 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05a46855-a143-4c6d-8c87-404dc271bdb3-istio-token\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s\" (UID: \"05a46855-a143-4c6d-8c87-404dc271bdb3\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" Oct 30 20:00:24.697909 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.697887 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a46855-a143-4c6d-8c87-404dc271bdb3-credential-socket\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s\" (UID: \"05a46855-a143-4c6d-8c87-404dc271bdb3\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" Oct 30 20:00:24.697968 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.697924 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05a46855-a143-4c6d-8c87-404dc271bdb3-istiod-ca-cert\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s\" (UID: \"05a46855-a143-4c6d-8c87-404dc271bdb3\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" Oct 30 20:00:24.698023 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.697969 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05a46855-a143-4c6d-8c87-404dc271bdb3-istio-envoy\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s\" (UID: \"05a46855-a143-4c6d-8c87-404dc271bdb3\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" Oct 30 20:00:24.698093 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.698074 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a46855-a143-4c6d-8c87-404dc271bdb3-workload-socket\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s\" (UID: \"05a46855-a143-4c6d-8c87-404dc271bdb3\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" Oct 30 20:00:24.698149 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.698112 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05a46855-a143-4c6d-8c87-404dc271bdb3-istio-podinfo\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s\" (UID: \"05a46855-a143-4c6d-8c87-404dc271bdb3\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" Oct 30 20:00:24.698233 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.698207 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05a46855-a143-4c6d-8c87-404dc271bdb3-workload-certs\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s\" (UID: \"05a46855-a143-4c6d-8c87-404dc271bdb3\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" Oct 30 20:00:24.698341 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.698285 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr2gs\" (UniqueName: \"kubernetes.io/projected/05a46855-a143-4c6d-8c87-404dc271bdb3-kube-api-access-jr2gs\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s\" (UID: \"05a46855-a143-4c6d-8c87-404dc271bdb3\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" Oct 30 20:00:24.728533 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.728506 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf"] Oct 30 20:00:24.736080 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.734730 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" Oct 30 20:00:24.757850 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.757822 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf"] Oct 30 20:00:24.779045 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.778963 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98"] Oct 30 20:00:24.779917 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:00:24.779894 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce337cd9_bd40_432b_998d_4357f13b46d1.slice/crio-0ad5cb9ad0b2e2b692c429adfe2239453e5d28cc2d65b300f0a0d411a9e4738d WatchSource:0}: Error finding container 0ad5cb9ad0b2e2b692c429adfe2239453e5d28cc2d65b300f0a0d411a9e4738d: Status 404 returned error can't find the container with id 0ad5cb9ad0b2e2b692c429adfe2239453e5d28cc2d65b300f0a0d411a9e4738d Oct 30 20:00:24.783440 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.783402 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:00:24.783538 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.783512 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:00:24.783606 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.783564 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:00:24.793463 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.793405 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-51dd62-openshift-default-dockercfg-678jt\"" Oct 30 20:00:24.799524 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.799484 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05a46855-a143-4c6d-8c87-404dc271bdb3-istio-token\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s\" (UID: \"05a46855-a143-4c6d-8c87-404dc271bdb3\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" Oct 30 20:00:24.799653 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.799545 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a46855-a143-4c6d-8c87-404dc271bdb3-credential-socket\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s\" (UID: \"05a46855-a143-4c6d-8c87-404dc271bdb3\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" Oct 30 20:00:24.799653 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.799577 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05a46855-a143-4c6d-8c87-404dc271bdb3-istiod-ca-cert\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s\" (UID: \"05a46855-a143-4c6d-8c87-404dc271bdb3\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" Oct 30 20:00:24.799653 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.799639 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05a46855-a143-4c6d-8c87-404dc271bdb3-istio-envoy\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s\" (UID: \"05a46855-a143-4c6d-8c87-404dc271bdb3\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" Oct 30 20:00:24.799811 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.799674 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a46855-a143-4c6d-8c87-404dc271bdb3-workload-socket\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s\" (UID: \"05a46855-a143-4c6d-8c87-404dc271bdb3\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" Oct 30 20:00:24.799811 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.799731 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05a46855-a143-4c6d-8c87-404dc271bdb3-istio-podinfo\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s\" (UID: \"05a46855-a143-4c6d-8c87-404dc271bdb3\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" Oct 30 20:00:24.799811 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.799772 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05a46855-a143-4c6d-8c87-404dc271bdb3-workload-certs\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s\" (UID: \"05a46855-a143-4c6d-8c87-404dc271bdb3\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" Oct 30 20:00:24.799956 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.799841 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr2gs\" (UniqueName: \"kubernetes.io/projected/05a46855-a143-4c6d-8c87-404dc271bdb3-kube-api-access-jr2gs\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s\" (UID: \"05a46855-a143-4c6d-8c87-404dc271bdb3\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" Oct 30 20:00:24.799956 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.799925 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05a46855-a143-4c6d-8c87-404dc271bdb3-istio-data\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s\" (UID: \"05a46855-a143-4c6d-8c87-404dc271bdb3\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" Oct 30 20:00:24.800499 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.800478 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05a46855-a143-4c6d-8c87-404dc271bdb3-istio-data\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s\" (UID: \"05a46855-a143-4c6d-8c87-404dc271bdb3\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" Oct 30 20:00:24.801111 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.801089 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a46855-a143-4c6d-8c87-404dc271bdb3-credential-socket\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s\" (UID: \"05a46855-a143-4c6d-8c87-404dc271bdb3\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" Oct 30 20:00:24.801761 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.801734 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05a46855-a143-4c6d-8c87-404dc271bdb3-istiod-ca-cert\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s\" (UID: \"05a46855-a143-4c6d-8c87-404dc271bdb3\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" Oct 30 20:00:24.801985 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.801966 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05a46855-a143-4c6d-8c87-404dc271bdb3-workload-certs\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s\" (UID: \"05a46855-a143-4c6d-8c87-404dc271bdb3\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" Oct 30 20:00:24.804087 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.804040 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a46855-a143-4c6d-8c87-404dc271bdb3-workload-socket\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s\" (UID: \"05a46855-a143-4c6d-8c87-404dc271bdb3\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" Oct 30 20:00:24.804777 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.804760 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05a46855-a143-4c6d-8c87-404dc271bdb3-istio-podinfo\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s\" (UID: \"05a46855-a143-4c6d-8c87-404dc271bdb3\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" Oct 30 20:00:24.806721 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.806699 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05a46855-a143-4c6d-8c87-404dc271bdb3-istio-envoy\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s\" (UID: \"05a46855-a143-4c6d-8c87-404dc271bdb3\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" Oct 30 20:00:24.873678 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.860965 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05a46855-a143-4c6d-8c87-404dc271bdb3-istio-token\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s\" (UID: \"05a46855-a143-4c6d-8c87-404dc271bdb3\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" Oct 30 20:00:24.873678 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.869293 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr2gs\" (UniqueName: \"kubernetes.io/projected/05a46855-a143-4c6d-8c87-404dc271bdb3-kube-api-access-jr2gs\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s\" (UID: \"05a46855-a143-4c6d-8c87-404dc271bdb3\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" Oct 30 20:00:24.920727 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.898364 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" Oct 30 20:00:24.920727 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.901439 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0748781-63b9-4af1-8eab-5d0cef276da6-credential-socket\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf\" (UID: \"c0748781-63b9-4af1-8eab-5d0cef276da6\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" Oct 30 20:00:24.920727 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.901494 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0748781-63b9-4af1-8eab-5d0cef276da6-istio-data\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf\" (UID: \"c0748781-63b9-4af1-8eab-5d0cef276da6\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" Oct 30 20:00:24.920727 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.901523 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0748781-63b9-4af1-8eab-5d0cef276da6-workload-socket\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf\" (UID: \"c0748781-63b9-4af1-8eab-5d0cef276da6\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" Oct 30 20:00:24.920727 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.901598 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0748781-63b9-4af1-8eab-5d0cef276da6-istio-token\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf\" (UID: \"c0748781-63b9-4af1-8eab-5d0cef276da6\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" Oct 30 20:00:24.920727 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.901639 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0748781-63b9-4af1-8eab-5d0cef276da6-workload-certs\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf\" (UID: \"c0748781-63b9-4af1-8eab-5d0cef276da6\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" Oct 30 20:00:24.920727 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.901673 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0748781-63b9-4af1-8eab-5d0cef276da6-istiod-ca-cert\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf\" (UID: \"c0748781-63b9-4af1-8eab-5d0cef276da6\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" Oct 30 20:00:24.920727 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.901707 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9lqb\" (UniqueName: \"kubernetes.io/projected/c0748781-63b9-4af1-8eab-5d0cef276da6-kube-api-access-k9lqb\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf\" (UID: \"c0748781-63b9-4af1-8eab-5d0cef276da6\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" Oct 30 20:00:24.920727 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.901776 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0748781-63b9-4af1-8eab-5d0cef276da6-istio-podinfo\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf\" (UID: \"c0748781-63b9-4af1-8eab-5d0cef276da6\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" Oct 30 20:00:24.920727 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:24.901852 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0748781-63b9-4af1-8eab-5d0cef276da6-istio-envoy\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf\" (UID: \"c0748781-63b9-4af1-8eab-5d0cef276da6\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" Oct 30 20:00:25.011823 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:25.009016 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0748781-63b9-4af1-8eab-5d0cef276da6-istio-token\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf\" (UID: \"c0748781-63b9-4af1-8eab-5d0cef276da6\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" Oct 30 20:00:25.011823 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:25.009083 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0748781-63b9-4af1-8eab-5d0cef276da6-workload-certs\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf\" (UID: \"c0748781-63b9-4af1-8eab-5d0cef276da6\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" Oct 30 20:00:25.011823 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:25.009121 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0748781-63b9-4af1-8eab-5d0cef276da6-istiod-ca-cert\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf\" (UID: \"c0748781-63b9-4af1-8eab-5d0cef276da6\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" Oct 30 20:00:25.011823 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:25.009161 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9lqb\" (UniqueName: \"kubernetes.io/projected/c0748781-63b9-4af1-8eab-5d0cef276da6-kube-api-access-k9lqb\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf\" (UID: \"c0748781-63b9-4af1-8eab-5d0cef276da6\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" Oct 30 20:00:25.011823 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:25.009229 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0748781-63b9-4af1-8eab-5d0cef276da6-istio-podinfo\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf\" (UID: \"c0748781-63b9-4af1-8eab-5d0cef276da6\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" Oct 30 20:00:25.011823 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:25.009285 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0748781-63b9-4af1-8eab-5d0cef276da6-istio-envoy\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf\" (UID: \"c0748781-63b9-4af1-8eab-5d0cef276da6\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" Oct 30 20:00:25.011823 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:25.009383 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0748781-63b9-4af1-8eab-5d0cef276da6-credential-socket\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf\" (UID: \"c0748781-63b9-4af1-8eab-5d0cef276da6\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" Oct 30 20:00:25.011823 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:25.009419 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0748781-63b9-4af1-8eab-5d0cef276da6-istio-data\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf\" (UID: \"c0748781-63b9-4af1-8eab-5d0cef276da6\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" Oct 30 20:00:25.011823 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:25.009447 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0748781-63b9-4af1-8eab-5d0cef276da6-workload-socket\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf\" (UID: \"c0748781-63b9-4af1-8eab-5d0cef276da6\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" Oct 30 20:00:25.011823 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:25.009952 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0748781-63b9-4af1-8eab-5d0cef276da6-workload-socket\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf\" (UID: \"c0748781-63b9-4af1-8eab-5d0cef276da6\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" Oct 30 20:00:25.011823 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:25.010538 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0748781-63b9-4af1-8eab-5d0cef276da6-workload-certs\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf\" (UID: \"c0748781-63b9-4af1-8eab-5d0cef276da6\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" Oct 30 20:00:25.011823 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:25.011188 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0748781-63b9-4af1-8eab-5d0cef276da6-istiod-ca-cert\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf\" (UID: \"c0748781-63b9-4af1-8eab-5d0cef276da6\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" Oct 30 20:00:25.018047 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:25.018027 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0748781-63b9-4af1-8eab-5d0cef276da6-credential-socket\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf\" (UID: \"c0748781-63b9-4af1-8eab-5d0cef276da6\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" Oct 30 20:00:25.018583 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:25.018468 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0748781-63b9-4af1-8eab-5d0cef276da6-istio-data\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf\" (UID: \"c0748781-63b9-4af1-8eab-5d0cef276da6\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" Oct 30 20:00:25.046160 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:25.045785 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0748781-63b9-4af1-8eab-5d0cef276da6-istio-token\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf\" (UID: \"c0748781-63b9-4af1-8eab-5d0cef276da6\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" Oct 30 20:00:25.051744 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:25.051705 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9lqb\" (UniqueName: \"kubernetes.io/projected/c0748781-63b9-4af1-8eab-5d0cef276da6-kube-api-access-k9lqb\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf\" (UID: \"c0748781-63b9-4af1-8eab-5d0cef276da6\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" Oct 30 20:00:25.055518 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:25.055454 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0748781-63b9-4af1-8eab-5d0cef276da6-istio-podinfo\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf\" (UID: \"c0748781-63b9-4af1-8eab-5d0cef276da6\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" Oct 30 20:00:25.056851 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:25.056827 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0748781-63b9-4af1-8eab-5d0cef276da6-istio-envoy\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf\" (UID: \"c0748781-63b9-4af1-8eab-5d0cef276da6\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" Oct 30 20:00:25.254339 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:25.254300 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s"] Oct 30 20:00:25.258907 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:00:25.258876 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05a46855_a143_4c6d_8c87_404dc271bdb3.slice/crio-e98b120b8cfb5b7d9eb2682c24467d32b571678a51792f76b426412a5284c49f WatchSource:0}: Error finding container e98b120b8cfb5b7d9eb2682c24467d32b571678a51792f76b426412a5284c49f: Status 404 returned error can't find the container with id e98b120b8cfb5b7d9eb2682c24467d32b571678a51792f76b426412a5284c49f Oct 30 20:00:25.261436 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:25.261404 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:00:25.261511 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:25.261472 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:00:25.261511 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:25.261501 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:00:25.345225 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:25.345195 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" event={"ID":"ce337cd9-bd40-432b-998d-4357f13b46d1","Type":"ContainerStarted","Data":"061917a119072909f856fe9e1d06cc2fc6221402faadd09ccfbf02ce0fff4c45"} Oct 30 20:00:25.345374 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:25.345261 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" event={"ID":"ce337cd9-bd40-432b-998d-4357f13b46d1","Type":"ContainerStarted","Data":"0ad5cb9ad0b2e2b692c429adfe2239453e5d28cc2d65b300f0a0d411a9e4738d"} Oct 30 20:00:25.346998 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:25.346969 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" event={"ID":"f4b44c6e-891a-4073-9192-035aeaad10a4","Type":"ContainerStarted","Data":"592772e60f2ac707fee713a5705108969e381b16166a0f80580a0819513cc1df"} Oct 30 20:00:25.347125 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:25.347037 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" event={"ID":"f4b44c6e-891a-4073-9192-035aeaad10a4","Type":"ContainerStarted","Data":"20e7dfeb87f6609d1712e902592e8a9b0e68189ad91a712c72c7bad4ab52cc03"} Oct 30 20:00:25.348642 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:25.348600 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" event={"ID":"05a46855-a143-4c6d-8c87-404dc271bdb3","Type":"ContainerStarted","Data":"ab404a6d0bb243a2362e993df58be2d5b55d6c8958bcc2f6e364ccc15da2c8f7"} Oct 30 20:00:25.348730 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:25.348664 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" event={"ID":"05a46855-a143-4c6d-8c87-404dc271bdb3","Type":"ContainerStarted","Data":"e98b120b8cfb5b7d9eb2682c24467d32b571678a51792f76b426412a5284c49f"} Oct 30 20:00:25.351269 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:25.351247 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" Oct 30 20:00:25.414944 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:25.412594 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" podStartSLOduration=1.412572396 podStartE2EDuration="1.412572396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:25.411068166 +0000 UTC m=+786.823781946" watchObservedRunningTime="2026-10-30 20:00:25.412572396 +0000 UTC m=+786.825286176" Oct 30 20:00:25.454045 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:25.454011 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" Oct 30 20:00:25.464398 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:25.460377 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" Oct 30 20:00:25.464976 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:25.464924 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" podStartSLOduration=2.4649074779999998 podStartE2EDuration="2.464907478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:25.464119409 +0000 UTC m=+786.876833193" watchObservedRunningTime="2026-10-30 20:00:25.464907478 +0000 UTC m=+786.877621261" Oct 30 20:00:25.516852 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:25.516741 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" podStartSLOduration=1.516726302 podStartE2EDuration="1.516726302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:25.515112204 +0000 UTC m=+786.927825978" watchObservedRunningTime="2026-10-30 20:00:25.516726302 +0000 UTC m=+786.929440080" Oct 30 20:00:25.675974 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:25.675943 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf"] Oct 30 20:00:25.677167 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:00:25.677141 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0748781_63b9_4af1_8eab_5d0cef276da6.slice/crio-dc36d72d10480d8b242c759d9b9086439678cd8802b0a9740ddf1ef74a40618f WatchSource:0}: Error finding container dc36d72d10480d8b242c759d9b9086439678cd8802b0a9740ddf1ef74a40618f: Status 404 returned error can't find the container with id dc36d72d10480d8b242c759d9b9086439678cd8802b0a9740ddf1ef74a40618f Oct 30 20:00:25.679472 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:25.679437 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:00:25.679553 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:25.679517 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:00:25.679647 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:25.679587 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:00:25.899289 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:25.899210 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" Oct 30 20:00:26.320513 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:26.320470 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" Oct 30 20:00:26.325800 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:26.325775 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" Oct 30 20:00:26.356580 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:26.356548 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" event={"ID":"c0748781-63b9-4af1-8eab-5d0cef276da6","Type":"ContainerStarted","Data":"a9bd049c96e0f99442c1823a643467f15e55bf80eb26fd607ffad0a2abee78bf"} Oct 30 20:00:26.356741 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:26.356604 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" event={"ID":"c0748781-63b9-4af1-8eab-5d0cef276da6","Type":"ContainerStarted","Data":"dc36d72d10480d8b242c759d9b9086439678cd8802b0a9740ddf1ef74a40618f"} Oct 30 20:00:26.357146 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:26.357119 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" Oct 30 20:00:26.357307 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:26.357171 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" Oct 30 20:00:26.357709 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:26.357687 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" Oct 30 20:00:26.357829 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:26.357810 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" Oct 30 20:00:26.529787 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:26.529602 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" podStartSLOduration=2.529586656 podStartE2EDuration="2.529586656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:26.529008319 +0000 UTC m=+787.941722099" watchObservedRunningTime="2026-10-30 20:00:26.529586656 +0000 UTC m=+787.942300435" Oct 30 20:00:26.904242 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:26.904212 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" Oct 30 20:00:27.352422 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:27.352392 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" Oct 30 20:00:27.357471 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:27.357448 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" Oct 30 20:00:27.362049 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:27.362016 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" Oct 30 20:00:27.362179 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:27.362065 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" Oct 30 20:00:27.362579 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:27.362558 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" Oct 30 20:00:27.362677 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:27.362642 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" Oct 30 20:00:32.118993 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:32.118958 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-efbac3-66db987bc9-ptftf"] Oct 30 20:00:32.123974 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:32.123956 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-efbac3-66db987bc9-ptftf" Oct 30 20:00:32.146077 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:32.146051 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-efbac3-66db987bc9-ptftf"] Oct 30 20:00:32.156395 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:32.156373 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-efbac3\"" Oct 30 20:00:32.156395 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:32.156386 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-efbac3-dockercfg-jvj4w\"" Oct 30 20:00:32.315161 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:32.315130 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t268w\" (UniqueName: \"kubernetes.io/projected/0ee0e296-2cf3-4cdc-9340-d202e75c6a14-kube-api-access-t268w\") pod \"payload-pre-processing-e2e-worker-w2-efbac3-66db987bc9-ptftf\" (UID: \"0ee0e296-2cf3-4cdc-9340-d202e75c6a14\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-efbac3-66db987bc9-ptftf" Oct 30 20:00:32.315338 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:32.315183 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ee0e296-2cf3-4cdc-9340-d202e75c6a14-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-efbac3-66db987bc9-ptftf\" (UID: \"0ee0e296-2cf3-4cdc-9340-d202e75c6a14\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-efbac3-66db987bc9-ptftf" Oct 30 20:00:32.415970 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:32.415868 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t268w\" (UniqueName: \"kubernetes.io/projected/0ee0e296-2cf3-4cdc-9340-d202e75c6a14-kube-api-access-t268w\") pod \"payload-pre-processing-e2e-worker-w2-efbac3-66db987bc9-ptftf\" (UID: \"0ee0e296-2cf3-4cdc-9340-d202e75c6a14\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-efbac3-66db987bc9-ptftf" Oct 30 20:00:32.416142 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:32.416024 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ee0e296-2cf3-4cdc-9340-d202e75c6a14-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-efbac3-66db987bc9-ptftf\" (UID: \"0ee0e296-2cf3-4cdc-9340-d202e75c6a14\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-efbac3-66db987bc9-ptftf" Oct 30 20:00:32.416941 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:32.416918 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ee0e296-2cf3-4cdc-9340-d202e75c6a14-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-efbac3-66db987bc9-ptftf\" (UID: \"0ee0e296-2cf3-4cdc-9340-d202e75c6a14\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-efbac3-66db987bc9-ptftf" Oct 30 20:00:32.444717 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:32.444689 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t268w\" (UniqueName: \"kubernetes.io/projected/0ee0e296-2cf3-4cdc-9340-d202e75c6a14-kube-api-access-t268w\") pod \"payload-pre-processing-e2e-worker-w2-efbac3-66db987bc9-ptftf\" (UID: \"0ee0e296-2cf3-4cdc-9340-d202e75c6a14\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-efbac3-66db987bc9-ptftf" Oct 30 20:00:32.737383 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:32.737351 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-efbac3-66db987bc9-ptftf" Oct 30 20:00:33.037604 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:33.037581 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-efbac3-66db987bc9-ptftf"] Oct 30 20:00:33.039362 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:00:33.039332 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ee0e296_2cf3_4cdc_9340_d202e75c6a14.slice/crio-69f193bc35b3e7c0d5e2ef2d18210b21866787ab2d67b7c4ee3c1f5e8895e51c WatchSource:0}: Error finding container 69f193bc35b3e7c0d5e2ef2d18210b21866787ab2d67b7c4ee3c1f5e8895e51c: Status 404 returned error can't find the container with id 69f193bc35b3e7c0d5e2ef2d18210b21866787ab2d67b7c4ee3c1f5e8895e51c Oct 30 20:00:33.427375 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:33.427291 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-efbac3-66db987bc9-ptftf" event={"ID":"0ee0e296-2cf3-4cdc-9340-d202e75c6a14","Type":"ContainerStarted","Data":"69f193bc35b3e7c0d5e2ef2d18210b21866787ab2d67b7c4ee3c1f5e8895e51c"} Oct 30 20:00:33.969962 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:33.969918 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:00:35.440682 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:35.440581 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-efbac3-66db987bc9-ptftf" event={"ID":"0ee0e296-2cf3-4cdc-9340-d202e75c6a14","Type":"ContainerStarted","Data":"253e2f1b3689f4cf53015de4c05a89f5d579f55bad8438f33b5ab9f23e73e376"} Oct 30 20:00:35.440682 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:35.440671 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-efbac3-66db987bc9-ptftf" Oct 30 20:00:35.814678 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:35.813998 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-efbac3-66db987bc9-ptftf" podStartSLOduration=1.6756078859999999 podStartE2EDuration="3.813973655s" totalImagesPullingTime="2.138365769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:35.506703452 +0000 UTC m=+796.919417232" watchObservedRunningTime="2026-10-30 20:00:35.813973655 +0000 UTC m=+797.226687434" Oct 30 20:00:35.882480 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:35.881045 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-62f4a1-645b9d8589-k6rx9"] Oct 30 20:00:35.928407 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:35.928372 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-62f4a1-645b9d8589-k6rx9"] Oct 30 20:00:35.928672 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:35.928495 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-62f4a1-645b9d8589-k6rx9" Oct 30 20:00:35.947065 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:35.947044 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-62f4a1-dockercfg-pfxkm\"" Oct 30 20:00:35.947284 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:35.947272 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-62f4a1\"" Oct 30 20:00:35.968410 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:35.968379 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/933b9fd2-86f8-4f62-a3e5-48636487d9b0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-62f4a1-645b9d8589-k6rx9\" (UID: \"933b9fd2-86f8-4f62-a3e5-48636487d9b0\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-62f4a1-645b9d8589-k6rx9" Oct 30 20:00:35.968505 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:35.968437 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk7th\" (UniqueName: \"kubernetes.io/projected/933b9fd2-86f8-4f62-a3e5-48636487d9b0-kube-api-access-sk7th\") pod \"payload-processing-e2e-worker-w0-62f4a1-645b9d8589-k6rx9\" (UID: \"933b9fd2-86f8-4f62-a3e5-48636487d9b0\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-62f4a1-645b9d8589-k6rx9" Oct 30 20:00:36.069677 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:36.069579 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/933b9fd2-86f8-4f62-a3e5-48636487d9b0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-62f4a1-645b9d8589-k6rx9\" (UID: \"933b9fd2-86f8-4f62-a3e5-48636487d9b0\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-62f4a1-645b9d8589-k6rx9" Oct 30 20:00:36.069677 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:36.069668 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk7th\" (UniqueName: \"kubernetes.io/projected/933b9fd2-86f8-4f62-a3e5-48636487d9b0-kube-api-access-sk7th\") pod \"payload-processing-e2e-worker-w0-62f4a1-645b9d8589-k6rx9\" (UID: \"933b9fd2-86f8-4f62-a3e5-48636487d9b0\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-62f4a1-645b9d8589-k6rx9" Oct 30 20:00:36.070247 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:36.070229 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/933b9fd2-86f8-4f62-a3e5-48636487d9b0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-62f4a1-645b9d8589-k6rx9\" (UID: \"933b9fd2-86f8-4f62-a3e5-48636487d9b0\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-62f4a1-645b9d8589-k6rx9" Oct 30 20:00:36.110560 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:36.110521 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk7th\" (UniqueName: \"kubernetes.io/projected/933b9fd2-86f8-4f62-a3e5-48636487d9b0-kube-api-access-sk7th\") pod \"payload-processing-e2e-worker-w0-62f4a1-645b9d8589-k6rx9\" (UID: \"933b9fd2-86f8-4f62-a3e5-48636487d9b0\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-62f4a1-645b9d8589-k6rx9" Oct 30 20:00:36.245498 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:36.245464 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-62f4a1-645b9d8589-k6rx9" Oct 30 20:00:36.581703 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:36.581676 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-62f4a1-645b9d8589-k6rx9"] Oct 30 20:00:36.582845 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:00:36.582819 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod933b9fd2_86f8_4f62_a3e5_48636487d9b0.slice/crio-f24bdbc40c5a8bd771f53a456989c661966a5cc56077b276321f6c9cd4a9f6e8 WatchSource:0}: Error finding container f24bdbc40c5a8bd771f53a456989c661966a5cc56077b276321f6c9cd4a9f6e8: Status 404 returned error can't find the container with id f24bdbc40c5a8bd771f53a456989c661966a5cc56077b276321f6c9cd4a9f6e8 Oct 30 20:00:36.708646 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:36.706204 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5f10d2-58d98b4775-jq5gd"] Oct 30 20:00:36.713090 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:36.713057 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f10d2-58d98b4775-jq5gd" Oct 30 20:00:36.735553 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:36.735528 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-5f10d2-dockercfg-zz9v8\"" Oct 30 20:00:36.735688 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:36.735595 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-5f10d2\"" Oct 30 20:00:36.740256 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:36.740235 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5f10d2-58d98b4775-jq5gd"] Oct 30 20:00:36.780506 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:36.780461 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcv6d\" (UniqueName: \"kubernetes.io/projected/24f9b32b-8908-4adf-bda3-5cfc8cf34bc5-kube-api-access-rcv6d\") pod \"payload-pre-processing-e2e-worker-w1-5f10d2-58d98b4775-jq5gd\" (UID: \"24f9b32b-8908-4adf-bda3-5cfc8cf34bc5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f10d2-58d98b4775-jq5gd" Oct 30 20:00:36.780745 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:36.780567 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24f9b32b-8908-4adf-bda3-5cfc8cf34bc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5f10d2-58d98b4775-jq5gd\" (UID: \"24f9b32b-8908-4adf-bda3-5cfc8cf34bc5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f10d2-58d98b4775-jq5gd" Oct 30 20:00:36.881522 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:36.881437 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24f9b32b-8908-4adf-bda3-5cfc8cf34bc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5f10d2-58d98b4775-jq5gd\" (UID: \"24f9b32b-8908-4adf-bda3-5cfc8cf34bc5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f10d2-58d98b4775-jq5gd" Oct 30 20:00:36.881522 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:36.881494 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcv6d\" (UniqueName: \"kubernetes.io/projected/24f9b32b-8908-4adf-bda3-5cfc8cf34bc5-kube-api-access-rcv6d\") pod \"payload-pre-processing-e2e-worker-w1-5f10d2-58d98b4775-jq5gd\" (UID: \"24f9b32b-8908-4adf-bda3-5cfc8cf34bc5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f10d2-58d98b4775-jq5gd" Oct 30 20:00:36.882414 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:36.882350 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24f9b32b-8908-4adf-bda3-5cfc8cf34bc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5f10d2-58d98b4775-jq5gd\" (UID: \"24f9b32b-8908-4adf-bda3-5cfc8cf34bc5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f10d2-58d98b4775-jq5gd" Oct 30 20:00:36.933821 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:36.933788 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcv6d\" (UniqueName: \"kubernetes.io/projected/24f9b32b-8908-4adf-bda3-5cfc8cf34bc5-kube-api-access-rcv6d\") pod \"payload-pre-processing-e2e-worker-w1-5f10d2-58d98b4775-jq5gd\" (UID: \"24f9b32b-8908-4adf-bda3-5cfc8cf34bc5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f10d2-58d98b4775-jq5gd" Oct 30 20:00:37.026384 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:37.026355 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f10d2-58d98b4775-jq5gd" Oct 30 20:00:37.328690 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:37.328665 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5f10d2-58d98b4775-jq5gd"] Oct 30 20:00:37.330461 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:00:37.330431 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24f9b32b_8908_4adf_bda3_5cfc8cf34bc5.slice/crio-55df9f8943fcffc304a9b09fc65184eb43b4ec213e479a6728601437e115c92c WatchSource:0}: Error finding container 55df9f8943fcffc304a9b09fc65184eb43b4ec213e479a6728601437e115c92c: Status 404 returned error can't find the container with id 55df9f8943fcffc304a9b09fc65184eb43b4ec213e479a6728601437e115c92c Oct 30 20:00:37.457149 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:37.457116 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-62f4a1-645b9d8589-k6rx9" event={"ID":"933b9fd2-86f8-4f62-a3e5-48636487d9b0","Type":"ContainerStarted","Data":"133ce1c35e98bbdfce32864378f02f84461a01e8d0bd8ae7160251bde03e9ece"} Oct 30 20:00:37.457286 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:37.457181 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-62f4a1-645b9d8589-k6rx9" event={"ID":"933b9fd2-86f8-4f62-a3e5-48636487d9b0","Type":"ContainerStarted","Data":"f24bdbc40c5a8bd771f53a456989c661966a5cc56077b276321f6c9cd4a9f6e8"} Oct 30 20:00:37.457333 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:37.457298 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-62f4a1-645b9d8589-k6rx9" Oct 30 20:00:37.458665 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:37.458643 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f10d2-58d98b4775-jq5gd" event={"ID":"24f9b32b-8908-4adf-bda3-5cfc8cf34bc5","Type":"ContainerStarted","Data":"489b0a7cc51dbb214bc5e596a066ac15f465bf372ee1bc62015e49f46dc7ae03"} Oct 30 20:00:37.458765 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:37.458684 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f10d2-58d98b4775-jq5gd" event={"ID":"24f9b32b-8908-4adf-bda3-5cfc8cf34bc5","Type":"ContainerStarted","Data":"55df9f8943fcffc304a9b09fc65184eb43b4ec213e479a6728601437e115c92c"} Oct 30 20:00:37.458807 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:37.458775 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f10d2-58d98b4775-jq5gd" Oct 30 20:00:37.509182 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:37.509099 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-62f4a1-645b9d8589-k6rx9" podStartSLOduration=2.509082356 podStartE2EDuration="2.509082356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:37.50881086 +0000 UTC m=+798.921524639" watchObservedRunningTime="2026-10-30 20:00:37.509082356 +0000 UTC m=+798.921796137" Oct 30 20:00:37.570433 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:37.570375 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f10d2-58d98b4775-jq5gd" podStartSLOduration=1.570355959 podStartE2EDuration="1.570355959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:37.567319453 +0000 UTC m=+798.980033243" watchObservedRunningTime="2026-10-30 20:00:37.570355959 +0000 UTC m=+798.983069741" Oct 30 20:00:38.621633 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:38.620284 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:00:39.738648 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:39.737718 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-51dd62-79f5b56fd7-ltqtf"] Oct 30 20:00:39.742935 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:39.742896 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-51dd62-79f5b56fd7-ltqtf"] Oct 30 20:00:39.743098 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:39.743025 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-51dd62-79f5b56fd7-ltqtf" Oct 30 20:00:39.777908 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:39.773289 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx95j\" (UniqueName: \"kubernetes.io/projected/ab7bcd6d-197b-4c1e-9ce1-4aa111e96e54-kube-api-access-rx95j\") pod \"payload-pre-processing-e2e-worker-w6-51dd62-79f5b56fd7-ltqtf\" (UID: \"ab7bcd6d-197b-4c1e-9ce1-4aa111e96e54\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-51dd62-79f5b56fd7-ltqtf" Oct 30 20:00:39.777908 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:39.773412 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab7bcd6d-197b-4c1e-9ce1-4aa111e96e54-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-51dd62-79f5b56fd7-ltqtf\" (UID: \"ab7bcd6d-197b-4c1e-9ce1-4aa111e96e54\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-51dd62-79f5b56fd7-ltqtf" Oct 30 20:00:39.781369 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:39.781343 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-51dd62-dockercfg-hwl6h\"" Oct 30 20:00:39.781599 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:39.781203 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-51dd62\"" Oct 30 20:00:39.811961 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:39.811933 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-51dd62-5668978b9c-5djrs"] Oct 30 20:00:39.929644 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:39.902217 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx95j\" (UniqueName: \"kubernetes.io/projected/ab7bcd6d-197b-4c1e-9ce1-4aa111e96e54-kube-api-access-rx95j\") pod \"payload-pre-processing-e2e-worker-w6-51dd62-79f5b56fd7-ltqtf\" (UID: \"ab7bcd6d-197b-4c1e-9ce1-4aa111e96e54\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-51dd62-79f5b56fd7-ltqtf" Oct 30 20:00:39.929644 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:39.902331 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab7bcd6d-197b-4c1e-9ce1-4aa111e96e54-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-51dd62-79f5b56fd7-ltqtf\" (UID: \"ab7bcd6d-197b-4c1e-9ce1-4aa111e96e54\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-51dd62-79f5b56fd7-ltqtf" Oct 30 20:00:39.929644 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:39.903306 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab7bcd6d-197b-4c1e-9ce1-4aa111e96e54-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-51dd62-79f5b56fd7-ltqtf\" (UID: \"ab7bcd6d-197b-4c1e-9ce1-4aa111e96e54\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-51dd62-79f5b56fd7-ltqtf" Oct 30 20:00:39.964537 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:39.964509 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-51dd62-5668978b9c-5djrs" Oct 30 20:00:40.026694 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:39.985058 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-51dd62-5668978b9c-5djrs"] Oct 30 20:00:40.058731 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:40.054165 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f2223b9-aebe-4aab-9437-af457ece8c69-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-51dd62-5668978b9c-5djrs\" (UID: \"2f2223b9-aebe-4aab-9437-af457ece8c69\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-51dd62-5668978b9c-5djrs" Oct 30 20:00:40.058731 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:40.054273 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pktm7\" (UniqueName: \"kubernetes.io/projected/2f2223b9-aebe-4aab-9437-af457ece8c69-kube-api-access-pktm7\") pod \"payload-processing-e2e-worker-w6-51dd62-5668978b9c-5djrs\" (UID: \"2f2223b9-aebe-4aab-9437-af457ece8c69\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-51dd62-5668978b9c-5djrs" Oct 30 20:00:40.068834 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:40.067214 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx95j\" (UniqueName: \"kubernetes.io/projected/ab7bcd6d-197b-4c1e-9ce1-4aa111e96e54-kube-api-access-rx95j\") pod \"payload-pre-processing-e2e-worker-w6-51dd62-79f5b56fd7-ltqtf\" (UID: \"ab7bcd6d-197b-4c1e-9ce1-4aa111e96e54\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-51dd62-79f5b56fd7-ltqtf" Oct 30 20:00:40.155807 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:40.155775 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pktm7\" (UniqueName: \"kubernetes.io/projected/2f2223b9-aebe-4aab-9437-af457ece8c69-kube-api-access-pktm7\") pod \"payload-processing-e2e-worker-w6-51dd62-5668978b9c-5djrs\" (UID: \"2f2223b9-aebe-4aab-9437-af457ece8c69\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-51dd62-5668978b9c-5djrs" Oct 30 20:00:40.156002 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:40.155924 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f2223b9-aebe-4aab-9437-af457ece8c69-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-51dd62-5668978b9c-5djrs\" (UID: \"2f2223b9-aebe-4aab-9437-af457ece8c69\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-51dd62-5668978b9c-5djrs" Oct 30 20:00:40.156685 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:40.156658 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f2223b9-aebe-4aab-9437-af457ece8c69-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-51dd62-5668978b9c-5djrs\" (UID: \"2f2223b9-aebe-4aab-9437-af457ece8c69\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-51dd62-5668978b9c-5djrs" Oct 30 20:00:40.202637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:40.183842 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pktm7\" (UniqueName: \"kubernetes.io/projected/2f2223b9-aebe-4aab-9437-af457ece8c69-kube-api-access-pktm7\") pod \"payload-processing-e2e-worker-w6-51dd62-5668978b9c-5djrs\" (UID: \"2f2223b9-aebe-4aab-9437-af457ece8c69\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-51dd62-5668978b9c-5djrs" Oct 30 20:00:40.360035 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:40.359954 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-51dd62-79f5b56fd7-ltqtf" Oct 30 20:00:40.373784 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:40.373573 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-51dd62-5668978b9c-5djrs" Oct 30 20:00:40.752883 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:40.752852 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-51dd62-79f5b56fd7-ltqtf"] Oct 30 20:00:40.754351 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:00:40.754329 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab7bcd6d_197b_4c1e_9ce1_4aa111e96e54.slice/crio-d2eb2124934140b1002aa7b022fecbcdbabba1bafdc94a9afdd430651f8f7228 WatchSource:0}: Error finding container d2eb2124934140b1002aa7b022fecbcdbabba1bafdc94a9afdd430651f8f7228: Status 404 returned error can't find the container with id d2eb2124934140b1002aa7b022fecbcdbabba1bafdc94a9afdd430651f8f7228 Oct 30 20:00:40.765297 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:40.765279 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-51dd62-5668978b9c-5djrs"] Oct 30 20:00:40.767832 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:00:40.767807 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f2223b9_aebe_4aab_9437_af457ece8c69.slice/crio-525b04fff0d6de0fcb97a0349c86f38750f52b631521c17c169eee1dfca2d29f WatchSource:0}: Error finding container 525b04fff0d6de0fcb97a0349c86f38750f52b631521c17c169eee1dfca2d29f: Status 404 returned error can't find the container with id 525b04fff0d6de0fcb97a0349c86f38750f52b631521c17c169eee1dfca2d29f Oct 30 20:00:41.514882 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:41.514840 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-51dd62-79f5b56fd7-ltqtf" event={"ID":"ab7bcd6d-197b-4c1e-9ce1-4aa111e96e54","Type":"ContainerStarted","Data":"36e4fdc76cc5e54e943202cd1add398d8d06a6e1797847f44f6ad6c8daa3b250"} Oct 30 20:00:41.515079 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:41.515003 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-51dd62-79f5b56fd7-ltqtf" Oct 30 20:00:41.515154 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:41.515076 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-51dd62-79f5b56fd7-ltqtf" event={"ID":"ab7bcd6d-197b-4c1e-9ce1-4aa111e96e54","Type":"ContainerStarted","Data":"d2eb2124934140b1002aa7b022fecbcdbabba1bafdc94a9afdd430651f8f7228"} Oct 30 20:00:41.516682 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:41.516656 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-51dd62-5668978b9c-5djrs" event={"ID":"2f2223b9-aebe-4aab-9437-af457ece8c69","Type":"ContainerStarted","Data":"a9b6f46f57963859ae28c7ec7bdbf404c5ec2de1c93b2e24e47bb988b171a3ca"} Oct 30 20:00:41.516816 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:41.516699 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-51dd62-5668978b9c-5djrs" event={"ID":"2f2223b9-aebe-4aab-9437-af457ece8c69","Type":"ContainerStarted","Data":"525b04fff0d6de0fcb97a0349c86f38750f52b631521c17c169eee1dfca2d29f"} Oct 30 20:00:41.516816 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:41.516737 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-51dd62-5668978b9c-5djrs" Oct 30 20:00:41.557487 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:41.557451 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:00:41.588953 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:41.568970 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-51dd62-79f5b56fd7-ltqtf" podStartSLOduration=2.568950869 podStartE2EDuration="2.568950869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:41.564961312 +0000 UTC m=+802.977675096" watchObservedRunningTime="2026-10-30 20:00:41.568950869 +0000 UTC m=+802.981664649" Oct 30 20:00:42.405079 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:42.404732 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:00:42.405079 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:42.404798 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" podUID="f4b44c6e-891a-4073-9192-035aeaad10a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:00:46.454094 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:46.454058 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-efbac3-66db987bc9-ptftf" Oct 30 20:00:46.510635 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:46.510369 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-51dd62-5668978b9c-5djrs" podStartSLOduration=7.510350912 podStartE2EDuration="7.510350912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:41.635917563 +0000 UTC m=+803.048631368" watchObservedRunningTime="2026-10-30 20:00:46.510350912 +0000 UTC m=+807.923064692" Oct 30 20:00:46.560201 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:46.560166 5088 generic.go:356] "Generic (PLEG): container finished" podID="f4b44c6e-891a-4073-9192-035aeaad10a4" containerID="592772e60f2ac707fee713a5705108969e381b16166a0f80580a0819513cc1df" exitCode=0 Oct 30 20:00:46.560376 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:46.560242 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" event={"ID":"f4b44c6e-891a-4073-9192-035aeaad10a4","Type":"ContainerDied","Data":"592772e60f2ac707fee713a5705108969e381b16166a0f80580a0819513cc1df"} Oct 30 20:00:46.560782 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:46.560761 5088 scope.go:122] "RemoveContainer" containerID="592772e60f2ac707fee713a5705108969e381b16166a0f80580a0819513cc1df" Oct 30 20:00:46.562164 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:46.562133 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:00:46.562262 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:46.562221 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:00:46.562327 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:46.562272 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:00:46.693443 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:46.692641 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:00:47.568533 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:47.567782 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" event={"ID":"f4b44c6e-891a-4073-9192-035aeaad10a4","Type":"ContainerStarted","Data":"3a6fcdf62cfa94bb23a47d8095c3f93e63a21035e5267a54f9439fb5482243d5"} Oct 30 20:00:48.055756 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:48.055188 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957"] Oct 30 20:00:48.074058 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:48.074019 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957"] Oct 30 20:00:48.074244 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:48.074093 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957" Oct 30 20:00:48.108126 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:48.108093 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:00:48.112682 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:48.108421 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:00:48.116723 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:48.108471 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-51dd62-serving-cert\"" Oct 30 20:00:48.116915 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:48.108526 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7wqnw\"" Oct 30 20:00:48.121692 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:48.108645 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 20:00:48.155915 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:48.155865 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be2ffe6a-d355-4d95-a7ae-46adeb952736-maas-api-tls\") pod \"maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957\" (UID: \"be2ffe6a-d355-4d95-a7ae-46adeb952736\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957" Oct 30 20:00:48.156111 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:48.155945 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be2ffe6a-d355-4d95-a7ae-46adeb952736-metrics-certs\") pod \"maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957\" (UID: \"be2ffe6a-d355-4d95-a7ae-46adeb952736\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957" Oct 30 20:00:48.156111 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:48.155974 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be2ffe6a-d355-4d95-a7ae-46adeb952736-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957\" (UID: \"be2ffe6a-d355-4d95-a7ae-46adeb952736\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957" Oct 30 20:00:48.156111 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:48.156008 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmd6k\" (UniqueName: \"kubernetes.io/projected/be2ffe6a-d355-4d95-a7ae-46adeb952736-kube-api-access-dmd6k\") pod \"maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957\" (UID: \"be2ffe6a-d355-4d95-a7ae-46adeb952736\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957" Oct 30 20:00:48.256994 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:48.256943 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be2ffe6a-d355-4d95-a7ae-46adeb952736-maas-api-tls\") pod \"maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957\" (UID: \"be2ffe6a-d355-4d95-a7ae-46adeb952736\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957" Oct 30 20:00:48.257214 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:48.257009 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be2ffe6a-d355-4d95-a7ae-46adeb952736-metrics-certs\") pod \"maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957\" (UID: \"be2ffe6a-d355-4d95-a7ae-46adeb952736\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957" Oct 30 20:00:48.257214 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:48.257028 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be2ffe6a-d355-4d95-a7ae-46adeb952736-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957\" (UID: \"be2ffe6a-d355-4d95-a7ae-46adeb952736\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957" Oct 30 20:00:48.257214 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:48.257059 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmd6k\" (UniqueName: \"kubernetes.io/projected/be2ffe6a-d355-4d95-a7ae-46adeb952736-kube-api-access-dmd6k\") pod \"maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957\" (UID: \"be2ffe6a-d355-4d95-a7ae-46adeb952736\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957" Oct 30 20:00:48.257560 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:00:48.257541 5088 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-51dd62-serving-cert: secret "maas-api-e2e-worker-w6-51dd62-serving-cert" not found Oct 30 20:00:48.257684 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:00:48.257594 5088 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be2ffe6a-d355-4d95-a7ae-46adeb952736-maas-api-tls podName:be2ffe6a-d355-4d95-a7ae-46adeb952736 nodeName:}" failed. No retries permitted until 2026-10-30 20:00:48.757579277 +0000 UTC m=+810.170293034 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/be2ffe6a-d355-4d95-a7ae-46adeb952736-maas-api-tls") pod "maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957" (UID: "be2ffe6a-d355-4d95-a7ae-46adeb952736") : secret "maas-api-e2e-worker-w6-51dd62-serving-cert" not found Oct 30 20:00:48.258649 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:48.258625 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be2ffe6a-d355-4d95-a7ae-46adeb952736-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957\" (UID: \"be2ffe6a-d355-4d95-a7ae-46adeb952736\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957" Oct 30 20:00:48.260241 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:48.260218 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be2ffe6a-d355-4d95-a7ae-46adeb952736-metrics-certs\") pod \"maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957\" (UID: \"be2ffe6a-d355-4d95-a7ae-46adeb952736\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957" Oct 30 20:00:48.310684 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:48.310335 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmd6k\" (UniqueName: \"kubernetes.io/projected/be2ffe6a-d355-4d95-a7ae-46adeb952736-kube-api-access-dmd6k\") pod \"maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957\" (UID: \"be2ffe6a-d355-4d95-a7ae-46adeb952736\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957" Oct 30 20:00:48.336647 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:48.334834 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" Oct 30 20:00:48.336647 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:48.335084 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 20:00:48.336647 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:48.335149 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" podUID="f4b44c6e-891a-4073-9192-035aeaad10a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 20:00:48.467075 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:48.467047 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f10d2-58d98b4775-jq5gd" Oct 30 20:00:48.467243 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:48.467114 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-62f4a1-645b9d8589-k6rx9" Oct 30 20:00:48.762797 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:48.762756 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be2ffe6a-d355-4d95-a7ae-46adeb952736-maas-api-tls\") pod \"maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957\" (UID: \"be2ffe6a-d355-4d95-a7ae-46adeb952736\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957" Oct 30 20:00:48.765131 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:48.765100 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be2ffe6a-d355-4d95-a7ae-46adeb952736-maas-api-tls\") pod \"maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957\" (UID: \"be2ffe6a-d355-4d95-a7ae-46adeb952736\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957" Oct 30 20:00:49.049660 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:49.049554 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957" Oct 30 20:00:49.320227 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:49.320143 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 20:00:49.320227 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:49.320207 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" podUID="f4b44c6e-891a-4073-9192-035aeaad10a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 20:00:49.463805 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:49.463779 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957"] Oct 30 20:00:49.466027 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:00:49.465982 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe2ffe6a_d355_4d95_a7ae_46adeb952736.slice/crio-45fca52d914b3c98643cfa4ecdadc6d423319c7f201f72c2759a25d95984ec1c WatchSource:0}: Error finding container 45fca52d914b3c98643cfa4ecdadc6d423319c7f201f72c2759a25d95984ec1c: Status 404 returned error can't find the container with id 45fca52d914b3c98643cfa4ecdadc6d423319c7f201f72c2759a25d95984ec1c Oct 30 20:00:49.629737 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:49.629495 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957" event={"ID":"be2ffe6a-d355-4d95-a7ae-46adeb952736","Type":"ContainerStarted","Data":"45fca52d914b3c98643cfa4ecdadc6d423319c7f201f72c2759a25d95984ec1c"} Oct 30 20:00:50.320160 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:50.320112 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 20:00:50.320728 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:50.320180 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" podUID="f4b44c6e-891a-4073-9192-035aeaad10a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 20:00:51.320296 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:51.320261 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 20:00:51.320754 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:51.320328 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" podUID="f4b44c6e-891a-4073-9192-035aeaad10a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 20:00:51.652907 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:51.652814 5088 generic.go:356] "Generic (PLEG): container finished" podID="ce337cd9-bd40-432b-998d-4357f13b46d1" containerID="061917a119072909f856fe9e1d06cc2fc6221402faadd09ccfbf02ce0fff4c45" exitCode=0 Oct 30 20:00:51.652907 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:51.652886 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" event={"ID":"ce337cd9-bd40-432b-998d-4357f13b46d1","Type":"ContainerDied","Data":"061917a119072909f856fe9e1d06cc2fc6221402faadd09ccfbf02ce0fff4c45"} Oct 30 20:00:51.653341 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:51.653319 5088 scope.go:122] "RemoveContainer" containerID="061917a119072909f856fe9e1d06cc2fc6221402faadd09ccfbf02ce0fff4c45" Oct 30 20:00:51.654769 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:51.654739 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:00:51.654879 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:51.654824 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:00:51.654935 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:51.654903 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:00:52.141632 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:52.141584 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj"] Oct 30 20:00:52.150750 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:52.147970 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" Oct 30 20:00:52.166404 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:52.166371 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj"] Oct 30 20:00:52.174257 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:52.174232 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-626870\"/\"simulator-w5-626870-kserve-self-signed-certs\"" Oct 30 20:00:52.174406 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:52.174300 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-626870\"/\"kube-root-ca.crt\"" Oct 30 20:00:52.174406 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:52.174305 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-626870\"/\"openshift-service-ca.crt\"" Oct 30 20:00:52.174501 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:52.174401 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-626870\"/\"default-dockercfg-b6x4h\"" Oct 30 20:00:52.257369 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:52.257336 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d117d60-d768-4d67-bb48-cd0522fe2a57-dshm\") pod \"simulator-w5-626870-kserve-79f7696b7-gz9wj\" (UID: \"3d117d60-d768-4d67-bb48-cd0522fe2a57\") " pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" Oct 30 20:00:52.257369 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:52.257369 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d117d60-d768-4d67-bb48-cd0522fe2a57-model-cache\") pod \"simulator-w5-626870-kserve-79f7696b7-gz9wj\" (UID: \"3d117d60-d768-4d67-bb48-cd0522fe2a57\") " pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" Oct 30 20:00:52.257645 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:52.257467 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d117d60-d768-4d67-bb48-cd0522fe2a57-tmp-dir\") pod \"simulator-w5-626870-kserve-79f7696b7-gz9wj\" (UID: \"3d117d60-d768-4d67-bb48-cd0522fe2a57\") " pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" Oct 30 20:00:52.257645 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:52.257518 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d117d60-d768-4d67-bb48-cd0522fe2a57-home\") pod \"simulator-w5-626870-kserve-79f7696b7-gz9wj\" (UID: \"3d117d60-d768-4d67-bb48-cd0522fe2a57\") " pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" Oct 30 20:00:52.257645 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:52.257555 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8bhj\" (UniqueName: \"kubernetes.io/projected/3d117d60-d768-4d67-bb48-cd0522fe2a57-kube-api-access-g8bhj\") pod \"simulator-w5-626870-kserve-79f7696b7-gz9wj\" (UID: \"3d117d60-d768-4d67-bb48-cd0522fe2a57\") " pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" Oct 30 20:00:52.257872 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:52.257645 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d117d60-d768-4d67-bb48-cd0522fe2a57-tls-certs\") pod \"simulator-w5-626870-kserve-79f7696b7-gz9wj\" (UID: \"3d117d60-d768-4d67-bb48-cd0522fe2a57\") " pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" Oct 30 20:00:52.319994 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:52.319956 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 20:00:52.320157 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:52.320035 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" podUID="f4b44c6e-891a-4073-9192-035aeaad10a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 20:00:52.359174 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:52.359139 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d117d60-d768-4d67-bb48-cd0522fe2a57-dshm\") pod \"simulator-w5-626870-kserve-79f7696b7-gz9wj\" (UID: \"3d117d60-d768-4d67-bb48-cd0522fe2a57\") " pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" Oct 30 20:00:52.359748 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:52.359184 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d117d60-d768-4d67-bb48-cd0522fe2a57-model-cache\") pod \"simulator-w5-626870-kserve-79f7696b7-gz9wj\" (UID: \"3d117d60-d768-4d67-bb48-cd0522fe2a57\") " pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" Oct 30 20:00:52.359748 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:52.359234 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d117d60-d768-4d67-bb48-cd0522fe2a57-tmp-dir\") pod \"simulator-w5-626870-kserve-79f7696b7-gz9wj\" (UID: \"3d117d60-d768-4d67-bb48-cd0522fe2a57\") " pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" Oct 30 20:00:52.359748 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:52.359271 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d117d60-d768-4d67-bb48-cd0522fe2a57-home\") pod \"simulator-w5-626870-kserve-79f7696b7-gz9wj\" (UID: \"3d117d60-d768-4d67-bb48-cd0522fe2a57\") " pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" Oct 30 20:00:52.359748 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:52.359302 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8bhj\" (UniqueName: \"kubernetes.io/projected/3d117d60-d768-4d67-bb48-cd0522fe2a57-kube-api-access-g8bhj\") pod \"simulator-w5-626870-kserve-79f7696b7-gz9wj\" (UID: \"3d117d60-d768-4d67-bb48-cd0522fe2a57\") " pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" Oct 30 20:00:52.359748 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:52.359332 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d117d60-d768-4d67-bb48-cd0522fe2a57-tls-certs\") pod \"simulator-w5-626870-kserve-79f7696b7-gz9wj\" (UID: \"3d117d60-d768-4d67-bb48-cd0522fe2a57\") " pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" Oct 30 20:00:52.359748 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:52.359652 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d117d60-d768-4d67-bb48-cd0522fe2a57-model-cache\") pod \"simulator-w5-626870-kserve-79f7696b7-gz9wj\" (UID: \"3d117d60-d768-4d67-bb48-cd0522fe2a57\") " pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" Oct 30 20:00:52.360254 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:52.359902 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d117d60-d768-4d67-bb48-cd0522fe2a57-home\") pod \"simulator-w5-626870-kserve-79f7696b7-gz9wj\" (UID: \"3d117d60-d768-4d67-bb48-cd0522fe2a57\") " pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" Oct 30 20:00:52.368475 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:52.367693 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d117d60-d768-4d67-bb48-cd0522fe2a57-dshm\") pod \"simulator-w5-626870-kserve-79f7696b7-gz9wj\" (UID: \"3d117d60-d768-4d67-bb48-cd0522fe2a57\") " pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" Oct 30 20:00:52.368475 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:52.368422 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d117d60-d768-4d67-bb48-cd0522fe2a57-tmp-dir\") pod \"simulator-w5-626870-kserve-79f7696b7-gz9wj\" (UID: \"3d117d60-d768-4d67-bb48-cd0522fe2a57\") " pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" Oct 30 20:00:52.417690 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:52.405030 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d117d60-d768-4d67-bb48-cd0522fe2a57-tls-certs\") pod \"simulator-w5-626870-kserve-79f7696b7-gz9wj\" (UID: \"3d117d60-d768-4d67-bb48-cd0522fe2a57\") " pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" Oct 30 20:00:52.424785 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:52.424715 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8bhj\" (UniqueName: \"kubernetes.io/projected/3d117d60-d768-4d67-bb48-cd0522fe2a57-kube-api-access-g8bhj\") pod \"simulator-w5-626870-kserve-79f7696b7-gz9wj\" (UID: \"3d117d60-d768-4d67-bb48-cd0522fe2a57\") " pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" Oct 30 20:00:52.471686 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:52.471644 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" Oct 30 20:00:52.478791 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:00:52.478763 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:00:52.537137 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:52.537103 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-51dd62-5668978b9c-5djrs" Oct 30 20:00:52.538585 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:52.538561 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-51dd62-79f5b56fd7-ltqtf" Oct 30 20:00:52.664272 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:52.664237 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" event={"ID":"ce337cd9-bd40-432b-998d-4357f13b46d1","Type":"ContainerStarted","Data":"e6aa99641e4b078628025af63172e36704b9d007e3d6997c78f025449ecab0cb"} Oct 30 20:00:52.836075 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:52.835922 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj"] Oct 30 20:00:52.839895 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:00:52.839800 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d117d60_d768_4d67_bb48_cd0522fe2a57.slice/crio-30f62f5e367eb5f09c24c6fbbc93375860604d5ab1ff77fca155fafc6ca299e2 WatchSource:0}: Error finding container 30f62f5e367eb5f09c24c6fbbc93375860604d5ab1ff77fca155fafc6ca299e2: Status 404 returned error can't find the container with id 30f62f5e367eb5f09c24c6fbbc93375860604d5ab1ff77fca155fafc6ca299e2 Oct 30 20:00:53.305636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:53.301770 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:00:53.362599 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:53.362569 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" Oct 30 20:00:53.363024 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:53.362931 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" Oct 30 20:00:53.364067 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:53.364046 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" Oct 30 20:00:53.457650 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:53.454082 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" Oct 30 20:00:53.457650 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:53.455572 5088 patch_prober.go:29] interesting pod/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 20:00:53.457650 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:53.455651 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" podUID="ce337cd9-bd40-432b-998d-4357f13b46d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 20:00:53.671202 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:53.671164 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" event={"ID":"3d117d60-d768-4d67-bb48-cd0522fe2a57","Type":"ContainerStarted","Data":"75803e24c40db568bfb7548584770fa5234fa42ec240f96e5397c06422744f0e"} Oct 30 20:00:53.671202 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:53.671208 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" event={"ID":"3d117d60-d768-4d67-bb48-cd0522fe2a57","Type":"ContainerStarted","Data":"30f62f5e367eb5f09c24c6fbbc93375860604d5ab1ff77fca155fafc6ca299e2"} Oct 30 20:00:53.720719 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:53.720658 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podStartSLOduration=1.7206412150000001 podStartE2EDuration="1.720641215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:53.7193161 +0000 UTC m=+815.132029891" watchObservedRunningTime="2026-10-30 20:00:53.720641215 +0000 UTC m=+815.133354996" Oct 30 20:00:54.027179 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.027140 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl"] Oct 30 20:00:54.031643 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.031598 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" Oct 30 20:00:54.053560 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.053528 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-f1f016-openshift-default-dockercfg-pgfjl\"" Oct 30 20:00:54.063531 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.063503 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl"] Oct 30 20:00:54.115986 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.115941 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-workload-certs\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl\" (UID: \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" Oct 30 20:00:54.116200 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.115998 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmt7j\" (UniqueName: \"kubernetes.io/projected/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-kube-api-access-zmt7j\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl\" (UID: \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" Oct 30 20:00:54.116200 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.116066 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-workload-socket\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl\" (UID: \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" Oct 30 20:00:54.116200 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.116156 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-istio-token\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl\" (UID: \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" Oct 30 20:00:54.116388 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.116223 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-istio-envoy\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl\" (UID: \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" Oct 30 20:00:54.116388 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.116286 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-istiod-ca-cert\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl\" (UID: \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" Oct 30 20:00:54.116388 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.116315 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-credential-socket\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl\" (UID: \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" Oct 30 20:00:54.116388 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.116356 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-istio-podinfo\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl\" (UID: \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" Oct 30 20:00:54.116600 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.116394 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-istio-data\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl\" (UID: \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" Oct 30 20:00:54.226638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.220286 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-istio-podinfo\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl\" (UID: \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" Oct 30 20:00:54.226638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.220354 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-istio-data\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl\" (UID: \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" Oct 30 20:00:54.226638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.220423 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-workload-certs\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl\" (UID: \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" Oct 30 20:00:54.226638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.220449 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmt7j\" (UniqueName: \"kubernetes.io/projected/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-kube-api-access-zmt7j\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl\" (UID: \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" Oct 30 20:00:54.226638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.220514 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-workload-socket\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl\" (UID: \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" Oct 30 20:00:54.226638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.220598 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-istio-token\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl\" (UID: \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" Oct 30 20:00:54.226638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.220687 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-istio-envoy\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl\" (UID: \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" Oct 30 20:00:54.226638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.220755 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-istiod-ca-cert\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl\" (UID: \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" Oct 30 20:00:54.226638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.220784 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-credential-socket\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl\" (UID: \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" Oct 30 20:00:54.226638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.221291 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-credential-socket\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl\" (UID: \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" Oct 30 20:00:54.226638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.221991 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-workload-socket\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl\" (UID: \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" Oct 30 20:00:54.226638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.222607 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-istiod-ca-cert\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl\" (UID: \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" Oct 30 20:00:54.226638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.222946 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-workload-certs\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl\" (UID: \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" Oct 30 20:00:54.226638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.223538 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-istio-data\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl\" (UID: \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" Oct 30 20:00:54.238642 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.236047 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-istio-envoy\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl\" (UID: \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" Oct 30 20:00:54.242648 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.242580 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-istio-podinfo\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl\" (UID: \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" Oct 30 20:00:54.252549 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.252487 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmt7j\" (UniqueName: \"kubernetes.io/projected/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-kube-api-access-zmt7j\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl\" (UID: \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" Oct 30 20:00:54.253651 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.253629 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-istio-token\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl\" (UID: \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" Oct 30 20:00:54.346048 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.345965 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" Oct 30 20:00:54.453833 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.453799 5088 patch_prober.go:29] interesting pod/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 20:00:54.454189 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.453862 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" podUID="ce337cd9-bd40-432b-998d-4357f13b46d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 20:00:54.472664 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.472632 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" Oct 30 20:00:54.679051 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.678964 5088 generic.go:356] "Generic (PLEG): container finished" podID="05a46855-a143-4c6d-8c87-404dc271bdb3" containerID="ab404a6d0bb243a2362e993df58be2d5b55d6c8958bcc2f6e364ccc15da2c8f7" exitCode=0 Oct 30 20:00:54.679051 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.679037 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" event={"ID":"05a46855-a143-4c6d-8c87-404dc271bdb3","Type":"ContainerDied","Data":"ab404a6d0bb243a2362e993df58be2d5b55d6c8958bcc2f6e364ccc15da2c8f7"} Oct 30 20:00:54.679525 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.679501 5088 scope.go:122] "RemoveContainer" containerID="ab404a6d0bb243a2362e993df58be2d5b55d6c8958bcc2f6e364ccc15da2c8f7" Oct 30 20:00:54.681204 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.681171 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:00:54.681308 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.681274 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:00:54.681362 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.681289 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957" event={"ID":"be2ffe6a-d355-4d95-a7ae-46adeb952736","Type":"ContainerStarted","Data":"f72efc6e822122441efed9c58d4563ca1068b6453cbace028b5bf3b95a20c799"} Oct 30 20:00:54.681362 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.681326 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:00:54.681594 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.681579 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957" Oct 30 20:00:54.713694 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.713656 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl"] Oct 30 20:00:54.716530 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:00:54.716499 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4789c6d9_e328_4890_a2b3_a2fdbfdb04b9.slice/crio-04f356b1c755e58eef8f654d2c69f81d87eb02bab82174166836131fa77ad2d1 WatchSource:0}: Error finding container 04f356b1c755e58eef8f654d2c69f81d87eb02bab82174166836131fa77ad2d1: Status 404 returned error can't find the container with id 04f356b1c755e58eef8f654d2c69f81d87eb02bab82174166836131fa77ad2d1 Oct 30 20:00:54.719546 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.719508 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:00:54.719700 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.719626 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:00:54.719700 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.719687 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:00:54.809641 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:54.802480 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957" podStartSLOduration=3.577926433 podStartE2EDuration="7.802459209s" totalImagesPullingTime="4.224532776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:54.801431119 +0000 UTC m=+816.214144927" watchObservedRunningTime="2026-10-30 20:00:54.802459209 +0000 UTC m=+816.215172991" Oct 30 20:00:55.454245 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:55.454201 5088 patch_prober.go:29] interesting pod/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 20:00:55.454684 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:55.454261 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" podUID="ce337cd9-bd40-432b-998d-4357f13b46d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 20:00:55.719698 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:55.719083 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" event={"ID":"05a46855-a143-4c6d-8c87-404dc271bdb3","Type":"ContainerStarted","Data":"e2852ce9442a037ded3c3a1a2d61a71a7261186d4e0bd39c8dbc326bacd5a9b7"} Oct 30 20:00:55.727858 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:55.727819 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" event={"ID":"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9","Type":"ContainerStarted","Data":"8df290432378123b70d4427acfe82432d938ea3a85aa1b705c55e635cde745c5"} Oct 30 20:00:55.728144 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:55.728121 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" event={"ID":"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9","Type":"ContainerStarted","Data":"04f356b1c755e58eef8f654d2c69f81d87eb02bab82174166836131fa77ad2d1"} Oct 30 20:00:55.906113 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:55.906077 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" Oct 30 20:00:55.906537 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:55.906513 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 20:00:55.906731 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:55.906706 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" podUID="05a46855-a143-4c6d-8c87-404dc271bdb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 20:00:56.348727 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:56.347217 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" Oct 30 20:00:56.361457 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:56.361091 5088 patch_prober.go:29] interesting pod/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 30 20:00:56.361457 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:56.361158 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" podUID="4789c6d9-e328-4890-a2b3-a2fdbfdb04b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 30 20:00:56.361743 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:56.361516 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" Oct 30 20:00:56.456543 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:56.456042 5088 patch_prober.go:29] interesting pod/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 20:00:56.456543 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:56.456114 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" podUID="ce337cd9-bd40-432b-998d-4357f13b46d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 20:00:56.901642 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:56.899759 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 20:00:56.901642 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:56.899823 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" podUID="05a46855-a143-4c6d-8c87-404dc271bdb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 20:00:57.352639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:57.352090 5088 patch_prober.go:29] interesting pod/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 30 20:00:57.352639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:57.352164 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" podUID="4789c6d9-e328-4890-a2b3-a2fdbfdb04b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 30 20:00:57.372636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:57.362032 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" Oct 30 20:00:57.457638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:57.457541 5088 patch_prober.go:29] interesting pod/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 20:00:57.457638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:57.457606 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" podUID="ce337cd9-bd40-432b-998d-4357f13b46d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 20:00:57.910638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:57.900729 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 20:00:57.910638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:57.900810 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" podUID="05a46855-a143-4c6d-8c87-404dc271bdb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 20:00:58.349193 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:58.349155 5088 patch_prober.go:29] interesting pod/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 30 20:00:58.349511 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:58.349481 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" podUID="4789c6d9-e328-4890-a2b3-a2fdbfdb04b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 30 20:00:58.466022 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:58.465985 5088 patch_prober.go:29] interesting pod/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 20:00:58.466701 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:58.466598 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" podUID="ce337cd9-bd40-432b-998d-4357f13b46d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 20:00:58.901643 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:58.900157 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 20:00:58.901643 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:58.900238 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" podUID="05a46855-a143-4c6d-8c87-404dc271bdb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 20:00:59.367636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:59.364228 5088 patch_prober.go:29] interesting pod/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 30 20:00:59.367636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:59.364309 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" podUID="4789c6d9-e328-4890-a2b3-a2fdbfdb04b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 30 20:00:59.546640 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:59.542968 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" Oct 30 20:00:59.583041 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:59.583006 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" Oct 30 20:00:59.659064 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:59.658931 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" podStartSLOduration=6.65890887 podStartE2EDuration="6.65890887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:55.847054639 +0000 UTC m=+817.259768423" watchObservedRunningTime="2026-10-30 20:00:59.65890887 +0000 UTC m=+821.071622652" Oct 30 20:00:59.904638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:59.902146 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 20:00:59.904638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:00:59.902223 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" podUID="05a46855-a143-4c6d-8c87-404dc271bdb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 20:01:00.350842 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:00.350800 5088 patch_prober.go:29] interesting pod/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 30 20:01:00.351141 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:00.351108 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" podUID="4789c6d9-e328-4890-a2b3-a2fdbfdb04b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 30 20:01:00.426636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:00.425594 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:01:00.764353 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:00.764318 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957" Oct 30 20:01:00.905648 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:00.898965 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 20:01:00.905931 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:00.905895 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" podUID="05a46855-a143-4c6d-8c87-404dc271bdb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 20:01:01.350012 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:01.349975 5088 patch_prober.go:29] interesting pod/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 30 20:01:01.350288 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:01.350260 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" podUID="4789c6d9-e328-4890-a2b3-a2fdbfdb04b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 30 20:01:01.898984 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:01.898952 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 20:01:01.899397 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:01.899008 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" podUID="05a46855-a143-4c6d-8c87-404dc271bdb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 20:01:02.351128 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:02.351098 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" Oct 30 20:01:02.351540 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:02.351509 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" Oct 30 20:01:02.352151 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:02.352135 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" Oct 30 20:01:02.474792 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:02.472895 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" Oct 30 20:01:02.494394 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:02.494097 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" Oct 30 20:01:02.908727 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:02.908691 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" Oct 30 20:01:02.910013 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:02.909989 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" Oct 30 20:01:02.913724 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:02.913701 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" Oct 30 20:01:05.921561 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:05.921527 5088 generic.go:356] "Generic (PLEG): container finished" podID="c0748781-63b9-4af1-8eab-5d0cef276da6" containerID="a9bd049c96e0f99442c1823a643467f15e55bf80eb26fd607ffad0a2abee78bf" exitCode=0 Oct 30 20:01:05.922056 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:05.921599 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" event={"ID":"c0748781-63b9-4af1-8eab-5d0cef276da6","Type":"ContainerDied","Data":"a9bd049c96e0f99442c1823a643467f15e55bf80eb26fd607ffad0a2abee78bf"} Oct 30 20:01:05.922056 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:05.921998 5088 scope.go:122] "RemoveContainer" containerID="a9bd049c96e0f99442c1823a643467f15e55bf80eb26fd607ffad0a2abee78bf" Oct 30 20:01:05.923366 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:05.923335 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:01:05.923478 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:05.923424 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:01:05.923543 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:05.923474 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:01:06.944502 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:06.944471 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" event={"ID":"c0748781-63b9-4af1-8eab-5d0cef276da6","Type":"ContainerStarted","Data":"cf771118fedc5389d16f43ca5d7c4e0f0b24d2193c33d1c77df884e756cd9c0f"} Oct 30 20:01:07.351930 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:07.351893 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" Oct 30 20:01:07.352159 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:07.352135 5088 patch_prober.go:29] interesting pod/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 20:01:07.352218 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:07.352186 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" podUID="c0748781-63b9-4af1-8eab-5d0cef276da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 20:01:08.352022 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:08.351985 5088 patch_prober.go:29] interesting pod/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 20:01:08.352443 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:08.352057 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" podUID="c0748781-63b9-4af1-8eab-5d0cef276da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 20:01:09.352258 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:09.352226 5088 patch_prober.go:29] interesting pod/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 20:01:09.352675 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:09.352288 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" podUID="c0748781-63b9-4af1-8eab-5d0cef276da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 20:01:10.352328 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:10.352284 5088 patch_prober.go:29] interesting pod/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 20:01:10.352810 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:10.352346 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" podUID="c0748781-63b9-4af1-8eab-5d0cef276da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 20:01:11.351946 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:11.351909 5088 patch_prober.go:29] interesting pod/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 20:01:11.352136 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:11.351975 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" podUID="c0748781-63b9-4af1-8eab-5d0cef276da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 20:01:12.362241 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:12.362199 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" Oct 30 20:01:12.472204 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:12.472168 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" Oct 30 20:01:12.607778 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:12.607740 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq"] Oct 30 20:01:12.613292 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:12.613213 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:01:12.634247 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:12.634206 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq"] Oct 30 20:01:12.636213 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:12.636187 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-62f4a1\"/\"default-dockercfg-pv79q\"" Oct 30 20:01:12.636369 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:12.636230 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-62f4a1\"/\"openshift-service-ca.crt\"" Oct 30 20:01:12.636369 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:12.636249 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-62f4a1\"/\"kube-root-ca.crt\"" Oct 30 20:01:12.636529 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:12.636375 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-62f4a1\"/\"simulator-w0-62f4a1-kserve-self-signed-certs\"" Oct 30 20:01:12.734602 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:12.734559 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21-dshm\") pod \"simulator-w0-62f4a1-kserve-7df66b8675-wmkxq\" (UID: \"c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21\") " pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:01:12.734819 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:12.734642 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21-tls-certs\") pod \"simulator-w0-62f4a1-kserve-7df66b8675-wmkxq\" (UID: \"c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21\") " pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:01:12.734819 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:12.734761 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21-tmp-dir\") pod \"simulator-w0-62f4a1-kserve-7df66b8675-wmkxq\" (UID: \"c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21\") " pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:01:12.734819 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:12.734815 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21-home\") pod \"simulator-w0-62f4a1-kserve-7df66b8675-wmkxq\" (UID: \"c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21\") " pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:01:12.734959 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:12.734877 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21-model-cache\") pod \"simulator-w0-62f4a1-kserve-7df66b8675-wmkxq\" (UID: \"c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21\") " pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:01:12.734959 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:12.734941 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qqfn\" (UniqueName: \"kubernetes.io/projected/c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21-kube-api-access-4qqfn\") pod \"simulator-w0-62f4a1-kserve-7df66b8675-wmkxq\" (UID: \"c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21\") " pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:01:12.836012 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:12.835969 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21-dshm\") pod \"simulator-w0-62f4a1-kserve-7df66b8675-wmkxq\" (UID: \"c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21\") " pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:01:12.836199 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:12.836018 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21-tls-certs\") pod \"simulator-w0-62f4a1-kserve-7df66b8675-wmkxq\" (UID: \"c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21\") " pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:01:12.836199 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:12.836089 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21-tmp-dir\") pod \"simulator-w0-62f4a1-kserve-7df66b8675-wmkxq\" (UID: \"c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21\") " pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:01:12.836199 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:12.836146 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21-home\") pod \"simulator-w0-62f4a1-kserve-7df66b8675-wmkxq\" (UID: \"c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21\") " pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:01:12.836199 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:12.836178 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21-model-cache\") pod \"simulator-w0-62f4a1-kserve-7df66b8675-wmkxq\" (UID: \"c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21\") " pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:01:12.836401 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:12.836277 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qqfn\" (UniqueName: \"kubernetes.io/projected/c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21-kube-api-access-4qqfn\") pod \"simulator-w0-62f4a1-kserve-7df66b8675-wmkxq\" (UID: \"c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21\") " pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:01:12.836550 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:12.836516 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21-tmp-dir\") pod \"simulator-w0-62f4a1-kserve-7df66b8675-wmkxq\" (UID: \"c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21\") " pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:01:12.836713 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:12.836563 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21-model-cache\") pod \"simulator-w0-62f4a1-kserve-7df66b8675-wmkxq\" (UID: \"c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21\") " pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:01:12.836713 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:12.836641 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21-home\") pod \"simulator-w0-62f4a1-kserve-7df66b8675-wmkxq\" (UID: \"c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21\") " pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:01:12.838470 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:12.838448 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21-dshm\") pod \"simulator-w0-62f4a1-kserve-7df66b8675-wmkxq\" (UID: \"c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21\") " pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:01:12.838561 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:12.838544 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21-tls-certs\") pod \"simulator-w0-62f4a1-kserve-7df66b8675-wmkxq\" (UID: \"c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21\") " pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:01:12.880075 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:12.879960 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qqfn\" (UniqueName: \"kubernetes.io/projected/c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21-kube-api-access-4qqfn\") pod \"simulator-w0-62f4a1-kserve-7df66b8675-wmkxq\" (UID: \"c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21\") " pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:01:12.926737 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:12.926703 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:01:12.983826 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:12.983741 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" Oct 30 20:01:13.170981 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:13.170896 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:01:13.245033 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:13.244993 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq"] Oct 30 20:01:13.245464 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:01:13.245439 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6cf0cd1_3f5f_4e72_b1cd_f9194e7dac21.slice/crio-5495d66b685d82a2e53bdc781873b30d6823504c503d3e26f1fd7691c3119811 WatchSource:0}: Error finding container 5495d66b685d82a2e53bdc781873b30d6823504c503d3e26f1fd7691c3119811: Status 404 returned error can't find the container with id 5495d66b685d82a2e53bdc781873b30d6823504c503d3e26f1fd7691c3119811 Oct 30 20:01:13.990356 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:13.990318 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" event={"ID":"c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21","Type":"ContainerStarted","Data":"108f700feb46b39497c4425638937653dc8316b3ea9c281460a178cf185138d2"} Oct 30 20:01:13.990913 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:13.990368 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" event={"ID":"c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21","Type":"ContainerStarted","Data":"5495d66b685d82a2e53bdc781873b30d6823504c503d3e26f1fd7691c3119811"} Oct 30 20:01:14.050992 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:14.050908 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podStartSLOduration=2.050889582 podStartE2EDuration="2.050889582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:14.046299723 +0000 UTC m=+835.459013527" watchObservedRunningTime="2026-10-30 20:01:14.050889582 +0000 UTC m=+835.463603363" Oct 30 20:01:14.927519 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:14.927480 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:01:22.437764 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:01:22.437730 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:01:22.927707 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:22.927677 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:01:22.940398 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:22.940373 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:01:23.051341 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:23.051310 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:01:23.785385 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:23.785356 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:01:37.783353 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:37.783274 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:01:39.189926 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:39.189894 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7"] Oct 30 20:01:39.194277 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:39.194252 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" Oct 30 20:01:39.201287 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:39.201258 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czgtn\" (UniqueName: \"kubernetes.io/projected/f2757424-fb74-48ff-82c3-ccb79a27fd2a-kube-api-access-czgtn\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7\" (UID: \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" Oct 30 20:01:39.201429 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:39.201301 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2757424-fb74-48ff-82c3-ccb79a27fd2a-credential-socket\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7\" (UID: \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" Oct 30 20:01:39.201429 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:39.201423 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2757424-fb74-48ff-82c3-ccb79a27fd2a-istio-podinfo\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7\" (UID: \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" Oct 30 20:01:39.201546 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:39.201441 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2757424-fb74-48ff-82c3-ccb79a27fd2a-istio-token\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7\" (UID: \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" Oct 30 20:01:39.201546 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:39.201464 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2757424-fb74-48ff-82c3-ccb79a27fd2a-istio-envoy\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7\" (UID: \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" Oct 30 20:01:39.201546 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:39.201480 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2757424-fb74-48ff-82c3-ccb79a27fd2a-istiod-ca-cert\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7\" (UID: \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" Oct 30 20:01:39.201729 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:39.201579 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2757424-fb74-48ff-82c3-ccb79a27fd2a-workload-certs\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7\" (UID: \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" Oct 30 20:01:39.201729 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:39.201643 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2757424-fb74-48ff-82c3-ccb79a27fd2a-istio-data\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7\" (UID: \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" Oct 30 20:01:39.201836 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:39.201750 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2757424-fb74-48ff-82c3-ccb79a27fd2a-workload-socket\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7\" (UID: \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" Oct 30 20:01:39.218032 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:39.217996 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7"] Oct 30 20:01:39.302585 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:39.302548 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2757424-fb74-48ff-82c3-ccb79a27fd2a-workload-socket\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7\" (UID: \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" Oct 30 20:01:39.302811 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:39.302634 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czgtn\" (UniqueName: \"kubernetes.io/projected/f2757424-fb74-48ff-82c3-ccb79a27fd2a-kube-api-access-czgtn\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7\" (UID: \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" Oct 30 20:01:39.302811 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:39.302655 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2757424-fb74-48ff-82c3-ccb79a27fd2a-credential-socket\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7\" (UID: \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" Oct 30 20:01:39.302943 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:39.302815 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2757424-fb74-48ff-82c3-ccb79a27fd2a-istio-podinfo\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7\" (UID: \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" Oct 30 20:01:39.302943 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:39.302850 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2757424-fb74-48ff-82c3-ccb79a27fd2a-istio-token\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7\" (UID: \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" Oct 30 20:01:39.302943 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:39.302893 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2757424-fb74-48ff-82c3-ccb79a27fd2a-istio-envoy\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7\" (UID: \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" Oct 30 20:01:39.302943 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:39.302915 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2757424-fb74-48ff-82c3-ccb79a27fd2a-istiod-ca-cert\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7\" (UID: \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" Oct 30 20:01:39.303152 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:39.302944 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2757424-fb74-48ff-82c3-ccb79a27fd2a-workload-socket\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7\" (UID: \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" Oct 30 20:01:39.303152 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:39.302958 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2757424-fb74-48ff-82c3-ccb79a27fd2a-workload-certs\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7\" (UID: \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" Oct 30 20:01:39.303152 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:39.302990 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2757424-fb74-48ff-82c3-ccb79a27fd2a-credential-socket\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7\" (UID: \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" Oct 30 20:01:39.303152 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:39.303006 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2757424-fb74-48ff-82c3-ccb79a27fd2a-istio-data\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7\" (UID: \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" Oct 30 20:01:39.303367 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:39.303243 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2757424-fb74-48ff-82c3-ccb79a27fd2a-workload-certs\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7\" (UID: \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" Oct 30 20:01:39.303367 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:39.303308 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2757424-fb74-48ff-82c3-ccb79a27fd2a-istio-data\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7\" (UID: \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" Oct 30 20:01:39.303671 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:39.303650 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2757424-fb74-48ff-82c3-ccb79a27fd2a-istiod-ca-cert\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7\" (UID: \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" Oct 30 20:01:39.305320 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:39.305299 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2757424-fb74-48ff-82c3-ccb79a27fd2a-istio-envoy\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7\" (UID: \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" Oct 30 20:01:39.305406 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:39.305384 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2757424-fb74-48ff-82c3-ccb79a27fd2a-istio-podinfo\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7\" (UID: \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" Oct 30 20:01:39.329962 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:39.329935 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2757424-fb74-48ff-82c3-ccb79a27fd2a-istio-token\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7\" (UID: \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" Oct 30 20:01:39.330213 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:39.330192 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czgtn\" (UniqueName: \"kubernetes.io/projected/f2757424-fb74-48ff-82c3-ccb79a27fd2a-kube-api-access-czgtn\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7\" (UID: \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" Oct 30 20:01:39.511148 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:39.511116 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" Oct 30 20:01:39.850856 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:01:39.850822 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2757424_fb74_48ff_82c3_ccb79a27fd2a.slice/crio-2827f0489f7560b8277adb716f5e5bb96d0ab022bf592552859071d64fd97b02 WatchSource:0}: Error finding container 2827f0489f7560b8277adb716f5e5bb96d0ab022bf592552859071d64fd97b02: Status 404 returned error can't find the container with id 2827f0489f7560b8277adb716f5e5bb96d0ab022bf592552859071d64fd97b02 Oct 30 20:01:39.852206 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:39.852182 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7"] Oct 30 20:01:39.853295 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:39.853263 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:01:39.853398 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:39.853338 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:01:39.853398 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:39.853381 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:01:40.147035 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:40.146949 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" event={"ID":"f2757424-fb74-48ff-82c3-ccb79a27fd2a","Type":"ContainerStarted","Data":"15c727db7065a1567294b0f8b8556b22587ec02b41504624305b564a569fbb00"} Oct 30 20:01:40.147035 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:40.147000 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" event={"ID":"f2757424-fb74-48ff-82c3-ccb79a27fd2a","Type":"ContainerStarted","Data":"2827f0489f7560b8277adb716f5e5bb96d0ab022bf592552859071d64fd97b02"} Oct 30 20:01:40.203314 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:40.203251 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" podStartSLOduration=1.203237869 podStartE2EDuration="1.203237869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:40.201760435 +0000 UTC m=+861.614474233" watchObservedRunningTime="2026-10-30 20:01:40.203237869 +0000 UTC m=+861.615951648" Oct 30 20:01:40.512136 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:40.512096 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" Oct 30 20:01:40.514017 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:40.513992 5088 patch_prober.go:29] interesting pod/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 20:01:40.514133 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:40.514043 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" podUID="f2757424-fb74-48ff-82c3-ccb79a27fd2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 20:01:41.512210 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:41.512169 5088 patch_prober.go:29] interesting pod/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 20:01:41.512601 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:41.512251 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" podUID="f2757424-fb74-48ff-82c3-ccb79a27fd2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 20:01:42.511636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:42.511515 5088 patch_prober.go:29] interesting pod/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 20:01:42.511636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:42.511592 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" podUID="f2757424-fb74-48ff-82c3-ccb79a27fd2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 20:01:43.512437 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:43.512399 5088 patch_prober.go:29] interesting pod/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 20:01:43.512836 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:43.512492 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" podUID="f2757424-fb74-48ff-82c3-ccb79a27fd2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 20:01:44.512331 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:44.512088 5088 patch_prober.go:29] interesting pod/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 20:01:44.512331 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:44.512155 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" podUID="f2757424-fb74-48ff-82c3-ccb79a27fd2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 20:01:45.511624 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:45.511573 5088 patch_prober.go:29] interesting pod/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 20:01:45.511807 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:45.511661 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" podUID="f2757424-fb74-48ff-82c3-ccb79a27fd2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 20:01:46.516548 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:46.516518 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" Oct 30 20:01:46.517000 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:46.516939 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" Oct 30 20:01:46.517698 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:46.517682 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" Oct 30 20:01:50.287685 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:50.287642 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r"] Oct 30 20:01:50.293152 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:50.293127 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" Oct 30 20:01:50.324439 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:50.324409 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-912f24\"/\"openshift-service-ca.crt\"" Oct 30 20:01:50.324439 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:50.324422 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-912f24\"/\"premium-w3-912f24-kserve-self-signed-certs\"" Oct 30 20:01:50.324659 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:50.324480 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-912f24\"/\"default-dockercfg-hw2ql\"" Oct 30 20:01:50.326683 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:50.326660 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r"] Oct 30 20:01:50.327386 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:50.327369 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-912f24\"/\"kube-root-ca.crt\"" Oct 30 20:01:50.332860 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:50.332840 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8d25fcd-f343-445b-ac68-b2248f619ee4-model-cache\") pod \"premium-w3-912f24-kserve-56d6b776d7-mbb7r\" (UID: \"f8d25fcd-f343-445b-ac68-b2248f619ee4\") " pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" Oct 30 20:01:50.332975 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:50.332954 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wsxf\" (UniqueName: \"kubernetes.io/projected/f8d25fcd-f343-445b-ac68-b2248f619ee4-kube-api-access-4wsxf\") pod \"premium-w3-912f24-kserve-56d6b776d7-mbb7r\" (UID: \"f8d25fcd-f343-445b-ac68-b2248f619ee4\") " pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" Oct 30 20:01:50.333057 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:50.333026 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8d25fcd-f343-445b-ac68-b2248f619ee4-tmp-dir\") pod \"premium-w3-912f24-kserve-56d6b776d7-mbb7r\" (UID: \"f8d25fcd-f343-445b-ac68-b2248f619ee4\") " pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" Oct 30 20:01:50.333120 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:50.333057 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8d25fcd-f343-445b-ac68-b2248f619ee4-dshm\") pod \"premium-w3-912f24-kserve-56d6b776d7-mbb7r\" (UID: \"f8d25fcd-f343-445b-ac68-b2248f619ee4\") " pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" Oct 30 20:01:50.333177 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:50.333163 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8d25fcd-f343-445b-ac68-b2248f619ee4-tls-certs\") pod \"premium-w3-912f24-kserve-56d6b776d7-mbb7r\" (UID: \"f8d25fcd-f343-445b-ac68-b2248f619ee4\") " pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" Oct 30 20:01:50.333237 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:50.333195 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8d25fcd-f343-445b-ac68-b2248f619ee4-home\") pod \"premium-w3-912f24-kserve-56d6b776d7-mbb7r\" (UID: \"f8d25fcd-f343-445b-ac68-b2248f619ee4\") " pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" Oct 30 20:01:50.434104 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:50.434074 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8d25fcd-f343-445b-ac68-b2248f619ee4-home\") pod \"premium-w3-912f24-kserve-56d6b776d7-mbb7r\" (UID: \"f8d25fcd-f343-445b-ac68-b2248f619ee4\") " pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" Oct 30 20:01:50.434275 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:50.434165 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8d25fcd-f343-445b-ac68-b2248f619ee4-model-cache\") pod \"premium-w3-912f24-kserve-56d6b776d7-mbb7r\" (UID: \"f8d25fcd-f343-445b-ac68-b2248f619ee4\") " pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" Oct 30 20:01:50.434275 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:50.434257 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wsxf\" (UniqueName: \"kubernetes.io/projected/f8d25fcd-f343-445b-ac68-b2248f619ee4-kube-api-access-4wsxf\") pod \"premium-w3-912f24-kserve-56d6b776d7-mbb7r\" (UID: \"f8d25fcd-f343-445b-ac68-b2248f619ee4\") " pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" Oct 30 20:01:50.434386 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:50.434325 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8d25fcd-f343-445b-ac68-b2248f619ee4-tmp-dir\") pod \"premium-w3-912f24-kserve-56d6b776d7-mbb7r\" (UID: \"f8d25fcd-f343-445b-ac68-b2248f619ee4\") " pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" Oct 30 20:01:50.434386 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:50.434361 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8d25fcd-f343-445b-ac68-b2248f619ee4-dshm\") pod \"premium-w3-912f24-kserve-56d6b776d7-mbb7r\" (UID: \"f8d25fcd-f343-445b-ac68-b2248f619ee4\") " pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" Oct 30 20:01:50.434481 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:50.434400 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8d25fcd-f343-445b-ac68-b2248f619ee4-tls-certs\") pod \"premium-w3-912f24-kserve-56d6b776d7-mbb7r\" (UID: \"f8d25fcd-f343-445b-ac68-b2248f619ee4\") " pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" Oct 30 20:01:50.434555 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:50.434533 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8d25fcd-f343-445b-ac68-b2248f619ee4-home\") pod \"premium-w3-912f24-kserve-56d6b776d7-mbb7r\" (UID: \"f8d25fcd-f343-445b-ac68-b2248f619ee4\") " pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" Oct 30 20:01:50.434636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:50.434582 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8d25fcd-f343-445b-ac68-b2248f619ee4-model-cache\") pod \"premium-w3-912f24-kserve-56d6b776d7-mbb7r\" (UID: \"f8d25fcd-f343-445b-ac68-b2248f619ee4\") " pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" Oct 30 20:01:50.434724 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:50.434707 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8d25fcd-f343-445b-ac68-b2248f619ee4-tmp-dir\") pod \"premium-w3-912f24-kserve-56d6b776d7-mbb7r\" (UID: \"f8d25fcd-f343-445b-ac68-b2248f619ee4\") " pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" Oct 30 20:01:50.436595 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:50.436576 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8d25fcd-f343-445b-ac68-b2248f619ee4-dshm\") pod \"premium-w3-912f24-kserve-56d6b776d7-mbb7r\" (UID: \"f8d25fcd-f343-445b-ac68-b2248f619ee4\") " pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" Oct 30 20:01:50.436883 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:50.436866 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8d25fcd-f343-445b-ac68-b2248f619ee4-tls-certs\") pod \"premium-w3-912f24-kserve-56d6b776d7-mbb7r\" (UID: \"f8d25fcd-f343-445b-ac68-b2248f619ee4\") " pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" Oct 30 20:01:50.460900 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:50.460877 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wsxf\" (UniqueName: \"kubernetes.io/projected/f8d25fcd-f343-445b-ac68-b2248f619ee4-kube-api-access-4wsxf\") pod \"premium-w3-912f24-kserve-56d6b776d7-mbb7r\" (UID: \"f8d25fcd-f343-445b-ac68-b2248f619ee4\") " pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" Oct 30 20:01:50.610064 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:50.609975 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" Oct 30 20:01:50.924624 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:50.924593 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r"] Oct 30 20:01:50.925743 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:01:50.925720 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8d25fcd_f343_445b_ac68_b2248f619ee4.slice/crio-ec546d234c701b753d70fb4ecb8075c340fddbdb7cdbab19faf2bf26cfedcaae WatchSource:0}: Error finding container ec546d234c701b753d70fb4ecb8075c340fddbdb7cdbab19faf2bf26cfedcaae: Status 404 returned error can't find the container with id ec546d234c701b753d70fb4ecb8075c340fddbdb7cdbab19faf2bf26cfedcaae Oct 30 20:01:51.206649 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:51.206534 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" event={"ID":"f8d25fcd-f343-445b-ac68-b2248f619ee4","Type":"ContainerStarted","Data":"f6298a28e1fd184800beed4f30d68e12df45ab4e8de89f98a576b58dbafc33ad"} Oct 30 20:01:51.206649 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:51.206587 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" event={"ID":"f8d25fcd-f343-445b-ac68-b2248f619ee4","Type":"ContainerStarted","Data":"ec546d234c701b753d70fb4ecb8075c340fddbdb7cdbab19faf2bf26cfedcaae"} Oct 30 20:01:51.267177 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:51.267128 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podStartSLOduration=1.267115062 podStartE2EDuration="1.267115062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:51.265744913 +0000 UTC m=+872.678458705" watchObservedRunningTime="2026-10-30 20:01:51.267115062 +0000 UTC m=+872.679828874" Oct 30 20:01:51.354909 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:51.354875 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:01:51.610215 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:01:51.610185 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" Oct 30 20:01:52.437059 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:01:52.437010 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:02:00.611070 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:00.611034 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" Oct 30 20:02:00.623364 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:00.623342 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" Oct 30 20:02:01.299294 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:01.299261 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" Oct 30 20:02:06.155427 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:06.155392 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:02:15.191239 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:15.191207 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t"] Oct 30 20:02:15.195303 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:15.195284 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:02:15.230889 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:15.230860 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-51dd62\"/\"premium-w6-51dd62-kserve-self-signed-certs\"" Oct 30 20:02:15.230889 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:15.230879 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-51dd62\"/\"default-dockercfg-wnz6m\"" Oct 30 20:02:15.231079 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:15.230861 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-51dd62\"/\"openshift-service-ca.crt\"" Oct 30 20:02:15.231079 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:15.231029 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-51dd62\"/\"kube-root-ca.crt\"" Oct 30 20:02:15.233841 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:15.233819 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t"] Oct 30 20:02:15.294949 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:15.294915 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b42fe97-2609-4fa1-a51a-c901e34997b5-tmp-dir\") pod \"premium-w6-51dd62-kserve-68487679fd-wwr8t\" (UID: \"5b42fe97-2609-4fa1-a51a-c901e34997b5\") " pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:02:15.295115 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:15.295057 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74hxr\" (UniqueName: \"kubernetes.io/projected/5b42fe97-2609-4fa1-a51a-c901e34997b5-kube-api-access-74hxr\") pod \"premium-w6-51dd62-kserve-68487679fd-wwr8t\" (UID: \"5b42fe97-2609-4fa1-a51a-c901e34997b5\") " pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:02:15.295115 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:15.295107 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b42fe97-2609-4fa1-a51a-c901e34997b5-model-cache\") pod \"premium-w6-51dd62-kserve-68487679fd-wwr8t\" (UID: \"5b42fe97-2609-4fa1-a51a-c901e34997b5\") " pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:02:15.295216 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:15.295169 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b42fe97-2609-4fa1-a51a-c901e34997b5-tls-certs\") pod \"premium-w6-51dd62-kserve-68487679fd-wwr8t\" (UID: \"5b42fe97-2609-4fa1-a51a-c901e34997b5\") " pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:02:15.295216 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:15.295204 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b42fe97-2609-4fa1-a51a-c901e34997b5-home\") pod \"premium-w6-51dd62-kserve-68487679fd-wwr8t\" (UID: \"5b42fe97-2609-4fa1-a51a-c901e34997b5\") " pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:02:15.295305 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:15.295236 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b42fe97-2609-4fa1-a51a-c901e34997b5-dshm\") pod \"premium-w6-51dd62-kserve-68487679fd-wwr8t\" (UID: \"5b42fe97-2609-4fa1-a51a-c901e34997b5\") " pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:02:15.443001 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:15.442890 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b42fe97-2609-4fa1-a51a-c901e34997b5-home\") pod \"premium-w6-51dd62-kserve-68487679fd-wwr8t\" (UID: \"5b42fe97-2609-4fa1-a51a-c901e34997b5\") " pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:02:15.443181 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:15.443029 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b42fe97-2609-4fa1-a51a-c901e34997b5-dshm\") pod \"premium-w6-51dd62-kserve-68487679fd-wwr8t\" (UID: \"5b42fe97-2609-4fa1-a51a-c901e34997b5\") " pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:02:15.443250 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:15.443178 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b42fe97-2609-4fa1-a51a-c901e34997b5-tmp-dir\") pod \"premium-w6-51dd62-kserve-68487679fd-wwr8t\" (UID: \"5b42fe97-2609-4fa1-a51a-c901e34997b5\") " pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:02:15.443305 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:15.443285 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74hxr\" (UniqueName: \"kubernetes.io/projected/5b42fe97-2609-4fa1-a51a-c901e34997b5-kube-api-access-74hxr\") pod \"premium-w6-51dd62-kserve-68487679fd-wwr8t\" (UID: \"5b42fe97-2609-4fa1-a51a-c901e34997b5\") " pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:02:15.443354 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:15.443333 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b42fe97-2609-4fa1-a51a-c901e34997b5-model-cache\") pod \"premium-w6-51dd62-kserve-68487679fd-wwr8t\" (UID: \"5b42fe97-2609-4fa1-a51a-c901e34997b5\") " pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:02:15.443401 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:15.443363 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b42fe97-2609-4fa1-a51a-c901e34997b5-tls-certs\") pod \"premium-w6-51dd62-kserve-68487679fd-wwr8t\" (UID: \"5b42fe97-2609-4fa1-a51a-c901e34997b5\") " pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:02:15.445728 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:15.444943 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b42fe97-2609-4fa1-a51a-c901e34997b5-tmp-dir\") pod \"premium-w6-51dd62-kserve-68487679fd-wwr8t\" (UID: \"5b42fe97-2609-4fa1-a51a-c901e34997b5\") " pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:02:15.445728 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:15.445319 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b42fe97-2609-4fa1-a51a-c901e34997b5-home\") pod \"premium-w6-51dd62-kserve-68487679fd-wwr8t\" (UID: \"5b42fe97-2609-4fa1-a51a-c901e34997b5\") " pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:02:15.445989 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:15.445965 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b42fe97-2609-4fa1-a51a-c901e34997b5-model-cache\") pod \"premium-w6-51dd62-kserve-68487679fd-wwr8t\" (UID: \"5b42fe97-2609-4fa1-a51a-c901e34997b5\") " pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:02:15.460719 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:15.460688 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b42fe97-2609-4fa1-a51a-c901e34997b5-dshm\") pod \"premium-w6-51dd62-kserve-68487679fd-wwr8t\" (UID: \"5b42fe97-2609-4fa1-a51a-c901e34997b5\") " pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:02:15.461500 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:15.461473 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b42fe97-2609-4fa1-a51a-c901e34997b5-tls-certs\") pod \"premium-w6-51dd62-kserve-68487679fd-wwr8t\" (UID: \"5b42fe97-2609-4fa1-a51a-c901e34997b5\") " pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:02:15.471087 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:15.471060 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74hxr\" (UniqueName: \"kubernetes.io/projected/5b42fe97-2609-4fa1-a51a-c901e34997b5-kube-api-access-74hxr\") pod \"premium-w6-51dd62-kserve-68487679fd-wwr8t\" (UID: \"5b42fe97-2609-4fa1-a51a-c901e34997b5\") " pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:02:15.508902 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:15.508875 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:02:15.816884 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:15.816861 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t"] Oct 30 20:02:15.817592 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:02:15.817567 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b42fe97_2609_4fa1_a51a_c901e34997b5.slice/crio-3378cd2fe8872ab9cbfdbae6f7ab420764ba4aaf2cf54f1aa113814d652fd91c WatchSource:0}: Error finding container 3378cd2fe8872ab9cbfdbae6f7ab420764ba4aaf2cf54f1aa113814d652fd91c: Status 404 returned error can't find the container with id 3378cd2fe8872ab9cbfdbae6f7ab420764ba4aaf2cf54f1aa113814d652fd91c Oct 30 20:02:15.970092 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:15.970060 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e92c58f0-jzwmm"] Oct 30 20:02:15.973938 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:15.973922 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92c58f0-jzwmm" Oct 30 20:02:15.993522 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:15.992649 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:02:16.001466 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:16.001433 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e92c58f0-jzwmm"] Oct 30 20:02:16.009130 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:16.009107 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:02:16.152533 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:16.152446 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj25m\" (UniqueName: \"kubernetes.io/projected/908e797e-bda5-4f61-ac6a-4c8c63638034-kube-api-access-pj25m\") pod \"maas-api-revoke-sub-e92c58f0-jzwmm\" (UID: \"908e797e-bda5-4f61-ac6a-4c8c63638034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92c58f0-jzwmm" Oct 30 20:02:16.152533 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:16.152486 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/908e797e-bda5-4f61-ac6a-4c8c63638034-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e92c58f0-jzwmm\" (UID: \"908e797e-bda5-4f61-ac6a-4c8c63638034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92c58f0-jzwmm" Oct 30 20:02:16.152756 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:16.152641 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/908e797e-bda5-4f61-ac6a-4c8c63638034-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e92c58f0-jzwmm\" (UID: \"908e797e-bda5-4f61-ac6a-4c8c63638034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92c58f0-jzwmm" Oct 30 20:02:16.253603 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:16.253571 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj25m\" (UniqueName: \"kubernetes.io/projected/908e797e-bda5-4f61-ac6a-4c8c63638034-kube-api-access-pj25m\") pod \"maas-api-revoke-sub-e92c58f0-jzwmm\" (UID: \"908e797e-bda5-4f61-ac6a-4c8c63638034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92c58f0-jzwmm" Oct 30 20:02:16.254011 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:16.253625 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/908e797e-bda5-4f61-ac6a-4c8c63638034-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e92c58f0-jzwmm\" (UID: \"908e797e-bda5-4f61-ac6a-4c8c63638034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92c58f0-jzwmm" Oct 30 20:02:16.254011 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:16.253704 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/908e797e-bda5-4f61-ac6a-4c8c63638034-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e92c58f0-jzwmm\" (UID: \"908e797e-bda5-4f61-ac6a-4c8c63638034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92c58f0-jzwmm" Oct 30 20:02:16.254403 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:16.254383 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/908e797e-bda5-4f61-ac6a-4c8c63638034-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e92c58f0-jzwmm\" (UID: \"908e797e-bda5-4f61-ac6a-4c8c63638034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92c58f0-jzwmm" Oct 30 20:02:16.255946 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:16.255929 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/908e797e-bda5-4f61-ac6a-4c8c63638034-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e92c58f0-jzwmm\" (UID: \"908e797e-bda5-4f61-ac6a-4c8c63638034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92c58f0-jzwmm" Oct 30 20:02:16.279927 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:16.279905 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj25m\" (UniqueName: \"kubernetes.io/projected/908e797e-bda5-4f61-ac6a-4c8c63638034-kube-api-access-pj25m\") pod \"maas-api-revoke-sub-e92c58f0-jzwmm\" (UID: \"908e797e-bda5-4f61-ac6a-4c8c63638034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92c58f0-jzwmm" Oct 30 20:02:16.286741 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:16.286716 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92c58f0-jzwmm" Oct 30 20:02:16.440231 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:16.440148 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" event={"ID":"5b42fe97-2609-4fa1-a51a-c901e34997b5","Type":"ContainerStarted","Data":"a577aa3fb052d00f2e6a84cdf22c1cb06b661a526ca1d38e74ebea0991977dad"} Oct 30 20:02:16.440231 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:16.440192 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" event={"ID":"5b42fe97-2609-4fa1-a51a-c901e34997b5","Type":"ContainerStarted","Data":"3378cd2fe8872ab9cbfdbae6f7ab420764ba4aaf2cf54f1aa113814d652fd91c"} Oct 30 20:02:16.488522 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:16.488473 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podStartSLOduration=1.488460847 podStartE2EDuration="1.488460847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:16.486144021 +0000 UTC m=+897.898857801" watchObservedRunningTime="2026-10-30 20:02:16.488460847 +0000 UTC m=+897.901174625" Oct 30 20:02:16.509342 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:16.509317 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:02:16.590452 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:16.590425 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e92c58f0-jzwmm"] Oct 30 20:02:16.591967 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:02:16.591939 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod908e797e_bda5_4f61_ac6a_4c8c63638034.slice/crio-a83e8582faba4119ed1340e19b3e0cedd7a58146c1d4be36c612a161c514a278 WatchSource:0}: Error finding container a83e8582faba4119ed1340e19b3e0cedd7a58146c1d4be36c612a161c514a278: Status 404 returned error can't find the container with id a83e8582faba4119ed1340e19b3e0cedd7a58146c1d4be36c612a161c514a278 Oct 30 20:02:17.447732 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:17.447699 5088 generic.go:356] "Generic (PLEG): container finished" podID="908e797e-bda5-4f61-ac6a-4c8c63638034" containerID="d1d18181ce66ed778e943e4fca72c0dd81f04ce988136fbea2a0e94269571404" exitCode=0 Oct 30 20:02:17.448084 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:17.447787 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92c58f0-jzwmm" event={"ID":"908e797e-bda5-4f61-ac6a-4c8c63638034","Type":"ContainerDied","Data":"d1d18181ce66ed778e943e4fca72c0dd81f04ce988136fbea2a0e94269571404"} Oct 30 20:02:17.448084 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:17.447841 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92c58f0-jzwmm" event={"ID":"908e797e-bda5-4f61-ac6a-4c8c63638034","Type":"ContainerStarted","Data":"a83e8582faba4119ed1340e19b3e0cedd7a58146c1d4be36c612a161c514a278"} Oct 30 20:02:18.600978 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:18.600950 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92c58f0-jzwmm" Oct 30 20:02:18.682381 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:18.682347 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/908e797e-bda5-4f61-ac6a-4c8c63638034-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/908e797e-bda5-4f61-ac6a-4c8c63638034-maas-api-openshift-ca\") pod \"908e797e-bda5-4f61-ac6a-4c8c63638034\" (UID: \"908e797e-bda5-4f61-ac6a-4c8c63638034\") " Oct 30 20:02:18.682561 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:18.682389 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/908e797e-bda5-4f61-ac6a-4c8c63638034-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/908e797e-bda5-4f61-ac6a-4c8c63638034-maas-api-xks-ca\") pod \"908e797e-bda5-4f61-ac6a-4c8c63638034\" (UID: \"908e797e-bda5-4f61-ac6a-4c8c63638034\") " Oct 30 20:02:18.682561 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:18.682432 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/908e797e-bda5-4f61-ac6a-4c8c63638034-kube-api-access-pj25m\" (UniqueName: \"kubernetes.io/projected/908e797e-bda5-4f61-ac6a-4c8c63638034-kube-api-access-pj25m\") pod \"908e797e-bda5-4f61-ac6a-4c8c63638034\" (UID: \"908e797e-bda5-4f61-ac6a-4c8c63638034\") " Oct 30 20:02:18.682766 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:18.682744 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/908e797e-bda5-4f61-ac6a-4c8c63638034-maas-api-openshift-ca" pod "908e797e-bda5-4f61-ac6a-4c8c63638034" (UID: "908e797e-bda5-4f61-ac6a-4c8c63638034"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:18.682843 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:18.682818 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/908e797e-bda5-4f61-ac6a-4c8c63638034-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:02:18.684554 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:18.684523 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/908e797e-bda5-4f61-ac6a-4c8c63638034-maas-api-xks-ca" pod "908e797e-bda5-4f61-ac6a-4c8c63638034" (UID: "908e797e-bda5-4f61-ac6a-4c8c63638034"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:18.684689 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:18.684585 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/908e797e-bda5-4f61-ac6a-4c8c63638034-kube-api-access-pj25m" pod "908e797e-bda5-4f61-ac6a-4c8c63638034" (UID: "908e797e-bda5-4f61-ac6a-4c8c63638034"). InnerVolumeSpecName "kube-api-access-pj25m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:18.784132 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:18.784105 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/908e797e-bda5-4f61-ac6a-4c8c63638034-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:02:18.784132 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:18.784133 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj25m\" (UniqueName: \"kubernetes.io/projected/908e797e-bda5-4f61-ac6a-4c8c63638034-kube-api-access-pj25m\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:02:18.819351 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:18.819316 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qdj4h_da850825-59ab-480f-9047-afd7d2bdc19f/console-operator/1.log" Oct 30 20:02:18.820430 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:18.820407 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qdj4h_da850825-59ab-480f-9047-afd7d2bdc19f/console-operator/1.log" Oct 30 20:02:18.824350 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:18.824330 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xpjpn_688d079a-932d-4f63-97da-5cd6222a03e1/ovn-acl-logging/0.log" Oct 30 20:02:18.826086 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:18.825884 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xpjpn_688d079a-932d-4f63-97da-5cd6222a03e1/ovn-acl-logging/0.log" Oct 30 20:02:19.460884 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:19.460851 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92c58f0-jzwmm" Oct 30 20:02:19.461077 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:19.460882 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92c58f0-jzwmm" event={"ID":"908e797e-bda5-4f61-ac6a-4c8c63638034","Type":"ContainerDied","Data":"a83e8582faba4119ed1340e19b3e0cedd7a58146c1d4be36c612a161c514a278"} Oct 30 20:02:19.461077 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:19.460969 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a83e8582faba4119ed1340e19b3e0cedd7a58146c1d4be36c612a161c514a278" Oct 30 20:02:22.437628 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:02:22.437584 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:02:24.342453 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.342419 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz"] Oct 30 20:02:24.342958 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.342944 5088 cpu_manager.go:414] "containerMap: removing container" podUID="908e797e-bda5-4f61-ac6a-4c8c63638034" containerName="revoke-keys" Oct 30 20:02:24.343016 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.342959 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="908e797e-bda5-4f61-ac6a-4c8c63638034" containerName="revoke-keys" Oct 30 20:02:24.343057 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.343051 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="908e797e-bda5-4f61-ac6a-4c8c63638034" containerName="revoke-keys" Oct 30 20:02:24.346643 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.346626 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" Oct 30 20:02:24.375094 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.375068 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz"] Oct 30 20:02:24.378193 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.378168 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/678518a9-8731-4f6f-a7a1-4b0e64070e92-istio-data\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz\" (UID: \"678518a9-8731-4f6f-a7a1-4b0e64070e92\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" Oct 30 20:02:24.378332 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.378243 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9s8x\" (UniqueName: \"kubernetes.io/projected/678518a9-8731-4f6f-a7a1-4b0e64070e92-kube-api-access-b9s8x\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz\" (UID: \"678518a9-8731-4f6f-a7a1-4b0e64070e92\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" Oct 30 20:02:24.378332 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.378313 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/678518a9-8731-4f6f-a7a1-4b0e64070e92-workload-certs\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz\" (UID: \"678518a9-8731-4f6f-a7a1-4b0e64070e92\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" Oct 30 20:02:24.378462 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.378352 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/678518a9-8731-4f6f-a7a1-4b0e64070e92-workload-socket\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz\" (UID: \"678518a9-8731-4f6f-a7a1-4b0e64070e92\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" Oct 30 20:02:24.378462 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.378372 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/678518a9-8731-4f6f-a7a1-4b0e64070e92-istiod-ca-cert\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz\" (UID: \"678518a9-8731-4f6f-a7a1-4b0e64070e92\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" Oct 30 20:02:24.378462 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.378435 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/678518a9-8731-4f6f-a7a1-4b0e64070e92-credential-socket\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz\" (UID: \"678518a9-8731-4f6f-a7a1-4b0e64070e92\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" Oct 30 20:02:24.378633 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.378474 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/678518a9-8731-4f6f-a7a1-4b0e64070e92-istio-podinfo\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz\" (UID: \"678518a9-8731-4f6f-a7a1-4b0e64070e92\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" Oct 30 20:02:24.378633 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.378526 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/678518a9-8731-4f6f-a7a1-4b0e64070e92-istio-token\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz\" (UID: \"678518a9-8731-4f6f-a7a1-4b0e64070e92\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" Oct 30 20:02:24.378633 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.378568 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/678518a9-8731-4f6f-a7a1-4b0e64070e92-istio-envoy\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz\" (UID: \"678518a9-8731-4f6f-a7a1-4b0e64070e92\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" Oct 30 20:02:24.480138 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.480106 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/678518a9-8731-4f6f-a7a1-4b0e64070e92-istio-data\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz\" (UID: \"678518a9-8731-4f6f-a7a1-4b0e64070e92\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" Oct 30 20:02:24.480322 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.480191 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9s8x\" (UniqueName: \"kubernetes.io/projected/678518a9-8731-4f6f-a7a1-4b0e64070e92-kube-api-access-b9s8x\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz\" (UID: \"678518a9-8731-4f6f-a7a1-4b0e64070e92\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" Oct 30 20:02:24.480322 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.480228 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/678518a9-8731-4f6f-a7a1-4b0e64070e92-workload-certs\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz\" (UID: \"678518a9-8731-4f6f-a7a1-4b0e64070e92\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" Oct 30 20:02:24.480322 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.480278 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/678518a9-8731-4f6f-a7a1-4b0e64070e92-workload-socket\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz\" (UID: \"678518a9-8731-4f6f-a7a1-4b0e64070e92\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" Oct 30 20:02:24.480322 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.480300 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/678518a9-8731-4f6f-a7a1-4b0e64070e92-istiod-ca-cert\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz\" (UID: \"678518a9-8731-4f6f-a7a1-4b0e64070e92\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" Oct 30 20:02:24.480595 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.480502 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/678518a9-8731-4f6f-a7a1-4b0e64070e92-credential-socket\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz\" (UID: \"678518a9-8731-4f6f-a7a1-4b0e64070e92\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" Oct 30 20:02:24.480595 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.480523 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/678518a9-8731-4f6f-a7a1-4b0e64070e92-istio-data\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz\" (UID: \"678518a9-8731-4f6f-a7a1-4b0e64070e92\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" Oct 30 20:02:24.480595 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.480569 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/678518a9-8731-4f6f-a7a1-4b0e64070e92-istio-podinfo\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz\" (UID: \"678518a9-8731-4f6f-a7a1-4b0e64070e92\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" Oct 30 20:02:24.480780 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.480655 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/678518a9-8731-4f6f-a7a1-4b0e64070e92-istio-token\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz\" (UID: \"678518a9-8731-4f6f-a7a1-4b0e64070e92\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" Oct 30 20:02:24.480780 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.480658 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/678518a9-8731-4f6f-a7a1-4b0e64070e92-workload-socket\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz\" (UID: \"678518a9-8731-4f6f-a7a1-4b0e64070e92\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" Oct 30 20:02:24.480780 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.480697 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/678518a9-8731-4f6f-a7a1-4b0e64070e92-istio-envoy\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz\" (UID: \"678518a9-8731-4f6f-a7a1-4b0e64070e92\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" Oct 30 20:02:24.480936 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.480809 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/678518a9-8731-4f6f-a7a1-4b0e64070e92-credential-socket\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz\" (UID: \"678518a9-8731-4f6f-a7a1-4b0e64070e92\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" Oct 30 20:02:24.480936 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.480859 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/678518a9-8731-4f6f-a7a1-4b0e64070e92-workload-certs\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz\" (UID: \"678518a9-8731-4f6f-a7a1-4b0e64070e92\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" Oct 30 20:02:24.481140 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.481119 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/678518a9-8731-4f6f-a7a1-4b0e64070e92-istiod-ca-cert\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz\" (UID: \"678518a9-8731-4f6f-a7a1-4b0e64070e92\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" Oct 30 20:02:24.482825 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.482807 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/678518a9-8731-4f6f-a7a1-4b0e64070e92-istio-envoy\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz\" (UID: \"678518a9-8731-4f6f-a7a1-4b0e64070e92\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" Oct 30 20:02:24.483170 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.483151 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/678518a9-8731-4f6f-a7a1-4b0e64070e92-istio-podinfo\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz\" (UID: \"678518a9-8731-4f6f-a7a1-4b0e64070e92\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" Oct 30 20:02:24.506815 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.506788 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/678518a9-8731-4f6f-a7a1-4b0e64070e92-istio-token\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz\" (UID: \"678518a9-8731-4f6f-a7a1-4b0e64070e92\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" Oct 30 20:02:24.506968 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.506949 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9s8x\" (UniqueName: \"kubernetes.io/projected/678518a9-8731-4f6f-a7a1-4b0e64070e92-kube-api-access-b9s8x\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz\" (UID: \"678518a9-8731-4f6f-a7a1-4b0e64070e92\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" Oct 30 20:02:24.660472 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.660394 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" Oct 30 20:02:24.756350 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.755921 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s"] Oct 30 20:02:24.768189 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.768160 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s"] Oct 30 20:02:24.768322 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.768280 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" Oct 30 20:02:24.788072 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.788044 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-62f4a1-openshift-default-dockercfg-hg55k\"" Oct 30 20:02:24.883806 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.883765 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff410c2e-9048-4e5c-965b-1261336561bb-istio-token\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s\" (UID: \"ff410c2e-9048-4e5c-965b-1261336561bb\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" Oct 30 20:02:24.883806 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.883812 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff410c2e-9048-4e5c-965b-1261336561bb-workload-certs\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s\" (UID: \"ff410c2e-9048-4e5c-965b-1261336561bb\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" Oct 30 20:02:24.884040 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.883895 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff410c2e-9048-4e5c-965b-1261336561bb-workload-socket\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s\" (UID: \"ff410c2e-9048-4e5c-965b-1261336561bb\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" Oct 30 20:02:24.884040 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.883919 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff410c2e-9048-4e5c-965b-1261336561bb-credential-socket\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s\" (UID: \"ff410c2e-9048-4e5c-965b-1261336561bb\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" Oct 30 20:02:24.884040 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.883937 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff410c2e-9048-4e5c-965b-1261336561bb-istiod-ca-cert\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s\" (UID: \"ff410c2e-9048-4e5c-965b-1261336561bb\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" Oct 30 20:02:24.884040 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.884021 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qckvd\" (UniqueName: \"kubernetes.io/projected/ff410c2e-9048-4e5c-965b-1261336561bb-kube-api-access-qckvd\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s\" (UID: \"ff410c2e-9048-4e5c-965b-1261336561bb\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" Oct 30 20:02:24.884245 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.884069 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff410c2e-9048-4e5c-965b-1261336561bb-istio-envoy\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s\" (UID: \"ff410c2e-9048-4e5c-965b-1261336561bb\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" Oct 30 20:02:24.884245 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.884089 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff410c2e-9048-4e5c-965b-1261336561bb-istio-podinfo\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s\" (UID: \"ff410c2e-9048-4e5c-965b-1261336561bb\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" Oct 30 20:02:24.884245 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.884160 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff410c2e-9048-4e5c-965b-1261336561bb-istio-data\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s\" (UID: \"ff410c2e-9048-4e5c-965b-1261336561bb\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" Oct 30 20:02:24.985009 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.984982 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff410c2e-9048-4e5c-965b-1261336561bb-istio-envoy\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s\" (UID: \"ff410c2e-9048-4e5c-965b-1261336561bb\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" Oct 30 20:02:24.985194 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.985021 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff410c2e-9048-4e5c-965b-1261336561bb-istio-podinfo\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s\" (UID: \"ff410c2e-9048-4e5c-965b-1261336561bb\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" Oct 30 20:02:24.985194 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.985048 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff410c2e-9048-4e5c-965b-1261336561bb-istio-data\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s\" (UID: \"ff410c2e-9048-4e5c-965b-1261336561bb\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" Oct 30 20:02:24.985194 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.985122 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff410c2e-9048-4e5c-965b-1261336561bb-istio-token\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s\" (UID: \"ff410c2e-9048-4e5c-965b-1261336561bb\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" Oct 30 20:02:24.985194 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.985169 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff410c2e-9048-4e5c-965b-1261336561bb-workload-certs\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s\" (UID: \"ff410c2e-9048-4e5c-965b-1261336561bb\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" Oct 30 20:02:24.985411 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.985214 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff410c2e-9048-4e5c-965b-1261336561bb-workload-socket\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s\" (UID: \"ff410c2e-9048-4e5c-965b-1261336561bb\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" Oct 30 20:02:24.985411 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.985238 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff410c2e-9048-4e5c-965b-1261336561bb-credential-socket\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s\" (UID: \"ff410c2e-9048-4e5c-965b-1261336561bb\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" Oct 30 20:02:24.985411 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.985263 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff410c2e-9048-4e5c-965b-1261336561bb-istiod-ca-cert\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s\" (UID: \"ff410c2e-9048-4e5c-965b-1261336561bb\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" Oct 30 20:02:24.985411 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.985325 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qckvd\" (UniqueName: \"kubernetes.io/projected/ff410c2e-9048-4e5c-965b-1261336561bb-kube-api-access-qckvd\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s\" (UID: \"ff410c2e-9048-4e5c-965b-1261336561bb\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" Oct 30 20:02:24.985646 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.985563 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff410c2e-9048-4e5c-965b-1261336561bb-istio-data\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s\" (UID: \"ff410c2e-9048-4e5c-965b-1261336561bb\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" Oct 30 20:02:24.985698 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.985653 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff410c2e-9048-4e5c-965b-1261336561bb-workload-certs\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s\" (UID: \"ff410c2e-9048-4e5c-965b-1261336561bb\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" Oct 30 20:02:24.985925 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.985900 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff410c2e-9048-4e5c-965b-1261336561bb-workload-socket\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s\" (UID: \"ff410c2e-9048-4e5c-965b-1261336561bb\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" Oct 30 20:02:24.986007 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.985924 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff410c2e-9048-4e5c-965b-1261336561bb-credential-socket\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s\" (UID: \"ff410c2e-9048-4e5c-965b-1261336561bb\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" Oct 30 20:02:24.986100 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.986083 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff410c2e-9048-4e5c-965b-1261336561bb-istiod-ca-cert\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s\" (UID: \"ff410c2e-9048-4e5c-965b-1261336561bb\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" Oct 30 20:02:24.987512 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.987491 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff410c2e-9048-4e5c-965b-1261336561bb-istio-podinfo\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s\" (UID: \"ff410c2e-9048-4e5c-965b-1261336561bb\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" Oct 30 20:02:24.987627 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.987528 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff410c2e-9048-4e5c-965b-1261336561bb-istio-envoy\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s\" (UID: \"ff410c2e-9048-4e5c-965b-1261336561bb\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" Oct 30 20:02:24.996022 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.996002 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz"] Oct 30 20:02:24.997266 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:02:24.997244 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod678518a9_8731_4f6f_a7a1_4b0e64070e92.slice/crio-ab441d6fa6aef2eedee3d13c580b44456a212a40006513cc2229ca20a9b9ec1b WatchSource:0}: Error finding container ab441d6fa6aef2eedee3d13c580b44456a212a40006513cc2229ca20a9b9ec1b: Status 404 returned error can't find the container with id ab441d6fa6aef2eedee3d13c580b44456a212a40006513cc2229ca20a9b9ec1b Oct 30 20:02:24.999479 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.999450 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:02:24.999552 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.999511 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:02:24.999552 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:24.999539 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:02:25.016911 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:25.016889 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff410c2e-9048-4e5c-965b-1261336561bb-istio-token\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s\" (UID: \"ff410c2e-9048-4e5c-965b-1261336561bb\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" Oct 30 20:02:25.017124 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:25.017105 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qckvd\" (UniqueName: \"kubernetes.io/projected/ff410c2e-9048-4e5c-965b-1261336561bb-kube-api-access-qckvd\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s\" (UID: \"ff410c2e-9048-4e5c-965b-1261336561bb\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" Oct 30 20:02:25.083107 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:25.083075 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" Oct 30 20:02:25.422146 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:25.422073 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s"] Oct 30 20:02:25.425875 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:02:25.425848 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff410c2e_9048_4e5c_965b_1261336561bb.slice/crio-00c2f8bdf4d598a3c83d6f14f27706d3fb1b1403ec1042fe9186bedd748cb828 WatchSource:0}: Error finding container 00c2f8bdf4d598a3c83d6f14f27706d3fb1b1403ec1042fe9186bedd748cb828: Status 404 returned error can't find the container with id 00c2f8bdf4d598a3c83d6f14f27706d3fb1b1403ec1042fe9186bedd748cb828 Oct 30 20:02:25.428941 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:25.428911 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:02:25.429004 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:25.428980 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:02:25.429036 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:25.429010 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:02:25.514742 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:25.512670 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:02:25.521389 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:25.521283 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" event={"ID":"678518a9-8731-4f6f-a7a1-4b0e64070e92","Type":"ContainerStarted","Data":"8e3731c9ad46f1c7da2c1fed08fddf94087b20095f37969b667fa8d70bfbcb89"} Oct 30 20:02:25.521389 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:25.521350 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" event={"ID":"678518a9-8731-4f6f-a7a1-4b0e64070e92","Type":"ContainerStarted","Data":"ab441d6fa6aef2eedee3d13c580b44456a212a40006513cc2229ca20a9b9ec1b"} Oct 30 20:02:25.522854 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:25.522813 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" event={"ID":"ff410c2e-9048-4e5c-965b-1261336561bb","Type":"ContainerStarted","Data":"00c2f8bdf4d598a3c83d6f14f27706d3fb1b1403ec1042fe9186bedd748cb828"} Oct 30 20:02:25.530497 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:25.530476 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:02:25.573419 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:25.573369 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" podStartSLOduration=1.573356225 podStartE2EDuration="1.573356225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:25.572227126 +0000 UTC m=+906.984940929" watchObservedRunningTime="2026-10-30 20:02:25.573356225 +0000 UTC m=+906.986070000" Oct 30 20:02:25.661295 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:25.661263 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" Oct 30 20:02:26.544709 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:26.544673 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" event={"ID":"ff410c2e-9048-4e5c-965b-1261336561bb","Type":"ContainerStarted","Data":"6cfacb45b1d70e6292533f4fefcaba52f16eaac85d2922a2b8acf5e5fb5791d1"} Oct 30 20:02:26.556458 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:26.556429 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:02:26.624636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:26.623833 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" podStartSLOduration=2.623813133 podStartE2EDuration="2.623813133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:26.598083679 +0000 UTC m=+908.010797498" watchObservedRunningTime="2026-10-30 20:02:26.623813133 +0000 UTC m=+908.036526913" Oct 30 20:02:26.663388 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:26.662942 5088 patch_prober.go:29] interesting pod/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 20:02:26.663388 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:26.662998 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" podUID="678518a9-8731-4f6f-a7a1-4b0e64070e92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 20:02:27.083462 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:27.083177 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" Oct 30 20:02:27.103635 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:27.097429 5088 patch_prober.go:29] interesting pod/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 20:02:27.103635 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:27.097501 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" podUID="ff410c2e-9048-4e5c-965b-1261336561bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 20:02:27.660897 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:27.660863 5088 patch_prober.go:29] interesting pod/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 20:02:27.661382 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:27.660925 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" podUID="678518a9-8731-4f6f-a7a1-4b0e64070e92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 20:02:28.127637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:28.126992 5088 patch_prober.go:29] interesting pod/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 20:02:28.127637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:28.127076 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" podUID="ff410c2e-9048-4e5c-965b-1261336561bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 20:02:28.661032 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:28.660987 5088 patch_prober.go:29] interesting pod/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 20:02:28.661427 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:28.661041 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" podUID="678518a9-8731-4f6f-a7a1-4b0e64070e92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 20:02:29.091638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:29.087902 5088 patch_prober.go:29] interesting pod/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 20:02:29.091638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:29.087961 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" podUID="ff410c2e-9048-4e5c-965b-1261336561bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 20:02:29.661343 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:29.661306 5088 patch_prober.go:29] interesting pod/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 20:02:29.661775 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:29.661359 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" podUID="678518a9-8731-4f6f-a7a1-4b0e64070e92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 20:02:30.111646 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:30.106710 5088 patch_prober.go:29] interesting pod/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 20:02:30.111646 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:30.106794 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" podUID="ff410c2e-9048-4e5c-965b-1261336561bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 20:02:30.661435 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:30.661400 5088 patch_prober.go:29] interesting pod/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 20:02:30.661845 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:30.661453 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" podUID="678518a9-8731-4f6f-a7a1-4b0e64070e92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 20:02:31.089360 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:31.089319 5088 patch_prober.go:29] interesting pod/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 20:02:31.089641 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:31.089585 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" podUID="ff410c2e-9048-4e5c-965b-1261336561bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 20:02:31.950234 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:31.950200 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" Oct 30 20:02:31.950699 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:31.950660 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" Oct 30 20:02:31.951787 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:31.951692 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" Oct 30 20:02:32.084254 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:32.084218 5088 patch_prober.go:29] interesting pod/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 20:02:32.084441 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:32.084302 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" podUID="ff410c2e-9048-4e5c-965b-1261336561bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 20:02:32.466020 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:32.465982 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3d36dcc-bsrzf"] Oct 30 20:02:32.471229 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:32.471200 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d36dcc-bsrzf" Oct 30 20:02:32.493757 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:32.493731 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:02:32.493886 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:32.493850 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:02:32.498230 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:32.498202 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3d36dcc-bsrzf"] Oct 30 20:02:32.561586 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:32.561553 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e30d967b-a226-43a9-98ba-acb2bcc23e4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3d36dcc-bsrzf\" (UID: \"e30d967b-a226-43a9-98ba-acb2bcc23e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d36dcc-bsrzf" Oct 30 20:02:32.561780 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:32.561600 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e30d967b-a226-43a9-98ba-acb2bcc23e4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3d36dcc-bsrzf\" (UID: \"e30d967b-a226-43a9-98ba-acb2bcc23e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d36dcc-bsrzf" Oct 30 20:02:32.561869 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:32.561779 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69hqp\" (UniqueName: \"kubernetes.io/projected/e30d967b-a226-43a9-98ba-acb2bcc23e4c-kube-api-access-69hqp\") pod \"maas-api-revoke-sub-e3d36dcc-bsrzf\" (UID: \"e30d967b-a226-43a9-98ba-acb2bcc23e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d36dcc-bsrzf" Oct 30 20:02:32.662583 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:32.662550 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69hqp\" (UniqueName: \"kubernetes.io/projected/e30d967b-a226-43a9-98ba-acb2bcc23e4c-kube-api-access-69hqp\") pod \"maas-api-revoke-sub-e3d36dcc-bsrzf\" (UID: \"e30d967b-a226-43a9-98ba-acb2bcc23e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d36dcc-bsrzf" Oct 30 20:02:32.662797 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:32.662737 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e30d967b-a226-43a9-98ba-acb2bcc23e4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3d36dcc-bsrzf\" (UID: \"e30d967b-a226-43a9-98ba-acb2bcc23e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d36dcc-bsrzf" Oct 30 20:02:32.662935 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:32.662904 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e30d967b-a226-43a9-98ba-acb2bcc23e4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3d36dcc-bsrzf\" (UID: \"e30d967b-a226-43a9-98ba-acb2bcc23e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d36dcc-bsrzf" Oct 30 20:02:32.663360 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:32.663339 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e30d967b-a226-43a9-98ba-acb2bcc23e4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3d36dcc-bsrzf\" (UID: \"e30d967b-a226-43a9-98ba-acb2bcc23e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d36dcc-bsrzf" Oct 30 20:02:32.665182 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:32.665162 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e30d967b-a226-43a9-98ba-acb2bcc23e4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3d36dcc-bsrzf\" (UID: \"e30d967b-a226-43a9-98ba-acb2bcc23e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d36dcc-bsrzf" Oct 30 20:02:32.688680 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:32.688654 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69hqp\" (UniqueName: \"kubernetes.io/projected/e30d967b-a226-43a9-98ba-acb2bcc23e4c-kube-api-access-69hqp\") pod \"maas-api-revoke-sub-e3d36dcc-bsrzf\" (UID: \"e30d967b-a226-43a9-98ba-acb2bcc23e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d36dcc-bsrzf" Oct 30 20:02:32.785943 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:32.785918 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d36dcc-bsrzf" Oct 30 20:02:33.088001 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:33.087932 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" Oct 30 20:02:33.088392 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:33.088226 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" Oct 30 20:02:33.088847 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:33.088830 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" Oct 30 20:02:33.095476 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:33.095450 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3d36dcc-bsrzf"] Oct 30 20:02:33.096592 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:02:33.096570 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode30d967b_a226_43a9_98ba_acb2bcc23e4c.slice/crio-8cf99fe48b8081316b0a5126a2ec1c49371e721ab14713310b74694aa0c4ad9e WatchSource:0}: Error finding container 8cf99fe48b8081316b0a5126a2ec1c49371e721ab14713310b74694aa0c4ad9e: Status 404 returned error can't find the container with id 8cf99fe48b8081316b0a5126a2ec1c49371e721ab14713310b74694aa0c4ad9e Oct 30 20:02:33.244490 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:33.244456 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:02:33.591845 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:33.591810 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d36dcc-bsrzf" event={"ID":"e30d967b-a226-43a9-98ba-acb2bcc23e4c","Type":"ContainerStarted","Data":"8782d39e2295528eba8400edb6578f331c4c348500de3da94aaf50029c2ab628"} Oct 30 20:02:33.592131 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:33.592101 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d36dcc-bsrzf" event={"ID":"e30d967b-a226-43a9-98ba-acb2bcc23e4c","Type":"ContainerStarted","Data":"8cf99fe48b8081316b0a5126a2ec1c49371e721ab14713310b74694aa0c4ad9e"} Oct 30 20:02:33.637373 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:33.637327 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d36dcc-bsrzf" podStartSLOduration=1.637315402 podStartE2EDuration="1.637315402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:33.637042878 +0000 UTC m=+915.049756657" watchObservedRunningTime="2026-10-30 20:02:33.637315402 +0000 UTC m=+915.050029180" Oct 30 20:02:34.639556 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:34.639518 5088 generic.go:356] "Generic (PLEG): container finished" podID="e30d967b-a226-43a9-98ba-acb2bcc23e4c" containerID="8782d39e2295528eba8400edb6578f331c4c348500de3da94aaf50029c2ab628" exitCode=0 Oct 30 20:02:34.647405 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:34.647371 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d36dcc-bsrzf" event={"ID":"e30d967b-a226-43a9-98ba-acb2bcc23e4c","Type":"ContainerDied","Data":"8782d39e2295528eba8400edb6578f331c4c348500de3da94aaf50029c2ab628"} Oct 30 20:02:35.790747 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:35.790723 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d36dcc-bsrzf" Oct 30 20:02:35.930503 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:35.930415 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e30d967b-a226-43a9-98ba-acb2bcc23e4c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e30d967b-a226-43a9-98ba-acb2bcc23e4c-maas-api-xks-ca\") pod \"e30d967b-a226-43a9-98ba-acb2bcc23e4c\" (UID: \"e30d967b-a226-43a9-98ba-acb2bcc23e4c\") " Oct 30 20:02:35.930503 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:35.930495 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e30d967b-a226-43a9-98ba-acb2bcc23e4c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e30d967b-a226-43a9-98ba-acb2bcc23e4c-maas-api-openshift-ca\") pod \"e30d967b-a226-43a9-98ba-acb2bcc23e4c\" (UID: \"e30d967b-a226-43a9-98ba-acb2bcc23e4c\") " Oct 30 20:02:35.930746 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:35.930549 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e30d967b-a226-43a9-98ba-acb2bcc23e4c-kube-api-access-69hqp\" (UniqueName: \"kubernetes.io/projected/e30d967b-a226-43a9-98ba-acb2bcc23e4c-kube-api-access-69hqp\") pod \"e30d967b-a226-43a9-98ba-acb2bcc23e4c\" (UID: \"e30d967b-a226-43a9-98ba-acb2bcc23e4c\") " Oct 30 20:02:35.930922 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:35.930897 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e30d967b-a226-43a9-98ba-acb2bcc23e4c-maas-api-openshift-ca" pod "e30d967b-a226-43a9-98ba-acb2bcc23e4c" (UID: "e30d967b-a226-43a9-98ba-acb2bcc23e4c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:35.931180 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:35.931157 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e30d967b-a226-43a9-98ba-acb2bcc23e4c-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:02:35.932658 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:35.932591 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e30d967b-a226-43a9-98ba-acb2bcc23e4c-maas-api-xks-ca" pod "e30d967b-a226-43a9-98ba-acb2bcc23e4c" (UID: "e30d967b-a226-43a9-98ba-acb2bcc23e4c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:35.932735 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:35.932679 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e30d967b-a226-43a9-98ba-acb2bcc23e4c-kube-api-access-69hqp" pod "e30d967b-a226-43a9-98ba-acb2bcc23e4c" (UID: "e30d967b-a226-43a9-98ba-acb2bcc23e4c"). InnerVolumeSpecName "kube-api-access-69hqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:36.037333 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:36.032246 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e30d967b-a226-43a9-98ba-acb2bcc23e4c-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:02:36.037333 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:36.032280 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69hqp\" (UniqueName: \"kubernetes.io/projected/e30d967b-a226-43a9-98ba-acb2bcc23e4c-kube-api-access-69hqp\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:02:36.406496 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:36.406463 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c36234df-r4njh"] Oct 30 20:02:36.407041 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:36.407026 5088 cpu_manager.go:414] "containerMap: removing container" podUID="e30d967b-a226-43a9-98ba-acb2bcc23e4c" containerName="revoke-keys" Oct 30 20:02:36.407091 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:36.407043 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e30d967b-a226-43a9-98ba-acb2bcc23e4c" containerName="revoke-keys" Oct 30 20:02:36.407143 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:36.407134 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="e30d967b-a226-43a9-98ba-acb2bcc23e4c" containerName="revoke-keys" Oct 30 20:02:36.410529 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:36.410511 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c36234df-r4njh" Oct 30 20:02:36.432649 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:36.432607 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c36234df-r4njh"] Oct 30 20:02:36.538312 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:36.538279 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77f149f0-fa20-41b0-919f-baf1395d6879-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c36234df-r4njh\" (UID: \"77f149f0-fa20-41b0-919f-baf1395d6879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c36234df-r4njh" Oct 30 20:02:36.538472 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:36.538337 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77f149f0-fa20-41b0-919f-baf1395d6879-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c36234df-r4njh\" (UID: \"77f149f0-fa20-41b0-919f-baf1395d6879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c36234df-r4njh" Oct 30 20:02:36.538539 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:36.538492 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mfz4\" (UniqueName: \"kubernetes.io/projected/77f149f0-fa20-41b0-919f-baf1395d6879-kube-api-access-7mfz4\") pod \"maas-api-revoke-sub-c36234df-r4njh\" (UID: \"77f149f0-fa20-41b0-919f-baf1395d6879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c36234df-r4njh" Oct 30 20:02:36.639604 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:36.639576 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77f149f0-fa20-41b0-919f-baf1395d6879-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c36234df-r4njh\" (UID: \"77f149f0-fa20-41b0-919f-baf1395d6879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c36234df-r4njh" Oct 30 20:02:36.639769 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:36.639645 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mfz4\" (UniqueName: \"kubernetes.io/projected/77f149f0-fa20-41b0-919f-baf1395d6879-kube-api-access-7mfz4\") pod \"maas-api-revoke-sub-c36234df-r4njh\" (UID: \"77f149f0-fa20-41b0-919f-baf1395d6879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c36234df-r4njh" Oct 30 20:02:36.639769 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:36.639745 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77f149f0-fa20-41b0-919f-baf1395d6879-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c36234df-r4njh\" (UID: \"77f149f0-fa20-41b0-919f-baf1395d6879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c36234df-r4njh" Oct 30 20:02:36.640348 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:36.640327 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77f149f0-fa20-41b0-919f-baf1395d6879-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c36234df-r4njh\" (UID: \"77f149f0-fa20-41b0-919f-baf1395d6879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c36234df-r4njh" Oct 30 20:02:36.641918 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:36.641896 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77f149f0-fa20-41b0-919f-baf1395d6879-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c36234df-r4njh\" (UID: \"77f149f0-fa20-41b0-919f-baf1395d6879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c36234df-r4njh" Oct 30 20:02:36.655165 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:36.655143 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d36dcc-bsrzf" Oct 30 20:02:36.655273 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:36.655175 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d36dcc-bsrzf" event={"ID":"e30d967b-a226-43a9-98ba-acb2bcc23e4c","Type":"ContainerDied","Data":"8cf99fe48b8081316b0a5126a2ec1c49371e721ab14713310b74694aa0c4ad9e"} Oct 30 20:02:36.655273 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:36.655224 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cf99fe48b8081316b0a5126a2ec1c49371e721ab14713310b74694aa0c4ad9e" Oct 30 20:02:36.665795 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:36.665777 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mfz4\" (UniqueName: \"kubernetes.io/projected/77f149f0-fa20-41b0-919f-baf1395d6879-kube-api-access-7mfz4\") pod \"maas-api-revoke-sub-c36234df-r4njh\" (UID: \"77f149f0-fa20-41b0-919f-baf1395d6879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c36234df-r4njh" Oct 30 20:02:36.724403 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:36.724380 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c36234df-r4njh" Oct 30 20:02:37.029212 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:37.029185 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c36234df-r4njh"] Oct 30 20:02:37.030099 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:02:37.030067 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77f149f0_fa20_41b0_919f_baf1395d6879.slice/crio-74d8b0035383f07217f014c3d786e3eb8dee91c916a74bd2a40b8193ee71a5dd WatchSource:0}: Error finding container 74d8b0035383f07217f014c3d786e3eb8dee91c916a74bd2a40b8193ee71a5dd: Status 404 returned error can't find the container with id 74d8b0035383f07217f014c3d786e3eb8dee91c916a74bd2a40b8193ee71a5dd Oct 30 20:02:37.664323 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:37.664199 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c36234df-r4njh" event={"ID":"77f149f0-fa20-41b0-919f-baf1395d6879","Type":"ContainerStarted","Data":"29568f1c1eda0617249dabb49e01503c963884e62d977c2e715649bfbadacb7b"} Oct 30 20:02:37.664323 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:37.664315 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c36234df-r4njh" event={"ID":"77f149f0-fa20-41b0-919f-baf1395d6879","Type":"ContainerStarted","Data":"74d8b0035383f07217f014c3d786e3eb8dee91c916a74bd2a40b8193ee71a5dd"} Oct 30 20:02:37.713490 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:37.713440 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c36234df-r4njh" podStartSLOduration=1.713428358 podStartE2EDuration="1.713428358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:37.712859916 +0000 UTC m=+919.125573695" watchObservedRunningTime="2026-10-30 20:02:37.713428358 +0000 UTC m=+919.126142137" Oct 30 20:02:38.671563 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:38.671524 5088 generic.go:356] "Generic (PLEG): container finished" podID="77f149f0-fa20-41b0-919f-baf1395d6879" containerID="29568f1c1eda0617249dabb49e01503c963884e62d977c2e715649bfbadacb7b" exitCode=0 Oct 30 20:02:38.671936 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:38.671607 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c36234df-r4njh" event={"ID":"77f149f0-fa20-41b0-919f-baf1395d6879","Type":"ContainerDied","Data":"29568f1c1eda0617249dabb49e01503c963884e62d977c2e715649bfbadacb7b"} Oct 30 20:02:39.914807 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:39.914786 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c36234df-r4njh" Oct 30 20:02:39.996021 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:39.995985 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77f149f0-fa20-41b0-919f-baf1395d6879-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77f149f0-fa20-41b0-919f-baf1395d6879-maas-api-xks-ca\") pod \"77f149f0-fa20-41b0-919f-baf1395d6879\" (UID: \"77f149f0-fa20-41b0-919f-baf1395d6879\") " Oct 30 20:02:39.996217 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:39.996051 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77f149f0-fa20-41b0-919f-baf1395d6879-kube-api-access-7mfz4\" (UniqueName: \"kubernetes.io/projected/77f149f0-fa20-41b0-919f-baf1395d6879-kube-api-access-7mfz4\") pod \"77f149f0-fa20-41b0-919f-baf1395d6879\" (UID: \"77f149f0-fa20-41b0-919f-baf1395d6879\") " Oct 30 20:02:39.996305 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:39.996285 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77f149f0-fa20-41b0-919f-baf1395d6879-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77f149f0-fa20-41b0-919f-baf1395d6879-maas-api-openshift-ca\") pod \"77f149f0-fa20-41b0-919f-baf1395d6879\" (UID: \"77f149f0-fa20-41b0-919f-baf1395d6879\") " Oct 30 20:02:39.996841 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:39.996815 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77f149f0-fa20-41b0-919f-baf1395d6879-maas-api-openshift-ca" pod "77f149f0-fa20-41b0-919f-baf1395d6879" (UID: "77f149f0-fa20-41b0-919f-baf1395d6879"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:39.998837 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:39.998811 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77f149f0-fa20-41b0-919f-baf1395d6879-maas-api-xks-ca" pod "77f149f0-fa20-41b0-919f-baf1395d6879" (UID: "77f149f0-fa20-41b0-919f-baf1395d6879"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:39.998959 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:39.998948 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77f149f0-fa20-41b0-919f-baf1395d6879-kube-api-access-7mfz4" pod "77f149f0-fa20-41b0-919f-baf1395d6879" (UID: "77f149f0-fa20-41b0-919f-baf1395d6879"). InnerVolumeSpecName "kube-api-access-7mfz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:40.045102 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.045073 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg"] Oct 30 20:02:40.045599 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.045585 5088 cpu_manager.go:414] "containerMap: removing container" podUID="77f149f0-fa20-41b0-919f-baf1395d6879" containerName="revoke-keys" Oct 30 20:02:40.045599 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.045599 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77f149f0-fa20-41b0-919f-baf1395d6879" containerName="revoke-keys" Oct 30 20:02:40.045742 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.045731 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="77f149f0-fa20-41b0-919f-baf1395d6879" containerName="revoke-keys" Oct 30 20:02:40.048949 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.048931 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" Oct 30 20:02:40.067881 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.067859 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5f10d2-openshift-default-dockercfg-mbtgp\"" Oct 30 20:02:40.075927 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.075901 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg"] Oct 30 20:02:40.097766 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.097739 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77f149f0-fa20-41b0-919f-baf1395d6879-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:02:40.097766 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.097764 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77f149f0-fa20-41b0-919f-baf1395d6879-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:02:40.097967 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.097775 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mfz4\" (UniqueName: \"kubernetes.io/projected/77f149f0-fa20-41b0-919f-baf1395d6879-kube-api-access-7mfz4\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:02:40.198921 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.198890 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj2vj\" (UniqueName: \"kubernetes.io/projected/419aecf7-a250-42ad-995b-e7ebd9630a76-kube-api-access-xj2vj\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg\" (UID: \"419aecf7-a250-42ad-995b-e7ebd9630a76\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" Oct 30 20:02:40.199085 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.198932 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/419aecf7-a250-42ad-995b-e7ebd9630a76-workload-certs\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg\" (UID: \"419aecf7-a250-42ad-995b-e7ebd9630a76\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" Oct 30 20:02:40.199085 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.198966 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/419aecf7-a250-42ad-995b-e7ebd9630a76-istio-envoy\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg\" (UID: \"419aecf7-a250-42ad-995b-e7ebd9630a76\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" Oct 30 20:02:40.199214 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.199171 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/419aecf7-a250-42ad-995b-e7ebd9630a76-istio-data\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg\" (UID: \"419aecf7-a250-42ad-995b-e7ebd9630a76\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" Oct 30 20:02:40.199275 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.199230 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/419aecf7-a250-42ad-995b-e7ebd9630a76-istiod-ca-cert\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg\" (UID: \"419aecf7-a250-42ad-995b-e7ebd9630a76\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" Oct 30 20:02:40.199330 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.199309 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/419aecf7-a250-42ad-995b-e7ebd9630a76-istio-token\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg\" (UID: \"419aecf7-a250-42ad-995b-e7ebd9630a76\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" Oct 30 20:02:40.199385 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.199341 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/419aecf7-a250-42ad-995b-e7ebd9630a76-workload-socket\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg\" (UID: \"419aecf7-a250-42ad-995b-e7ebd9630a76\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" Oct 30 20:02:40.199385 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.199374 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/419aecf7-a250-42ad-995b-e7ebd9630a76-credential-socket\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg\" (UID: \"419aecf7-a250-42ad-995b-e7ebd9630a76\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" Oct 30 20:02:40.199464 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.199446 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/419aecf7-a250-42ad-995b-e7ebd9630a76-istio-podinfo\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg\" (UID: \"419aecf7-a250-42ad-995b-e7ebd9630a76\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" Oct 30 20:02:40.300185 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.300108 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/419aecf7-a250-42ad-995b-e7ebd9630a76-istio-token\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg\" (UID: \"419aecf7-a250-42ad-995b-e7ebd9630a76\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" Oct 30 20:02:40.300185 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.300145 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/419aecf7-a250-42ad-995b-e7ebd9630a76-workload-socket\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg\" (UID: \"419aecf7-a250-42ad-995b-e7ebd9630a76\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" Oct 30 20:02:40.300185 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.300166 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/419aecf7-a250-42ad-995b-e7ebd9630a76-credential-socket\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg\" (UID: \"419aecf7-a250-42ad-995b-e7ebd9630a76\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" Oct 30 20:02:40.300466 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.300253 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/419aecf7-a250-42ad-995b-e7ebd9630a76-istio-podinfo\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg\" (UID: \"419aecf7-a250-42ad-995b-e7ebd9630a76\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" Oct 30 20:02:40.300871 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.300772 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj2vj\" (UniqueName: \"kubernetes.io/projected/419aecf7-a250-42ad-995b-e7ebd9630a76-kube-api-access-xj2vj\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg\" (UID: \"419aecf7-a250-42ad-995b-e7ebd9630a76\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" Oct 30 20:02:40.300871 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.300821 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/419aecf7-a250-42ad-995b-e7ebd9630a76-workload-socket\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg\" (UID: \"419aecf7-a250-42ad-995b-e7ebd9630a76\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" Oct 30 20:02:40.300871 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.300845 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/419aecf7-a250-42ad-995b-e7ebd9630a76-workload-certs\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg\" (UID: \"419aecf7-a250-42ad-995b-e7ebd9630a76\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" Oct 30 20:02:40.301104 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.300927 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/419aecf7-a250-42ad-995b-e7ebd9630a76-istio-envoy\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg\" (UID: \"419aecf7-a250-42ad-995b-e7ebd9630a76\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" Oct 30 20:02:40.301104 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.300932 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/419aecf7-a250-42ad-995b-e7ebd9630a76-credential-socket\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg\" (UID: \"419aecf7-a250-42ad-995b-e7ebd9630a76\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" Oct 30 20:02:40.301104 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.301059 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/419aecf7-a250-42ad-995b-e7ebd9630a76-istio-data\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg\" (UID: \"419aecf7-a250-42ad-995b-e7ebd9630a76\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" Oct 30 20:02:40.301244 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.301116 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/419aecf7-a250-42ad-995b-e7ebd9630a76-istiod-ca-cert\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg\" (UID: \"419aecf7-a250-42ad-995b-e7ebd9630a76\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" Oct 30 20:02:40.301653 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.301422 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/419aecf7-a250-42ad-995b-e7ebd9630a76-workload-certs\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg\" (UID: \"419aecf7-a250-42ad-995b-e7ebd9630a76\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" Oct 30 20:02:40.301759 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.301737 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/419aecf7-a250-42ad-995b-e7ebd9630a76-istio-data\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg\" (UID: \"419aecf7-a250-42ad-995b-e7ebd9630a76\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" Oct 30 20:02:40.302030 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.302007 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/419aecf7-a250-42ad-995b-e7ebd9630a76-istiod-ca-cert\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg\" (UID: \"419aecf7-a250-42ad-995b-e7ebd9630a76\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" Oct 30 20:02:40.304549 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.304015 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/419aecf7-a250-42ad-995b-e7ebd9630a76-istio-envoy\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg\" (UID: \"419aecf7-a250-42ad-995b-e7ebd9630a76\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" Oct 30 20:02:40.304549 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.304469 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/419aecf7-a250-42ad-995b-e7ebd9630a76-istio-podinfo\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg\" (UID: \"419aecf7-a250-42ad-995b-e7ebd9630a76\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" Oct 30 20:02:40.327186 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.327159 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj2vj\" (UniqueName: \"kubernetes.io/projected/419aecf7-a250-42ad-995b-e7ebd9630a76-kube-api-access-xj2vj\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg\" (UID: \"419aecf7-a250-42ad-995b-e7ebd9630a76\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" Oct 30 20:02:40.332552 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.332506 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/419aecf7-a250-42ad-995b-e7ebd9630a76-istio-token\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg\" (UID: \"419aecf7-a250-42ad-995b-e7ebd9630a76\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" Oct 30 20:02:40.363396 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.363374 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" Oct 30 20:02:40.685064 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.685021 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg"] Oct 30 20:02:40.686569 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:02:40.686543 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod419aecf7_a250_42ad_995b_e7ebd9630a76.slice/crio-a4a188e6739c6e39675d0b3e5a71629da637b3589df8952dfe382ae853ec085c WatchSource:0}: Error finding container a4a188e6739c6e39675d0b3e5a71629da637b3589df8952dfe382ae853ec085c: Status 404 returned error can't find the container with id a4a188e6739c6e39675d0b3e5a71629da637b3589df8952dfe382ae853ec085c Oct 30 20:02:40.688937 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.688912 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c36234df-r4njh" Oct 30 20:02:40.689044 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.688913 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c36234df-r4njh" event={"ID":"77f149f0-fa20-41b0-919f-baf1395d6879","Type":"ContainerDied","Data":"74d8b0035383f07217f014c3d786e3eb8dee91c916a74bd2a40b8193ee71a5dd"} Oct 30 20:02:40.689102 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.689081 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74d8b0035383f07217f014c3d786e3eb8dee91c916a74bd2a40b8193ee71a5dd" Oct 30 20:02:40.689430 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.689401 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:02:40.689502 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.689474 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:02:40.689564 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:40.689521 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:02:41.736466 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:41.736417 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" event={"ID":"419aecf7-a250-42ad-995b-e7ebd9630a76","Type":"ContainerStarted","Data":"b05e9b0d77de96b9f1159bdf32750a69a00a0f90e63cece81107203cb2040bc2"} Oct 30 20:02:41.737138 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:41.736479 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" event={"ID":"419aecf7-a250-42ad-995b-e7ebd9630a76","Type":"ContainerStarted","Data":"a4a188e6739c6e39675d0b3e5a71629da637b3589df8952dfe382ae853ec085c"} Oct 30 20:02:41.825301 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:41.825245 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" podStartSLOduration=1.825231827 podStartE2EDuration="1.825231827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:41.80911713 +0000 UTC m=+923.221830909" watchObservedRunningTime="2026-10-30 20:02:41.825231827 +0000 UTC m=+923.237945605" Oct 30 20:02:42.363908 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:42.363879 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" Oct 30 20:02:42.365586 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:42.365565 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 20:02:42.365657 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:42.365625 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" podUID="419aecf7-a250-42ad-995b-e7ebd9630a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 20:02:42.801819 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:42.801785 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6237426-dcvxw"] Oct 30 20:02:42.810211 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:42.810185 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6237426-dcvxw" Oct 30 20:02:42.818717 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:42.818679 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6237426-dcvxw"] Oct 30 20:02:42.832636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:42.828332 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:02:42.832636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:42.828359 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:02:42.939152 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:42.939123 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb43b116-cc3d-4d6b-a0c0-06dd170822ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6237426-dcvxw\" (UID: \"fb43b116-cc3d-4d6b-a0c0-06dd170822ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6237426-dcvxw" Oct 30 20:02:42.939311 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:42.939168 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb43b116-cc3d-4d6b-a0c0-06dd170822ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6237426-dcvxw\" (UID: \"fb43b116-cc3d-4d6b-a0c0-06dd170822ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6237426-dcvxw" Oct 30 20:02:42.939373 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:42.939347 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spvj6\" (UniqueName: \"kubernetes.io/projected/fb43b116-cc3d-4d6b-a0c0-06dd170822ed-kube-api-access-spvj6\") pod \"maas-api-revoke-sub-d6237426-dcvxw\" (UID: \"fb43b116-cc3d-4d6b-a0c0-06dd170822ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6237426-dcvxw" Oct 30 20:02:43.046637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:43.041327 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb43b116-cc3d-4d6b-a0c0-06dd170822ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6237426-dcvxw\" (UID: \"fb43b116-cc3d-4d6b-a0c0-06dd170822ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6237426-dcvxw" Oct 30 20:02:43.046637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:43.041402 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb43b116-cc3d-4d6b-a0c0-06dd170822ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6237426-dcvxw\" (UID: \"fb43b116-cc3d-4d6b-a0c0-06dd170822ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6237426-dcvxw" Oct 30 20:02:43.046637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:43.041539 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spvj6\" (UniqueName: \"kubernetes.io/projected/fb43b116-cc3d-4d6b-a0c0-06dd170822ed-kube-api-access-spvj6\") pod \"maas-api-revoke-sub-d6237426-dcvxw\" (UID: \"fb43b116-cc3d-4d6b-a0c0-06dd170822ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6237426-dcvxw" Oct 30 20:02:43.068682 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:43.053627 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb43b116-cc3d-4d6b-a0c0-06dd170822ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6237426-dcvxw\" (UID: \"fb43b116-cc3d-4d6b-a0c0-06dd170822ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6237426-dcvxw" Oct 30 20:02:43.094637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:43.092324 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spvj6\" (UniqueName: \"kubernetes.io/projected/fb43b116-cc3d-4d6b-a0c0-06dd170822ed-kube-api-access-spvj6\") pod \"maas-api-revoke-sub-d6237426-dcvxw\" (UID: \"fb43b116-cc3d-4d6b-a0c0-06dd170822ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6237426-dcvxw" Oct 30 20:02:43.114236 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:43.109065 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb43b116-cc3d-4d6b-a0c0-06dd170822ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6237426-dcvxw\" (UID: \"fb43b116-cc3d-4d6b-a0c0-06dd170822ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6237426-dcvxw" Oct 30 20:02:43.134633 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:43.132914 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6237426-dcvxw" Oct 30 20:02:43.475138 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:43.473935 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 20:02:43.475138 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:43.473993 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" podUID="419aecf7-a250-42ad-995b-e7ebd9630a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 20:02:43.933635 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:43.928001 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6237426-dcvxw"] Oct 30 20:02:44.366372 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:44.366338 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 20:02:44.366639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:44.366592 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" podUID="419aecf7-a250-42ad-995b-e7ebd9630a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 20:02:44.771360 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:44.771324 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6237426-dcvxw" event={"ID":"fb43b116-cc3d-4d6b-a0c0-06dd170822ed","Type":"ContainerStarted","Data":"c7abd59e44ced870d8ebb875bfa37f908ddd0b5eb248a581e9bf77ac41ef1c8e"} Oct 30 20:02:44.771525 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:44.771389 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6237426-dcvxw" event={"ID":"fb43b116-cc3d-4d6b-a0c0-06dd170822ed","Type":"ContainerStarted","Data":"0b5e9970288a4806232388f95878d818f5afe46b15e83c39ae2011d23d229c9d"} Oct 30 20:02:44.839255 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:44.839200 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6237426-dcvxw" podStartSLOduration=2.839184306 podStartE2EDuration="2.839184306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:44.8377963 +0000 UTC m=+926.250510098" watchObservedRunningTime="2026-10-30 20:02:44.839184306 +0000 UTC m=+926.251898079" Oct 30 20:02:45.364124 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:45.364094 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 20:02:45.364538 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:45.364152 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" podUID="419aecf7-a250-42ad-995b-e7ebd9630a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 20:02:45.778223 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:45.778190 5088 generic.go:356] "Generic (PLEG): container finished" podID="fb43b116-cc3d-4d6b-a0c0-06dd170822ed" containerID="c7abd59e44ced870d8ebb875bfa37f908ddd0b5eb248a581e9bf77ac41ef1c8e" exitCode=0 Oct 30 20:02:45.778407 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:45.778264 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6237426-dcvxw" event={"ID":"fb43b116-cc3d-4d6b-a0c0-06dd170822ed","Type":"ContainerDied","Data":"c7abd59e44ced870d8ebb875bfa37f908ddd0b5eb248a581e9bf77ac41ef1c8e"} Oct 30 20:02:46.364520 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:46.364487 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 20:02:46.364921 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:46.364538 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" podUID="419aecf7-a250-42ad-995b-e7ebd9630a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 20:02:46.779186 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:46.779152 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41ff877d-ps998"] Oct 30 20:02:46.783438 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:46.783415 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ff877d-ps998" Oct 30 20:02:46.815183 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:46.815151 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41ff877d-ps998"] Oct 30 20:02:46.916069 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:46.915974 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpcmg\" (UniqueName: \"kubernetes.io/projected/d8d2c833-630a-4bcd-a943-f576ca32d634-kube-api-access-xpcmg\") pod \"maas-api-revoke-sub-41ff877d-ps998\" (UID: \"d8d2c833-630a-4bcd-a943-f576ca32d634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ff877d-ps998" Oct 30 20:02:46.916069 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:46.916042 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8d2c833-630a-4bcd-a943-f576ca32d634-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41ff877d-ps998\" (UID: \"d8d2c833-630a-4bcd-a943-f576ca32d634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ff877d-ps998" Oct 30 20:02:46.916325 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:46.916180 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8d2c833-630a-4bcd-a943-f576ca32d634-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41ff877d-ps998\" (UID: \"d8d2c833-630a-4bcd-a943-f576ca32d634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ff877d-ps998" Oct 30 20:02:47.017218 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:47.017186 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpcmg\" (UniqueName: \"kubernetes.io/projected/d8d2c833-630a-4bcd-a943-f576ca32d634-kube-api-access-xpcmg\") pod \"maas-api-revoke-sub-41ff877d-ps998\" (UID: \"d8d2c833-630a-4bcd-a943-f576ca32d634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ff877d-ps998" Oct 30 20:02:47.017392 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:47.017247 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8d2c833-630a-4bcd-a943-f576ca32d634-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41ff877d-ps998\" (UID: \"d8d2c833-630a-4bcd-a943-f576ca32d634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ff877d-ps998" Oct 30 20:02:47.017392 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:47.017307 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8d2c833-630a-4bcd-a943-f576ca32d634-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41ff877d-ps998\" (UID: \"d8d2c833-630a-4bcd-a943-f576ca32d634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ff877d-ps998" Oct 30 20:02:47.018340 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:47.018313 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8d2c833-630a-4bcd-a943-f576ca32d634-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41ff877d-ps998\" (UID: \"d8d2c833-630a-4bcd-a943-f576ca32d634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ff877d-ps998" Oct 30 20:02:47.021635 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:47.020234 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8d2c833-630a-4bcd-a943-f576ca32d634-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41ff877d-ps998\" (UID: \"d8d2c833-630a-4bcd-a943-f576ca32d634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ff877d-ps998" Oct 30 20:02:47.031198 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:47.031145 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6237426-dcvxw" Oct 30 20:02:47.043707 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:47.043685 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpcmg\" (UniqueName: \"kubernetes.io/projected/d8d2c833-630a-4bcd-a943-f576ca32d634-kube-api-access-xpcmg\") pod \"maas-api-revoke-sub-41ff877d-ps998\" (UID: \"d8d2c833-630a-4bcd-a943-f576ca32d634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ff877d-ps998" Oct 30 20:02:47.118467 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:47.118421 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb43b116-cc3d-4d6b-a0c0-06dd170822ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb43b116-cc3d-4d6b-a0c0-06dd170822ed-maas-api-xks-ca\") pod \"fb43b116-cc3d-4d6b-a0c0-06dd170822ed\" (UID: \"fb43b116-cc3d-4d6b-a0c0-06dd170822ed\") " Oct 30 20:02:47.118710 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:47.118688 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb43b116-cc3d-4d6b-a0c0-06dd170822ed-kube-api-access-spvj6\" (UniqueName: \"kubernetes.io/projected/fb43b116-cc3d-4d6b-a0c0-06dd170822ed-kube-api-access-spvj6\") pod \"fb43b116-cc3d-4d6b-a0c0-06dd170822ed\" (UID: \"fb43b116-cc3d-4d6b-a0c0-06dd170822ed\") " Oct 30 20:02:47.118799 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:47.118748 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb43b116-cc3d-4d6b-a0c0-06dd170822ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb43b116-cc3d-4d6b-a0c0-06dd170822ed-maas-api-openshift-ca\") pod \"fb43b116-cc3d-4d6b-a0c0-06dd170822ed\" (UID: \"fb43b116-cc3d-4d6b-a0c0-06dd170822ed\") " Oct 30 20:02:47.119123 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:47.119094 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb43b116-cc3d-4d6b-a0c0-06dd170822ed-maas-api-openshift-ca" pod "fb43b116-cc3d-4d6b-a0c0-06dd170822ed" (UID: "fb43b116-cc3d-4d6b-a0c0-06dd170822ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:47.119293 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:47.119269 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb43b116-cc3d-4d6b-a0c0-06dd170822ed-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:02:47.120582 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:47.120558 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb43b116-cc3d-4d6b-a0c0-06dd170822ed-maas-api-xks-ca" pod "fb43b116-cc3d-4d6b-a0c0-06dd170822ed" (UID: "fb43b116-cc3d-4d6b-a0c0-06dd170822ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:47.120971 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:47.120953 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb43b116-cc3d-4d6b-a0c0-06dd170822ed-kube-api-access-spvj6" pod "fb43b116-cc3d-4d6b-a0c0-06dd170822ed" (UID: "fb43b116-cc3d-4d6b-a0c0-06dd170822ed"). InnerVolumeSpecName "kube-api-access-spvj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:47.125728 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:47.125408 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ff877d-ps998" Oct 30 20:02:47.220510 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:47.220476 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb43b116-cc3d-4d6b-a0c0-06dd170822ed-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:02:47.220510 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:47.220506 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spvj6\" (UniqueName: \"kubernetes.io/projected/fb43b116-cc3d-4d6b-a0c0-06dd170822ed-kube-api-access-spvj6\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:02:47.364180 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:47.364094 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 20:02:47.364180 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:47.364161 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" podUID="419aecf7-a250-42ad-995b-e7ebd9630a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 20:02:47.436529 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:47.436500 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41ff877d-ps998"] Oct 30 20:02:47.437156 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:02:47.437133 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8d2c833_630a_4bcd_a943_f576ca32d634.slice/crio-c1dcc90ed4757ed22f0ec8b9b08d43f6a6dba2e3ef54b816c2255b086784a12a WatchSource:0}: Error finding container c1dcc90ed4757ed22f0ec8b9b08d43f6a6dba2e3ef54b816c2255b086784a12a: Status 404 returned error can't find the container with id c1dcc90ed4757ed22f0ec8b9b08d43f6a6dba2e3ef54b816c2255b086784a12a Oct 30 20:02:47.814210 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:47.814180 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6237426-dcvxw" Oct 30 20:02:47.814417 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:47.814227 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6237426-dcvxw" event={"ID":"fb43b116-cc3d-4d6b-a0c0-06dd170822ed","Type":"ContainerDied","Data":"0b5e9970288a4806232388f95878d818f5afe46b15e83c39ae2011d23d229c9d"} Oct 30 20:02:47.814417 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:47.814283 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b5e9970288a4806232388f95878d818f5afe46b15e83c39ae2011d23d229c9d" Oct 30 20:02:47.816218 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:47.816188 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ff877d-ps998" event={"ID":"d8d2c833-630a-4bcd-a943-f576ca32d634","Type":"ContainerStarted","Data":"7973cb77382d83ca511eaaaa0c9f731f93cb413a97a9ccacff841279a2228925"} Oct 30 20:02:47.816354 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:47.816237 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ff877d-ps998" event={"ID":"d8d2c833-630a-4bcd-a943-f576ca32d634","Type":"ContainerStarted","Data":"c1dcc90ed4757ed22f0ec8b9b08d43f6a6dba2e3ef54b816c2255b086784a12a"} Oct 30 20:02:47.863837 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:47.863780 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ff877d-ps998" podStartSLOduration=1.863762694 podStartE2EDuration="1.863762694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:47.861021452 +0000 UTC m=+929.273735255" watchObservedRunningTime="2026-10-30 20:02:47.863762694 +0000 UTC m=+929.276476474" Oct 30 20:02:48.752909 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:48.752881 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" Oct 30 20:02:48.753358 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:48.753266 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" Oct 30 20:02:48.754801 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:48.754775 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" Oct 30 20:02:48.823226 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:48.823195 5088 generic.go:356] "Generic (PLEG): container finished" podID="d8d2c833-630a-4bcd-a943-f576ca32d634" containerID="7973cb77382d83ca511eaaaa0c9f731f93cb413a97a9ccacff841279a2228925" exitCode=0 Oct 30 20:02:48.823369 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:48.823266 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ff877d-ps998" event={"ID":"d8d2c833-630a-4bcd-a943-f576ca32d634","Type":"ContainerDied","Data":"7973cb77382d83ca511eaaaa0c9f731f93cb413a97a9ccacff841279a2228925"} Oct 30 20:02:49.978795 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:49.978774 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ff877d-ps998" Oct 30 20:02:50.162137 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:50.162065 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8d2c833-630a-4bcd-a943-f576ca32d634-kube-api-access-xpcmg\" (UniqueName: \"kubernetes.io/projected/d8d2c833-630a-4bcd-a943-f576ca32d634-kube-api-access-xpcmg\") pod \"d8d2c833-630a-4bcd-a943-f576ca32d634\" (UID: \"d8d2c833-630a-4bcd-a943-f576ca32d634\") " Oct 30 20:02:50.162137 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:50.162116 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8d2c833-630a-4bcd-a943-f576ca32d634-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8d2c833-630a-4bcd-a943-f576ca32d634-maas-api-xks-ca\") pod \"d8d2c833-630a-4bcd-a943-f576ca32d634\" (UID: \"d8d2c833-630a-4bcd-a943-f576ca32d634\") " Oct 30 20:02:50.162350 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:50.162154 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8d2c833-630a-4bcd-a943-f576ca32d634-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8d2c833-630a-4bcd-a943-f576ca32d634-maas-api-openshift-ca\") pod \"d8d2c833-630a-4bcd-a943-f576ca32d634\" (UID: \"d8d2c833-630a-4bcd-a943-f576ca32d634\") " Oct 30 20:02:50.162570 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:50.162545 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8d2c833-630a-4bcd-a943-f576ca32d634-maas-api-openshift-ca" pod "d8d2c833-630a-4bcd-a943-f576ca32d634" (UID: "d8d2c833-630a-4bcd-a943-f576ca32d634"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:50.164113 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:50.164094 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8d2c833-630a-4bcd-a943-f576ca32d634-maas-api-xks-ca" pod "d8d2c833-630a-4bcd-a943-f576ca32d634" (UID: "d8d2c833-630a-4bcd-a943-f576ca32d634"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:50.164203 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:50.164185 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8d2c833-630a-4bcd-a943-f576ca32d634-kube-api-access-xpcmg" pod "d8d2c833-630a-4bcd-a943-f576ca32d634" (UID: "d8d2c833-630a-4bcd-a943-f576ca32d634"). InnerVolumeSpecName "kube-api-access-xpcmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:50.263585 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:50.263560 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8d2c833-630a-4bcd-a943-f576ca32d634-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:02:50.263585 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:50.263583 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8d2c833-630a-4bcd-a943-f576ca32d634-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:02:50.263745 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:50.263592 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpcmg\" (UniqueName: \"kubernetes.io/projected/d8d2c833-630a-4bcd-a943-f576ca32d634-kube-api-access-xpcmg\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:02:50.835819 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:50.835790 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ff877d-ps998" Oct 30 20:02:50.835989 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:50.835825 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ff877d-ps998" event={"ID":"d8d2c833-630a-4bcd-a943-f576ca32d634","Type":"ContainerDied","Data":"c1dcc90ed4757ed22f0ec8b9b08d43f6a6dba2e3ef54b816c2255b086784a12a"} Oct 30 20:02:50.835989 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:50.835872 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1dcc90ed4757ed22f0ec8b9b08d43f6a6dba2e3ef54b816c2255b086784a12a" Oct 30 20:02:52.151126 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:52.151097 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:02:52.437299 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:02:52.437218 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:02:52.515976 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:52.515937 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a199d8f-hqq5n"] Oct 30 20:02:52.517346 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:52.517318 5088 cpu_manager.go:414] "containerMap: removing container" podUID="fb43b116-cc3d-4d6b-a0c0-06dd170822ed" containerName="revoke-keys" Oct 30 20:02:52.517346 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:52.517344 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb43b116-cc3d-4d6b-a0c0-06dd170822ed" containerName="revoke-keys" Oct 30 20:02:52.517532 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:52.517378 5088 cpu_manager.go:414] "containerMap: removing container" podUID="d8d2c833-630a-4bcd-a943-f576ca32d634" containerName="revoke-keys" Oct 30 20:02:52.517532 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:52.517385 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8d2c833-630a-4bcd-a943-f576ca32d634" containerName="revoke-keys" Oct 30 20:02:52.517653 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:52.517637 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8d2c833-630a-4bcd-a943-f576ca32d634" containerName="revoke-keys" Oct 30 20:02:52.517710 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:52.517670 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb43b116-cc3d-4d6b-a0c0-06dd170822ed" containerName="revoke-keys" Oct 30 20:02:52.526050 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:52.526028 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a199d8f-hqq5n" Oct 30 20:02:52.550678 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:52.550650 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a199d8f-hqq5n"] Oct 30 20:02:52.559755 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:52.559731 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:02:52.559904 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:52.559869 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:02:52.585337 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:52.585308 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a6dccf5-7f1b-42c8-a974-1cf5510af915-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a199d8f-hqq5n\" (UID: \"7a6dccf5-7f1b-42c8-a974-1cf5510af915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a199d8f-hqq5n" Oct 30 20:02:52.585496 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:52.585364 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a6dccf5-7f1b-42c8-a974-1cf5510af915-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a199d8f-hqq5n\" (UID: \"7a6dccf5-7f1b-42c8-a974-1cf5510af915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a199d8f-hqq5n" Oct 30 20:02:52.585496 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:52.585477 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg45g\" (UniqueName: \"kubernetes.io/projected/7a6dccf5-7f1b-42c8-a974-1cf5510af915-kube-api-access-hg45g\") pod \"maas-api-revoke-sub-9a199d8f-hqq5n\" (UID: \"7a6dccf5-7f1b-42c8-a974-1cf5510af915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a199d8f-hqq5n" Oct 30 20:02:52.686217 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:52.686177 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a6dccf5-7f1b-42c8-a974-1cf5510af915-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a199d8f-hqq5n\" (UID: \"7a6dccf5-7f1b-42c8-a974-1cf5510af915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a199d8f-hqq5n" Oct 30 20:02:52.686404 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:52.686232 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg45g\" (UniqueName: \"kubernetes.io/projected/7a6dccf5-7f1b-42c8-a974-1cf5510af915-kube-api-access-hg45g\") pod \"maas-api-revoke-sub-9a199d8f-hqq5n\" (UID: \"7a6dccf5-7f1b-42c8-a974-1cf5510af915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a199d8f-hqq5n" Oct 30 20:02:52.686485 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:52.686410 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a6dccf5-7f1b-42c8-a974-1cf5510af915-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a199d8f-hqq5n\" (UID: \"7a6dccf5-7f1b-42c8-a974-1cf5510af915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a199d8f-hqq5n" Oct 30 20:02:52.687070 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:52.687047 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a6dccf5-7f1b-42c8-a974-1cf5510af915-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a199d8f-hqq5n\" (UID: \"7a6dccf5-7f1b-42c8-a974-1cf5510af915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a199d8f-hqq5n" Oct 30 20:02:52.688967 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:52.688906 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a6dccf5-7f1b-42c8-a974-1cf5510af915-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a199d8f-hqq5n\" (UID: \"7a6dccf5-7f1b-42c8-a974-1cf5510af915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a199d8f-hqq5n" Oct 30 20:02:52.712739 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:52.712716 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg45g\" (UniqueName: \"kubernetes.io/projected/7a6dccf5-7f1b-42c8-a974-1cf5510af915-kube-api-access-hg45g\") pod \"maas-api-revoke-sub-9a199d8f-hqq5n\" (UID: \"7a6dccf5-7f1b-42c8-a974-1cf5510af915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a199d8f-hqq5n" Oct 30 20:02:52.842633 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:52.841836 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a199d8f-hqq5n" Oct 30 20:02:53.580035 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:53.579907 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a199d8f-hqq5n"] Oct 30 20:02:53.680128 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:02:53.680078 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a6dccf5_7f1b_42c8_a974_1cf5510af915.slice/crio-a8411d094d5ac8b2f4d7bc58ca11f27f04d82261d098ce915ba1599af18b0ff6 WatchSource:0}: Error finding container a8411d094d5ac8b2f4d7bc58ca11f27f04d82261d098ce915ba1599af18b0ff6: Status 404 returned error can't find the container with id a8411d094d5ac8b2f4d7bc58ca11f27f04d82261d098ce915ba1599af18b0ff6 Oct 30 20:02:53.913903 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:53.913817 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a199d8f-hqq5n" event={"ID":"7a6dccf5-7f1b-42c8-a974-1cf5510af915","Type":"ContainerStarted","Data":"a8411d094d5ac8b2f4d7bc58ca11f27f04d82261d098ce915ba1599af18b0ff6"} Oct 30 20:02:54.760636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:54.759729 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5"] Oct 30 20:02:54.769646 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:54.768950 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5"] Oct 30 20:02:54.769646 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:54.769112 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" Oct 30 20:02:55.010029 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:55.009980 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf9a8162-01f0-4226-b8f4-783125c277a2-istio-data\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5\" (UID: \"cf9a8162-01f0-4226-b8f4-783125c277a2\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" Oct 30 20:02:55.020697 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:55.017667 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a199d8f-hqq5n" event={"ID":"7a6dccf5-7f1b-42c8-a974-1cf5510af915","Type":"ContainerStarted","Data":"5bebaecf163264904fdb275b0369c6f6a91e01b5cc9e2c91094ea9efd8d92def"} Oct 30 20:02:55.021009 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:55.020982 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf9a8162-01f0-4226-b8f4-783125c277a2-workload-certs\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5\" (UID: \"cf9a8162-01f0-4226-b8f4-783125c277a2\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" Oct 30 20:02:55.021278 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:55.021172 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9a8162-01f0-4226-b8f4-783125c277a2-credential-socket\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5\" (UID: \"cf9a8162-01f0-4226-b8f4-783125c277a2\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" Oct 30 20:02:55.021400 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:55.021387 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf9a8162-01f0-4226-b8f4-783125c277a2-istio-envoy\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5\" (UID: \"cf9a8162-01f0-4226-b8f4-783125c277a2\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" Oct 30 20:02:55.038922 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:55.038796 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf9a8162-01f0-4226-b8f4-783125c277a2-istio-podinfo\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5\" (UID: \"cf9a8162-01f0-4226-b8f4-783125c277a2\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" Oct 30 20:02:55.039159 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:55.039138 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf9a8162-01f0-4226-b8f4-783125c277a2-istiod-ca-cert\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5\" (UID: \"cf9a8162-01f0-4226-b8f4-783125c277a2\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" Oct 30 20:02:55.042280 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:55.042239 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggqxg\" (UniqueName: \"kubernetes.io/projected/cf9a8162-01f0-4226-b8f4-783125c277a2-kube-api-access-ggqxg\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5\" (UID: \"cf9a8162-01f0-4226-b8f4-783125c277a2\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" Oct 30 20:02:55.042603 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:55.042573 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf9a8162-01f0-4226-b8f4-783125c277a2-istio-token\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5\" (UID: \"cf9a8162-01f0-4226-b8f4-783125c277a2\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" Oct 30 20:02:55.043020 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:55.042996 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9a8162-01f0-4226-b8f4-783125c277a2-workload-socket\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5\" (UID: \"cf9a8162-01f0-4226-b8f4-783125c277a2\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" Oct 30 20:02:55.145714 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:55.145192 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9a8162-01f0-4226-b8f4-783125c277a2-workload-socket\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5\" (UID: \"cf9a8162-01f0-4226-b8f4-783125c277a2\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" Oct 30 20:02:55.145714 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:55.145418 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf9a8162-01f0-4226-b8f4-783125c277a2-istio-data\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5\" (UID: \"cf9a8162-01f0-4226-b8f4-783125c277a2\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" Oct 30 20:02:55.145714 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:55.145515 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf9a8162-01f0-4226-b8f4-783125c277a2-workload-certs\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5\" (UID: \"cf9a8162-01f0-4226-b8f4-783125c277a2\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" Oct 30 20:02:55.145714 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:55.145558 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9a8162-01f0-4226-b8f4-783125c277a2-credential-socket\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5\" (UID: \"cf9a8162-01f0-4226-b8f4-783125c277a2\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" Oct 30 20:02:55.145714 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:55.145585 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf9a8162-01f0-4226-b8f4-783125c277a2-istio-envoy\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5\" (UID: \"cf9a8162-01f0-4226-b8f4-783125c277a2\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" Oct 30 20:02:55.145714 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:55.145637 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf9a8162-01f0-4226-b8f4-783125c277a2-istio-podinfo\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5\" (UID: \"cf9a8162-01f0-4226-b8f4-783125c277a2\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" Oct 30 20:02:55.145714 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:55.145661 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf9a8162-01f0-4226-b8f4-783125c277a2-istiod-ca-cert\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5\" (UID: \"cf9a8162-01f0-4226-b8f4-783125c277a2\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" Oct 30 20:02:55.145714 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:55.145688 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggqxg\" (UniqueName: \"kubernetes.io/projected/cf9a8162-01f0-4226-b8f4-783125c277a2-kube-api-access-ggqxg\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5\" (UID: \"cf9a8162-01f0-4226-b8f4-783125c277a2\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" Oct 30 20:02:55.145714 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:55.145724 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf9a8162-01f0-4226-b8f4-783125c277a2-istio-token\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5\" (UID: \"cf9a8162-01f0-4226-b8f4-783125c277a2\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" Oct 30 20:02:55.149666 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:55.146459 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9a8162-01f0-4226-b8f4-783125c277a2-workload-socket\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5\" (UID: \"cf9a8162-01f0-4226-b8f4-783125c277a2\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" Oct 30 20:02:55.149666 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:55.147409 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf9a8162-01f0-4226-b8f4-783125c277a2-istio-data\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5\" (UID: \"cf9a8162-01f0-4226-b8f4-783125c277a2\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" Oct 30 20:02:55.149666 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:55.147749 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf9a8162-01f0-4226-b8f4-783125c277a2-workload-certs\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5\" (UID: \"cf9a8162-01f0-4226-b8f4-783125c277a2\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" Oct 30 20:02:55.149666 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:55.148026 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9a8162-01f0-4226-b8f4-783125c277a2-credential-socket\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5\" (UID: \"cf9a8162-01f0-4226-b8f4-783125c277a2\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" Oct 30 20:02:55.149666 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:55.149012 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf9a8162-01f0-4226-b8f4-783125c277a2-istiod-ca-cert\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5\" (UID: \"cf9a8162-01f0-4226-b8f4-783125c277a2\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" Oct 30 20:02:55.168632 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:55.165001 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf9a8162-01f0-4226-b8f4-783125c277a2-istio-envoy\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5\" (UID: \"cf9a8162-01f0-4226-b8f4-783125c277a2\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" Oct 30 20:02:55.169639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:55.168916 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf9a8162-01f0-4226-b8f4-783125c277a2-istio-podinfo\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5\" (UID: \"cf9a8162-01f0-4226-b8f4-783125c277a2\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" Oct 30 20:02:55.174926 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:55.174904 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf9a8162-01f0-4226-b8f4-783125c277a2-istio-token\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5\" (UID: \"cf9a8162-01f0-4226-b8f4-783125c277a2\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" Oct 30 20:02:55.183023 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:55.182994 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggqxg\" (UniqueName: \"kubernetes.io/projected/cf9a8162-01f0-4226-b8f4-783125c277a2-kube-api-access-ggqxg\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5\" (UID: \"cf9a8162-01f0-4226-b8f4-783125c277a2\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" Oct 30 20:02:55.472332 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:55.472240 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" Oct 30 20:02:55.966810 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:55.966604 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a199d8f-hqq5n" podStartSLOduration=3.96658332 podStartE2EDuration="3.96658332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:55.075300371 +0000 UTC m=+936.488014171" watchObservedRunningTime="2026-10-30 20:02:55.96658332 +0000 UTC m=+937.379297100" Oct 30 20:02:55.967183 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:55.966992 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5"] Oct 30 20:02:55.968133 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:02:55.968094 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf9a8162_01f0_4226_b8f4_783125c277a2.slice/crio-38c6e49fb5aea702a7e3e884decb08e20b853a19e9cbb2319942d6a927ef2598 WatchSource:0}: Error finding container 38c6e49fb5aea702a7e3e884decb08e20b853a19e9cbb2319942d6a927ef2598: Status 404 returned error can't find the container with id 38c6e49fb5aea702a7e3e884decb08e20b853a19e9cbb2319942d6a927ef2598 Oct 30 20:02:55.971525 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:55.971146 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:02:55.971525 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:55.971206 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:02:55.971525 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:55.971233 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:02:56.070629 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:56.070242 5088 generic.go:356] "Generic (PLEG): container finished" podID="7a6dccf5-7f1b-42c8-a974-1cf5510af915" containerID="5bebaecf163264904fdb275b0369c6f6a91e01b5cc9e2c91094ea9efd8d92def" exitCode=0 Oct 30 20:02:56.070629 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:56.070325 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a199d8f-hqq5n" event={"ID":"7a6dccf5-7f1b-42c8-a974-1cf5510af915","Type":"ContainerDied","Data":"5bebaecf163264904fdb275b0369c6f6a91e01b5cc9e2c91094ea9efd8d92def"} Oct 30 20:02:56.075543 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:56.072592 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" event={"ID":"cf9a8162-01f0-4226-b8f4-783125c277a2","Type":"ContainerStarted","Data":"38c6e49fb5aea702a7e3e884decb08e20b853a19e9cbb2319942d6a927ef2598"} Oct 30 20:02:57.080090 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:57.080049 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" event={"ID":"cf9a8162-01f0-4226-b8f4-783125c277a2","Type":"ContainerStarted","Data":"4c13990224bbb25058dad815e340b413c6171b14b68a186e923c3ecf17a7c4fa"} Oct 30 20:02:57.169187 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:57.169133 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" podStartSLOduration=3.169114305 podStartE2EDuration="3.169114305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:57.1528052 +0000 UTC m=+938.565519011" watchObservedRunningTime="2026-10-30 20:02:57.169114305 +0000 UTC m=+938.581828086" Oct 30 20:02:57.271010 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:57.270986 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a199d8f-hqq5n" Oct 30 20:02:57.317910 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:57.317880 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a6dccf5-7f1b-42c8-a974-1cf5510af915-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a6dccf5-7f1b-42c8-a974-1cf5510af915-maas-api-xks-ca\") pod \"7a6dccf5-7f1b-42c8-a974-1cf5510af915\" (UID: \"7a6dccf5-7f1b-42c8-a974-1cf5510af915\") " Oct 30 20:02:57.318103 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:57.317966 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a6dccf5-7f1b-42c8-a974-1cf5510af915-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a6dccf5-7f1b-42c8-a974-1cf5510af915-maas-api-openshift-ca\") pod \"7a6dccf5-7f1b-42c8-a974-1cf5510af915\" (UID: \"7a6dccf5-7f1b-42c8-a974-1cf5510af915\") " Oct 30 20:02:57.318103 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:57.318066 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a6dccf5-7f1b-42c8-a974-1cf5510af915-kube-api-access-hg45g\" (UniqueName: \"kubernetes.io/projected/7a6dccf5-7f1b-42c8-a974-1cf5510af915-kube-api-access-hg45g\") pod \"7a6dccf5-7f1b-42c8-a974-1cf5510af915\" (UID: \"7a6dccf5-7f1b-42c8-a974-1cf5510af915\") " Oct 30 20:02:57.318337 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:57.318312 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a6dccf5-7f1b-42c8-a974-1cf5510af915-maas-api-openshift-ca" pod "7a6dccf5-7f1b-42c8-a974-1cf5510af915" (UID: "7a6dccf5-7f1b-42c8-a974-1cf5510af915"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:57.318574 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:57.318558 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a6dccf5-7f1b-42c8-a974-1cf5510af915-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:02:57.320094 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:57.320071 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a6dccf5-7f1b-42c8-a974-1cf5510af915-maas-api-xks-ca" pod "7a6dccf5-7f1b-42c8-a974-1cf5510af915" (UID: "7a6dccf5-7f1b-42c8-a974-1cf5510af915"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:57.320177 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:57.320139 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a6dccf5-7f1b-42c8-a974-1cf5510af915-kube-api-access-hg45g" pod "7a6dccf5-7f1b-42c8-a974-1cf5510af915" (UID: "7a6dccf5-7f1b-42c8-a974-1cf5510af915"). InnerVolumeSpecName "kube-api-access-hg45g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:57.419774 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:57.419691 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a6dccf5-7f1b-42c8-a974-1cf5510af915-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:02:57.419774 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:57.419721 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg45g\" (UniqueName: \"kubernetes.io/projected/7a6dccf5-7f1b-42c8-a974-1cf5510af915-kube-api-access-hg45g\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:02:57.473025 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:57.472997 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" Oct 30 20:02:57.474432 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:57.474407 5088 patch_prober.go:29] interesting pod/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 20:02:57.474553 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:57.474458 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" podUID="cf9a8162-01f0-4226-b8f4-783125c277a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 20:02:58.111120 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:58.111085 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a199d8f-hqq5n" Oct 30 20:02:58.111120 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:58.111116 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a199d8f-hqq5n" event={"ID":"7a6dccf5-7f1b-42c8-a974-1cf5510af915","Type":"ContainerDied","Data":"a8411d094d5ac8b2f4d7bc58ca11f27f04d82261d098ce915ba1599af18b0ff6"} Oct 30 20:02:58.111706 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:58.111167 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8411d094d5ac8b2f4d7bc58ca11f27f04d82261d098ce915ba1599af18b0ff6" Oct 30 20:02:58.473328 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:58.473246 5088 patch_prober.go:29] interesting pod/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 20:02:58.473328 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:58.473297 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" podUID="cf9a8162-01f0-4226-b8f4-783125c277a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 20:02:59.472763 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:59.472727 5088 patch_prober.go:29] interesting pod/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 20:02:59.473148 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:59.472781 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" podUID="cf9a8162-01f0-4226-b8f4-783125c277a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 20:02:59.939189 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:59.939147 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e38036ba-75zcz"] Oct 30 20:02:59.939796 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:59.939775 5088 cpu_manager.go:414] "containerMap: removing container" podUID="7a6dccf5-7f1b-42c8-a974-1cf5510af915" containerName="revoke-keys" Oct 30 20:02:59.939902 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:59.939797 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a6dccf5-7f1b-42c8-a974-1cf5510af915" containerName="revoke-keys" Oct 30 20:02:59.940012 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:59.939999 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a6dccf5-7f1b-42c8-a974-1cf5510af915" containerName="revoke-keys" Oct 30 20:02:59.943526 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:59.943505 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38036ba-75zcz" Oct 30 20:02:59.964274 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:59.964246 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e38036ba-75zcz"] Oct 30 20:02:59.973692 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:59.973665 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:02:59.973847 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:02:59.973780 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:03:00.052359 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:00.052329 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp95d\" (UniqueName: \"kubernetes.io/projected/5c713054-69fb-4a8d-8670-19264c9c8e0c-kube-api-access-cp95d\") pod \"maas-api-revoke-sub-e38036ba-75zcz\" (UID: \"5c713054-69fb-4a8d-8670-19264c9c8e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38036ba-75zcz" Oct 30 20:03:00.052526 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:00.052487 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c713054-69fb-4a8d-8670-19264c9c8e0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e38036ba-75zcz\" (UID: \"5c713054-69fb-4a8d-8670-19264c9c8e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38036ba-75zcz" Oct 30 20:03:00.052674 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:00.052649 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c713054-69fb-4a8d-8670-19264c9c8e0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e38036ba-75zcz\" (UID: \"5c713054-69fb-4a8d-8670-19264c9c8e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38036ba-75zcz" Oct 30 20:03:00.117963 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:00.117925 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c990cbdd-229pr"] Oct 30 20:03:00.124304 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:00.124273 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990cbdd-229pr" Oct 30 20:03:00.141146 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:00.141121 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c990cbdd-229pr"] Oct 30 20:03:00.153985 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:00.153961 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c990cbdd-229pr\" (UID: \"c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990cbdd-229pr" Oct 30 20:03:00.154124 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:00.154006 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c713054-69fb-4a8d-8670-19264c9c8e0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e38036ba-75zcz\" (UID: \"5c713054-69fb-4a8d-8670-19264c9c8e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38036ba-75zcz" Oct 30 20:03:00.154199 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:00.154156 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c713054-69fb-4a8d-8670-19264c9c8e0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e38036ba-75zcz\" (UID: \"5c713054-69fb-4a8d-8670-19264c9c8e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38036ba-75zcz" Oct 30 20:03:00.154256 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:00.154239 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rzmq\" (UniqueName: \"kubernetes.io/projected/c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a-kube-api-access-7rzmq\") pod \"maas-api-revoke-sub-c990cbdd-229pr\" (UID: \"c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990cbdd-229pr" Oct 30 20:03:00.154328 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:00.154310 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp95d\" (UniqueName: \"kubernetes.io/projected/5c713054-69fb-4a8d-8670-19264c9c8e0c-kube-api-access-cp95d\") pod \"maas-api-revoke-sub-e38036ba-75zcz\" (UID: \"5c713054-69fb-4a8d-8670-19264c9c8e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38036ba-75zcz" Oct 30 20:03:00.154385 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:00.154358 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c990cbdd-229pr\" (UID: \"c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990cbdd-229pr" Oct 30 20:03:00.154896 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:00.154878 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c713054-69fb-4a8d-8670-19264c9c8e0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e38036ba-75zcz\" (UID: \"5c713054-69fb-4a8d-8670-19264c9c8e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38036ba-75zcz" Oct 30 20:03:00.158815 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:00.158796 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c713054-69fb-4a8d-8670-19264c9c8e0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e38036ba-75zcz\" (UID: \"5c713054-69fb-4a8d-8670-19264c9c8e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38036ba-75zcz" Oct 30 20:03:00.181029 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:00.180997 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp95d\" (UniqueName: \"kubernetes.io/projected/5c713054-69fb-4a8d-8670-19264c9c8e0c-kube-api-access-cp95d\") pod \"maas-api-revoke-sub-e38036ba-75zcz\" (UID: \"5c713054-69fb-4a8d-8670-19264c9c8e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38036ba-75zcz" Oct 30 20:03:00.255709 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:00.255670 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rzmq\" (UniqueName: \"kubernetes.io/projected/c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a-kube-api-access-7rzmq\") pod \"maas-api-revoke-sub-c990cbdd-229pr\" (UID: \"c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990cbdd-229pr" Oct 30 20:03:00.255867 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:00.255763 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c990cbdd-229pr\" (UID: \"c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990cbdd-229pr" Oct 30 20:03:00.255867 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:00.255845 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c990cbdd-229pr\" (UID: \"c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990cbdd-229pr" Oct 30 20:03:00.256521 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:00.256500 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c990cbdd-229pr\" (UID: \"c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990cbdd-229pr" Oct 30 20:03:00.257281 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:00.257264 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38036ba-75zcz" Oct 30 20:03:00.258164 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:00.258141 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c990cbdd-229pr\" (UID: \"c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990cbdd-229pr" Oct 30 20:03:00.282110 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:00.282085 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rzmq\" (UniqueName: \"kubernetes.io/projected/c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a-kube-api-access-7rzmq\") pod \"maas-api-revoke-sub-c990cbdd-229pr\" (UID: \"c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990cbdd-229pr" Oct 30 20:03:00.439650 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:00.439602 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990cbdd-229pr" Oct 30 20:03:00.473463 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:00.473006 5088 patch_prober.go:29] interesting pod/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 20:03:00.473463 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:00.473072 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" podUID="cf9a8162-01f0-4226-b8f4-783125c277a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 20:03:00.612707 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:03:00.612535 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c713054_69fb_4a8d_8670_19264c9c8e0c.slice/crio-7f9a3b19096b9afbce597d85fc558ab28624e31b120eb8d9566da9a625e80315 WatchSource:0}: Error finding container 7f9a3b19096b9afbce597d85fc558ab28624e31b120eb8d9566da9a625e80315: Status 404 returned error can't find the container with id 7f9a3b19096b9afbce597d85fc558ab28624e31b120eb8d9566da9a625e80315 Oct 30 20:03:00.614573 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:00.614232 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e38036ba-75zcz"] Oct 30 20:03:00.776982 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:00.776957 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c990cbdd-229pr"] Oct 30 20:03:00.809938 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:03:00.809889 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7f49ac2_73d1_4789_a8f0_cdab1a9cf52a.slice/crio-f01620ae364d07c1d08235d7a0fb44eb311adae70a6c9431d142e790bd800482 WatchSource:0}: Error finding container f01620ae364d07c1d08235d7a0fb44eb311adae70a6c9431d142e790bd800482: Status 404 returned error can't find the container with id f01620ae364d07c1d08235d7a0fb44eb311adae70a6c9431d142e790bd800482 Oct 30 20:03:01.130677 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:01.130648 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990cbdd-229pr" event={"ID":"c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a","Type":"ContainerStarted","Data":"2397d8f8f2eb7160a3fa9c243030f3c589b299cfd4ac606570a17bda5269158c"} Oct 30 20:03:01.130783 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:01.130710 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990cbdd-229pr" event={"ID":"c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a","Type":"ContainerStarted","Data":"f01620ae364d07c1d08235d7a0fb44eb311adae70a6c9431d142e790bd800482"} Oct 30 20:03:01.132279 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:01.132256 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38036ba-75zcz" event={"ID":"5c713054-69fb-4a8d-8670-19264c9c8e0c","Type":"ContainerStarted","Data":"f6aabab1231d6244f5ca2ce70e510ac29ac7e567fcf76a684f741161dc0d2342"} Oct 30 20:03:01.132380 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:01.132302 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38036ba-75zcz" event={"ID":"5c713054-69fb-4a8d-8670-19264c9c8e0c","Type":"ContainerStarted","Data":"7f9a3b19096b9afbce597d85fc558ab28624e31b120eb8d9566da9a625e80315"} Oct 30 20:03:01.178575 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:01.178517 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990cbdd-229pr" podStartSLOduration=1.178504222 podStartE2EDuration="1.178504222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:01.175339111 +0000 UTC m=+942.588052912" watchObservedRunningTime="2026-10-30 20:03:01.178504222 +0000 UTC m=+942.591218033" Oct 30 20:03:01.235682 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:01.235630 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38036ba-75zcz" podStartSLOduration=2.235598976 podStartE2EDuration="2.235598976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:01.233121786 +0000 UTC m=+942.645835579" watchObservedRunningTime="2026-10-30 20:03:01.235598976 +0000 UTC m=+942.648312755" Oct 30 20:03:01.472824 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:01.472744 5088 patch_prober.go:29] interesting pod/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 20:03:01.472824 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:01.472796 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" podUID="cf9a8162-01f0-4226-b8f4-783125c277a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 20:03:02.139853 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:02.139820 5088 generic.go:356] "Generic (PLEG): container finished" podID="5c713054-69fb-4a8d-8670-19264c9c8e0c" containerID="f6aabab1231d6244f5ca2ce70e510ac29ac7e567fcf76a684f741161dc0d2342" exitCode=0 Oct 30 20:03:02.140288 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:02.139903 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38036ba-75zcz" event={"ID":"5c713054-69fb-4a8d-8670-19264c9c8e0c","Type":"ContainerDied","Data":"f6aabab1231d6244f5ca2ce70e510ac29ac7e567fcf76a684f741161dc0d2342"} Oct 30 20:03:02.141581 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:02.141559 5088 generic.go:356] "Generic (PLEG): container finished" podID="c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a" containerID="2397d8f8f2eb7160a3fa9c243030f3c589b299cfd4ac606570a17bda5269158c" exitCode=0 Oct 30 20:03:02.141719 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:02.141595 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990cbdd-229pr" event={"ID":"c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a","Type":"ContainerDied","Data":"2397d8f8f2eb7160a3fa9c243030f3c589b299cfd4ac606570a17bda5269158c"} Oct 30 20:03:02.473079 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:02.472967 5088 patch_prober.go:29] interesting pod/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 20:03:02.473079 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:02.473024 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" podUID="cf9a8162-01f0-4226-b8f4-783125c277a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 20:03:03.370085 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:03.370060 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990cbdd-229pr" Oct 30 20:03:03.373898 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:03.373878 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38036ba-75zcz" Oct 30 20:03:03.426747 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:03.426657 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a-maas-api-openshift-ca\") pod \"c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a\" (UID: \"c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a\") " Oct 30 20:03:03.426910 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:03.426791 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a-maas-api-xks-ca\") pod \"c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a\" (UID: \"c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a\") " Oct 30 20:03:03.426910 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:03.426821 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a-kube-api-access-7rzmq\" (UniqueName: \"kubernetes.io/projected/c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a-kube-api-access-7rzmq\") pod \"c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a\" (UID: \"c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a\") " Oct 30 20:03:03.427071 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:03.427046 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a-maas-api-openshift-ca" pod "c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a" (UID: "c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:03.427399 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:03.427365 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:03:03.428920 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:03.428894 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a-maas-api-xks-ca" pod "c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a" (UID: "c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:03:03.429105 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:03.429083 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a-kube-api-access-7rzmq" pod "c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a" (UID: "c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a"). InnerVolumeSpecName "kube-api-access-7rzmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:03.480837 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:03.480803 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" Oct 30 20:03:03.481952 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:03.481827 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" Oct 30 20:03:03.482458 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:03.482440 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" Oct 30 20:03:03.527956 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:03.527928 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c713054-69fb-4a8d-8670-19264c9c8e0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c713054-69fb-4a8d-8670-19264c9c8e0c-maas-api-openshift-ca\") pod \"5c713054-69fb-4a8d-8670-19264c9c8e0c\" (UID: \"5c713054-69fb-4a8d-8670-19264c9c8e0c\") " Oct 30 20:03:03.528056 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:03.528003 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c713054-69fb-4a8d-8670-19264c9c8e0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c713054-69fb-4a8d-8670-19264c9c8e0c-maas-api-xks-ca\") pod \"5c713054-69fb-4a8d-8670-19264c9c8e0c\" (UID: \"5c713054-69fb-4a8d-8670-19264c9c8e0c\") " Oct 30 20:03:03.528133 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:03.528115 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c713054-69fb-4a8d-8670-19264c9c8e0c-kube-api-access-cp95d\" (UniqueName: \"kubernetes.io/projected/5c713054-69fb-4a8d-8670-19264c9c8e0c-kube-api-access-cp95d\") pod \"5c713054-69fb-4a8d-8670-19264c9c8e0c\" (UID: \"5c713054-69fb-4a8d-8670-19264c9c8e0c\") " Oct 30 20:03:03.528341 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:03.528319 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c713054-69fb-4a8d-8670-19264c9c8e0c-maas-api-openshift-ca" pod "5c713054-69fb-4a8d-8670-19264c9c8e0c" (UID: "5c713054-69fb-4a8d-8670-19264c9c8e0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:03.528516 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:03.528493 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c713054-69fb-4a8d-8670-19264c9c8e0c-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:03:03.528516 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:03.528516 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:03:03.528703 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:03.528530 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rzmq\" (UniqueName: \"kubernetes.io/projected/c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a-kube-api-access-7rzmq\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:03:03.530207 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:03.530185 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c713054-69fb-4a8d-8670-19264c9c8e0c-maas-api-xks-ca" pod "5c713054-69fb-4a8d-8670-19264c9c8e0c" (UID: "5c713054-69fb-4a8d-8670-19264c9c8e0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:03:03.530290 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:03.530269 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c713054-69fb-4a8d-8670-19264c9c8e0c-kube-api-access-cp95d" pod "5c713054-69fb-4a8d-8670-19264c9c8e0c" (UID: "5c713054-69fb-4a8d-8670-19264c9c8e0c"). InnerVolumeSpecName "kube-api-access-cp95d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:03.630445 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:03.630407 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c713054-69fb-4a8d-8670-19264c9c8e0c-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:03:03.630445 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:03.630443 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp95d\" (UniqueName: \"kubernetes.io/projected/5c713054-69fb-4a8d-8670-19264c9c8e0c-kube-api-access-cp95d\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:03:04.159204 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:04.159166 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990cbdd-229pr" Oct 30 20:03:04.159390 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:04.159196 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c990cbdd-229pr" event={"ID":"c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a","Type":"ContainerDied","Data":"f01620ae364d07c1d08235d7a0fb44eb311adae70a6c9431d142e790bd800482"} Oct 30 20:03:04.159390 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:04.159251 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f01620ae364d07c1d08235d7a0fb44eb311adae70a6c9431d142e790bd800482" Oct 30 20:03:04.160588 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:04.160564 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38036ba-75zcz" event={"ID":"5c713054-69fb-4a8d-8670-19264c9c8e0c","Type":"ContainerDied","Data":"7f9a3b19096b9afbce597d85fc558ab28624e31b120eb8d9566da9a625e80315"} Oct 30 20:03:04.160735 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:04.160600 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38036ba-75zcz" Oct 30 20:03:04.160735 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:04.160621 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f9a3b19096b9afbce597d85fc558ab28624e31b120eb8d9566da9a625e80315" Oct 30 20:03:09.552371 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:09.552330 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt"] Oct 30 20:03:09.564950 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:09.558715 5088 cpu_manager.go:414] "containerMap: removing container" podUID="c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a" containerName="revoke-keys" Oct 30 20:03:09.567362 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:09.567335 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a" containerName="revoke-keys" Oct 30 20:03:09.567587 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:09.567571 5088 cpu_manager.go:414] "containerMap: removing container" podUID="5c713054-69fb-4a8d-8670-19264c9c8e0c" containerName="revoke-keys" Oct 30 20:03:09.567735 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:09.567719 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c713054-69fb-4a8d-8670-19264c9c8e0c" containerName="revoke-keys" Oct 30 20:03:09.568045 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:09.568028 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c713054-69fb-4a8d-8670-19264c9c8e0c" containerName="revoke-keys" Oct 30 20:03:09.568154 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:09.568141 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a" containerName="revoke-keys" Oct 30 20:03:09.592084 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:09.592048 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt"] Oct 30 20:03:09.601642 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:09.592392 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" Oct 30 20:03:09.742648 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:09.692317 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-istio-data\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt\" (UID: \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" Oct 30 20:03:09.742648 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:09.692429 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-credential-socket\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt\" (UID: \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" Oct 30 20:03:09.742648 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:09.692458 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-istiod-ca-cert\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt\" (UID: \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" Oct 30 20:03:09.742648 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:09.692484 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-istio-podinfo\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt\" (UID: \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" Oct 30 20:03:09.742648 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:09.692557 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmdrb\" (UniqueName: \"kubernetes.io/projected/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-kube-api-access-jmdrb\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt\" (UID: \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" Oct 30 20:03:09.742648 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:09.692627 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-workload-socket\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt\" (UID: \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" Oct 30 20:03:09.742648 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:09.692672 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-workload-certs\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt\" (UID: \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" Oct 30 20:03:09.742648 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:09.692715 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-istio-envoy\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt\" (UID: \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" Oct 30 20:03:09.742648 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:09.692776 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-istio-token\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt\" (UID: \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" Oct 30 20:03:09.826927 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:09.826820 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmdrb\" (UniqueName: \"kubernetes.io/projected/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-kube-api-access-jmdrb\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt\" (UID: \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" Oct 30 20:03:09.827221 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:09.827200 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-workload-socket\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt\" (UID: \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" Oct 30 20:03:09.827398 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:09.827373 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-workload-certs\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt\" (UID: \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" Oct 30 20:03:09.827542 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:09.827526 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-istio-envoy\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt\" (UID: \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" Oct 30 20:03:09.841459 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:09.841421 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-istio-token\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt\" (UID: \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" Oct 30 20:03:09.841655 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:09.841533 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-istio-data\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt\" (UID: \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" Oct 30 20:03:09.841944 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:09.841721 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-credential-socket\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt\" (UID: \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" Oct 30 20:03:09.841944 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:09.841753 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-istiod-ca-cert\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt\" (UID: \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" Oct 30 20:03:09.841944 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:09.841786 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-istio-podinfo\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt\" (UID: \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" Oct 30 20:03:09.842969 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:09.842940 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-istio-data\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt\" (UID: \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" Oct 30 20:03:09.843860 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:09.843838 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-workload-certs\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt\" (UID: \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" Oct 30 20:03:09.844257 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:09.844238 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-workload-socket\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt\" (UID: \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" Oct 30 20:03:09.844500 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:09.844469 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-credential-socket\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt\" (UID: \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" Oct 30 20:03:09.845187 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:09.845164 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-istiod-ca-cert\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt\" (UID: \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" Oct 30 20:03:09.870640 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:09.863228 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-istio-podinfo\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt\" (UID: \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" Oct 30 20:03:10.015635 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:10.007229 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-istio-token\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt\" (UID: \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" Oct 30 20:03:10.056638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:10.053735 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmdrb\" (UniqueName: \"kubernetes.io/projected/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-kube-api-access-jmdrb\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt\" (UID: \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" Oct 30 20:03:10.138951 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:10.133393 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-istio-envoy\") pod \"e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt\" (UID: \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\") " pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" Oct 30 20:03:10.283826 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:10.279736 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr"] Oct 30 20:03:10.347010 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:10.346796 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" Oct 30 20:03:10.386513 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:10.386470 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr"] Oct 30 20:03:10.387082 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:10.387044 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" Oct 30 20:03:10.440663 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:10.440558 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zp8v\" (UniqueName: \"kubernetes.io/projected/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-kube-api-access-8zp8v\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4w2qr\" (UID: \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" Oct 30 20:03:10.465881 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:10.465829 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4w2qr\" (UID: \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" Oct 30 20:03:10.466134 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:10.466104 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4w2qr\" (UID: \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" Oct 30 20:03:10.466381 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:10.466364 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4w2qr\" (UID: \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" Oct 30 20:03:10.466675 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:10.466652 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4w2qr\" (UID: \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" Oct 30 20:03:10.572416 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:10.572382 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4w2qr\" (UID: \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" Oct 30 20:03:10.594627 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:10.593341 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4w2qr\" (UID: \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" Oct 30 20:03:10.633636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:10.619823 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4w2qr\" (UID: \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" Oct 30 20:03:10.633636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:10.619963 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4w2qr\" (UID: \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" Oct 30 20:03:10.727695 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:10.727100 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4w2qr\" (UID: \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" Oct 30 20:03:10.727695 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:10.727173 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4w2qr\" (UID: \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" Oct 30 20:03:10.727695 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:10.727268 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zp8v\" (UniqueName: \"kubernetes.io/projected/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-kube-api-access-8zp8v\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4w2qr\" (UID: \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" Oct 30 20:03:10.727695 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:10.727310 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4w2qr\" (UID: \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" Oct 30 20:03:10.727695 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:10.727337 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4w2qr\" (UID: \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" Oct 30 20:03:10.727695 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:10.727420 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4w2qr\" (UID: \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" Oct 30 20:03:10.764652 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:10.728440 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4w2qr\" (UID: \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" Oct 30 20:03:10.764652 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:10.728937 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4w2qr\" (UID: \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" Oct 30 20:03:10.764652 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:10.730198 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4w2qr\" (UID: \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" Oct 30 20:03:10.764652 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:10.730249 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4w2qr\" (UID: \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" Oct 30 20:03:10.764652 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:10.730312 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4w2qr\" (UID: \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" Oct 30 20:03:10.764652 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:10.730930 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4w2qr\" (UID: \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" Oct 30 20:03:10.764652 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:10.731186 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4w2qr\" (UID: \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" Oct 30 20:03:10.764652 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:10.747728 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4w2qr\" (UID: \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" Oct 30 20:03:10.776090 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:10.776020 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4w2qr\" (UID: \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" Oct 30 20:03:10.777324 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:10.777298 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4w2qr\" (UID: \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" Oct 30 20:03:10.783260 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:10.783202 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4w2qr\" (UID: \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" Oct 30 20:03:10.784339 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:10.784265 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zp8v\" (UniqueName: \"kubernetes.io/projected/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-kube-api-access-8zp8v\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4w2qr\" (UID: \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" Oct 30 20:03:10.937945 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:10.937861 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" Oct 30 20:03:11.365577 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:11.365551 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt"] Oct 30 20:03:11.366333 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:03:11.366311 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdf81fa2_7f24_4dd4_bff8_0ff7f4f64980.slice/crio-3ea6babf52d5fdb46c7999a8d376ec03c02109bc5ba7036b0572eb937561cf39 WatchSource:0}: Error finding container 3ea6babf52d5fdb46c7999a8d376ec03c02109bc5ba7036b0572eb937561cf39: Status 404 returned error can't find the container with id 3ea6babf52d5fdb46c7999a8d376ec03c02109bc5ba7036b0572eb937561cf39 Oct 30 20:03:11.369470 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:11.369412 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:03:11.369775 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:11.369744 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:03:11.369945 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:11.369927 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:03:11.608554 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:11.608520 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr"] Oct 30 20:03:11.622282 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:03:11.622207 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba76bfa7_6ba7_4862_8348_ca7c17487b9f.slice/crio-3e810d14352a4a960f9a277b8f09fb5d57f7da797a280bc33e507e10764f5abf WatchSource:0}: Error finding container 3e810d14352a4a960f9a277b8f09fb5d57f7da797a280bc33e507e10764f5abf: Status 404 returned error can't find the container with id 3e810d14352a4a960f9a277b8f09fb5d57f7da797a280bc33e507e10764f5abf Oct 30 20:03:11.625416 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:11.625380 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:03:11.625531 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:11.625469 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:03:11.625531 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:11.625515 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:03:12.352772 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:12.352733 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" event={"ID":"ba76bfa7-6ba7-4862-8348-ca7c17487b9f","Type":"ContainerStarted","Data":"f54a4ec28ad66ba296a4360924c66536ec38efc33f4ff1e36c3bb003b0ba44ed"} Oct 30 20:03:12.352984 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:12.352795 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" event={"ID":"ba76bfa7-6ba7-4862-8348-ca7c17487b9f","Type":"ContainerStarted","Data":"3e810d14352a4a960f9a277b8f09fb5d57f7da797a280bc33e507e10764f5abf"} Oct 30 20:03:12.354907 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:12.354873 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" event={"ID":"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980","Type":"ContainerStarted","Data":"af78a77d20af1212cd0cbe7378f61b37a49b69e6b21ab698149d0670432b8a81"} Oct 30 20:03:12.355038 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:12.354923 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" event={"ID":"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980","Type":"ContainerStarted","Data":"3ea6babf52d5fdb46c7999a8d376ec03c02109bc5ba7036b0572eb937561cf39"} Oct 30 20:03:12.402636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:12.401673 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" Oct 30 20:03:12.402636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:12.402309 5088 patch_prober.go:29] interesting pod/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 20:03:12.402636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:12.402368 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" podUID="cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 20:03:12.432826 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:12.432774 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" podStartSLOduration=2.432757812 podStartE2EDuration="2.432757812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:12.418977448 +0000 UTC m=+953.831691266" watchObservedRunningTime="2026-10-30 20:03:12.432757812 +0000 UTC m=+953.845471644" Oct 30 20:03:12.471428 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:12.471365 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" podStartSLOduration=3.471349924 podStartE2EDuration="3.471349924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:12.469166159 +0000 UTC m=+953.881879974" watchObservedRunningTime="2026-10-30 20:03:12.471349924 +0000 UTC m=+953.884063702" Oct 30 20:03:12.940383 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:12.940349 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" Oct 30 20:03:12.941207 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:12.941183 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4w2qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 20:03:12.941364 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:12.941340 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" podUID="ba76bfa7-6ba7-4862-8348-ca7c17487b9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 20:03:13.417633 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:13.407050 5088 patch_prober.go:29] interesting pod/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 20:03:13.417633 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:13.407122 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" podUID="cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 20:03:13.939220 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:13.939185 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4w2qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 20:03:13.939516 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:13.939491 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" podUID="ba76bfa7-6ba7-4862-8348-ca7c17487b9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 20:03:14.395637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:14.392019 5088 patch_prober.go:29] interesting pod/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 20:03:14.395637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:14.392076 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" podUID="cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 20:03:14.938223 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:14.938189 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4w2qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 20:03:14.938418 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:14.938240 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" podUID="ba76bfa7-6ba7-4862-8348-ca7c17487b9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 20:03:15.392978 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:15.392943 5088 patch_prober.go:29] interesting pod/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 20:03:15.393270 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:15.393239 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" podUID="cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 20:03:15.939038 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:15.938998 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4w2qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 20:03:15.939435 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:15.939067 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" podUID="ba76bfa7-6ba7-4862-8348-ca7c17487b9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 20:03:16.395635 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:16.391947 5088 patch_prober.go:29] interesting pod/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 20:03:16.395635 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:16.392009 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" podUID="cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 20:03:16.939674 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:16.939001 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4w2qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 20:03:16.939674 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:16.939072 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" podUID="ba76bfa7-6ba7-4862-8348-ca7c17487b9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 20:03:17.140034 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:17.133710 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2a06b2b-vvbdx"] Oct 30 20:03:17.146430 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:17.146400 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2a06b2b-vvbdx"] Oct 30 20:03:17.146678 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:17.146660 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a06b2b-vvbdx" Oct 30 20:03:17.197980 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:17.194946 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:03:17.197980 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:17.195023 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:03:17.306637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:17.299549 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzhgh\" (UniqueName: \"kubernetes.io/projected/61dbdeab-407a-4902-aa37-28ec91a3770e-kube-api-access-jzhgh\") pod \"maas-api-revoke-sub-e2a06b2b-vvbdx\" (UID: \"61dbdeab-407a-4902-aa37-28ec91a3770e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a06b2b-vvbdx" Oct 30 20:03:17.306637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:17.299598 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61dbdeab-407a-4902-aa37-28ec91a3770e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2a06b2b-vvbdx\" (UID: \"61dbdeab-407a-4902-aa37-28ec91a3770e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a06b2b-vvbdx" Oct 30 20:03:17.306637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:17.299659 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61dbdeab-407a-4902-aa37-28ec91a3770e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2a06b2b-vvbdx\" (UID: \"61dbdeab-407a-4902-aa37-28ec91a3770e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a06b2b-vvbdx" Oct 30 20:03:17.393637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:17.390289 5088 patch_prober.go:29] interesting pod/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 20:03:17.393637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:17.390343 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" podUID="cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 20:03:17.431635 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:17.411774 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzhgh\" (UniqueName: \"kubernetes.io/projected/61dbdeab-407a-4902-aa37-28ec91a3770e-kube-api-access-jzhgh\") pod \"maas-api-revoke-sub-e2a06b2b-vvbdx\" (UID: \"61dbdeab-407a-4902-aa37-28ec91a3770e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a06b2b-vvbdx" Oct 30 20:03:17.431635 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:17.428793 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61dbdeab-407a-4902-aa37-28ec91a3770e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2a06b2b-vvbdx\" (UID: \"61dbdeab-407a-4902-aa37-28ec91a3770e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a06b2b-vvbdx" Oct 30 20:03:17.431635 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:17.428874 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61dbdeab-407a-4902-aa37-28ec91a3770e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2a06b2b-vvbdx\" (UID: \"61dbdeab-407a-4902-aa37-28ec91a3770e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a06b2b-vvbdx" Oct 30 20:03:17.431635 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:17.429817 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61dbdeab-407a-4902-aa37-28ec91a3770e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2a06b2b-vvbdx\" (UID: \"61dbdeab-407a-4902-aa37-28ec91a3770e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a06b2b-vvbdx" Oct 30 20:03:17.432827 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:17.432778 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61dbdeab-407a-4902-aa37-28ec91a3770e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2a06b2b-vvbdx\" (UID: \"61dbdeab-407a-4902-aa37-28ec91a3770e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a06b2b-vvbdx" Oct 30 20:03:17.450785 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:17.450727 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzhgh\" (UniqueName: \"kubernetes.io/projected/61dbdeab-407a-4902-aa37-28ec91a3770e-kube-api-access-jzhgh\") pod \"maas-api-revoke-sub-e2a06b2b-vvbdx\" (UID: \"61dbdeab-407a-4902-aa37-28ec91a3770e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a06b2b-vvbdx" Oct 30 20:03:17.460771 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:17.460749 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a06b2b-vvbdx" Oct 30 20:03:17.818054 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:17.818024 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2a06b2b-vvbdx"] Oct 30 20:03:17.819709 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:03:17.819671 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61dbdeab_407a_4902_aa37_28ec91a3770e.slice/crio-a255e99c5e99ed47e8fb421755ece20fa7a0e7e74b63f7f20d9d5ba963309c97 WatchSource:0}: Error finding container a255e99c5e99ed47e8fb421755ece20fa7a0e7e74b63f7f20d9d5ba963309c97: Status 404 returned error can't find the container with id a255e99c5e99ed47e8fb421755ece20fa7a0e7e74b63f7f20d9d5ba963309c97 Oct 30 20:03:17.939153 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:17.939122 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4w2qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 20:03:17.939309 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:17.939172 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" podUID="ba76bfa7-6ba7-4862-8348-ca7c17487b9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 20:03:18.423567 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:18.423523 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a06b2b-vvbdx" event={"ID":"61dbdeab-407a-4902-aa37-28ec91a3770e","Type":"ContainerStarted","Data":"41517793b1607f48e8419b8ef0df6e2114c329899772b7f8cbdcd7c000efb1e9"} Oct 30 20:03:18.424119 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:18.423597 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a06b2b-vvbdx" event={"ID":"61dbdeab-407a-4902-aa37-28ec91a3770e","Type":"ContainerStarted","Data":"a255e99c5e99ed47e8fb421755ece20fa7a0e7e74b63f7f20d9d5ba963309c97"} Oct 30 20:03:18.480129 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:18.480038 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a06b2b-vvbdx" podStartSLOduration=1.480020207 podStartE2EDuration="1.480020207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:18.476204522 +0000 UTC m=+959.888918321" watchObservedRunningTime="2026-10-30 20:03:18.480020207 +0000 UTC m=+959.892733988" Oct 30 20:03:18.590102 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:18.590074 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" Oct 30 20:03:18.591035 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:18.591012 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" Oct 30 20:03:18.591853 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:18.591826 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" Oct 30 20:03:18.938944 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:18.938915 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4w2qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 20:03:18.939054 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:18.938963 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" podUID="ba76bfa7-6ba7-4862-8348-ca7c17487b9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 20:03:19.432979 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:19.432930 5088 generic.go:356] "Generic (PLEG): container finished" podID="61dbdeab-407a-4902-aa37-28ec91a3770e" containerID="41517793b1607f48e8419b8ef0df6e2114c329899772b7f8cbdcd7c000efb1e9" exitCode=0 Oct 30 20:03:19.433735 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:19.433699 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a06b2b-vvbdx" event={"ID":"61dbdeab-407a-4902-aa37-28ec91a3770e","Type":"ContainerDied","Data":"41517793b1607f48e8419b8ef0df6e2114c329899772b7f8cbdcd7c000efb1e9"} Oct 30 20:03:20.614847 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:20.614819 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a06b2b-vvbdx" Oct 30 20:03:20.686016 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:20.685978 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61dbdeab-407a-4902-aa37-28ec91a3770e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61dbdeab-407a-4902-aa37-28ec91a3770e-maas-api-openshift-ca\") pod \"61dbdeab-407a-4902-aa37-28ec91a3770e\" (UID: \"61dbdeab-407a-4902-aa37-28ec91a3770e\") " Oct 30 20:03:20.686214 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:20.686121 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61dbdeab-407a-4902-aa37-28ec91a3770e-kube-api-access-jzhgh\" (UniqueName: \"kubernetes.io/projected/61dbdeab-407a-4902-aa37-28ec91a3770e-kube-api-access-jzhgh\") pod \"61dbdeab-407a-4902-aa37-28ec91a3770e\" (UID: \"61dbdeab-407a-4902-aa37-28ec91a3770e\") " Oct 30 20:03:20.686214 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:20.686165 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61dbdeab-407a-4902-aa37-28ec91a3770e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61dbdeab-407a-4902-aa37-28ec91a3770e-maas-api-xks-ca\") pod \"61dbdeab-407a-4902-aa37-28ec91a3770e\" (UID: \"61dbdeab-407a-4902-aa37-28ec91a3770e\") " Oct 30 20:03:20.686867 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:20.686821 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61dbdeab-407a-4902-aa37-28ec91a3770e-maas-api-openshift-ca" pod "61dbdeab-407a-4902-aa37-28ec91a3770e" (UID: "61dbdeab-407a-4902-aa37-28ec91a3770e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:20.688795 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:20.688768 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61dbdeab-407a-4902-aa37-28ec91a3770e-maas-api-xks-ca" pod "61dbdeab-407a-4902-aa37-28ec91a3770e" (UID: "61dbdeab-407a-4902-aa37-28ec91a3770e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:03:20.689170 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:20.689142 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61dbdeab-407a-4902-aa37-28ec91a3770e-kube-api-access-jzhgh" pod "61dbdeab-407a-4902-aa37-28ec91a3770e" (UID: "61dbdeab-407a-4902-aa37-28ec91a3770e"). InnerVolumeSpecName "kube-api-access-jzhgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:20.776654 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:20.776590 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" Oct 30 20:03:20.777033 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:20.777009 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" Oct 30 20:03:20.778003 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:20.777975 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" Oct 30 20:03:20.788089 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:20.788062 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzhgh\" (UniqueName: \"kubernetes.io/projected/61dbdeab-407a-4902-aa37-28ec91a3770e-kube-api-access-jzhgh\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.788202 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:20.788092 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61dbdeab-407a-4902-aa37-28ec91a3770e-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.788202 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:20.788108 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61dbdeab-407a-4902-aa37-28ec91a3770e-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:03:21.449433 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:21.449402 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a06b2b-vvbdx" Oct 30 20:03:21.449642 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:21.449441 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a06b2b-vvbdx" event={"ID":"61dbdeab-407a-4902-aa37-28ec91a3770e","Type":"ContainerDied","Data":"a255e99c5e99ed47e8fb421755ece20fa7a0e7e74b63f7f20d9d5ba963309c97"} Oct 30 20:03:21.449642 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:21.449486 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a255e99c5e99ed47e8fb421755ece20fa7a0e7e74b63f7f20d9d5ba963309c97" Oct 30 20:03:22.438184 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:03:22.438149 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:03:22.785531 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:22.785498 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:03:46.841273 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:46.841237 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:03:52.439485 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:03:52.439447 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:03:52.461222 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:52.461188 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk"] Oct 30 20:03:52.462141 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:52.462120 5088 cpu_manager.go:414] "containerMap: removing container" podUID="61dbdeab-407a-4902-aa37-28ec91a3770e" containerName="revoke-keys" Oct 30 20:03:52.462200 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:52.462145 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61dbdeab-407a-4902-aa37-28ec91a3770e" containerName="revoke-keys" Oct 30 20:03:52.462335 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:52.462321 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="61dbdeab-407a-4902-aa37-28ec91a3770e" containerName="revoke-keys" Oct 30 20:03:52.466993 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:52.466969 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:03:52.486891 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:52.486845 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5f10d2\"/\"openshift-service-ca.crt\"" Oct 30 20:03:52.498631 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:52.492457 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk"] Oct 30 20:03:52.499687 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:52.499654 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5f10d2\"/\"kube-root-ca.crt\"" Oct 30 20:03:52.506831 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:52.506799 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5f10d2\"/\"distinct-2-w1-5f10d2-kserve-self-signed-certs\"" Oct 30 20:03:52.506973 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:52.506833 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5f10d2\"/\"default-dockercfg-jk2qk\"" Oct 30 20:03:52.546820 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:52.546781 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/682a764a-19ca-49e8-a1d4-1258b8d74bf4-tmp-dir\") pod \"distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk\" (UID: \"682a764a-19ca-49e8-a1d4-1258b8d74bf4\") " pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:03:52.547001 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:52.546911 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/682a764a-19ca-49e8-a1d4-1258b8d74bf4-tls-certs\") pod \"distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk\" (UID: \"682a764a-19ca-49e8-a1d4-1258b8d74bf4\") " pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:03:52.547001 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:52.546979 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/682a764a-19ca-49e8-a1d4-1258b8d74bf4-model-cache\") pod \"distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk\" (UID: \"682a764a-19ca-49e8-a1d4-1258b8d74bf4\") " pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:03:52.547101 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:52.547086 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtf95\" (UniqueName: \"kubernetes.io/projected/682a764a-19ca-49e8-a1d4-1258b8d74bf4-kube-api-access-dtf95\") pod \"distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk\" (UID: \"682a764a-19ca-49e8-a1d4-1258b8d74bf4\") " pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:03:52.547142 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:52.547111 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/682a764a-19ca-49e8-a1d4-1258b8d74bf4-dshm\") pod \"distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk\" (UID: \"682a764a-19ca-49e8-a1d4-1258b8d74bf4\") " pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:03:52.547142 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:52.547133 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/682a764a-19ca-49e8-a1d4-1258b8d74bf4-home\") pod \"distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk\" (UID: \"682a764a-19ca-49e8-a1d4-1258b8d74bf4\") " pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:03:52.669287 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:52.669245 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/682a764a-19ca-49e8-a1d4-1258b8d74bf4-tls-certs\") pod \"distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk\" (UID: \"682a764a-19ca-49e8-a1d4-1258b8d74bf4\") " pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:03:52.669623 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:52.669584 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/682a764a-19ca-49e8-a1d4-1258b8d74bf4-model-cache\") pod \"distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk\" (UID: \"682a764a-19ca-49e8-a1d4-1258b8d74bf4\") " pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:03:52.681635 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:52.670375 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtf95\" (UniqueName: \"kubernetes.io/projected/682a764a-19ca-49e8-a1d4-1258b8d74bf4-kube-api-access-dtf95\") pod \"distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk\" (UID: \"682a764a-19ca-49e8-a1d4-1258b8d74bf4\") " pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:03:52.681923 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:52.681899 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/682a764a-19ca-49e8-a1d4-1258b8d74bf4-dshm\") pod \"distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk\" (UID: \"682a764a-19ca-49e8-a1d4-1258b8d74bf4\") " pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:03:52.682073 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:52.682055 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/682a764a-19ca-49e8-a1d4-1258b8d74bf4-home\") pod \"distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk\" (UID: \"682a764a-19ca-49e8-a1d4-1258b8d74bf4\") " pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:03:52.683062 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:52.677146 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/682a764a-19ca-49e8-a1d4-1258b8d74bf4-model-cache\") pod \"distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk\" (UID: \"682a764a-19ca-49e8-a1d4-1258b8d74bf4\") " pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:03:52.683062 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:52.682326 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/682a764a-19ca-49e8-a1d4-1258b8d74bf4-tmp-dir\") pod \"distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk\" (UID: \"682a764a-19ca-49e8-a1d4-1258b8d74bf4\") " pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:03:52.683062 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:52.682562 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/682a764a-19ca-49e8-a1d4-1258b8d74bf4-home\") pod \"distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk\" (UID: \"682a764a-19ca-49e8-a1d4-1258b8d74bf4\") " pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:03:52.683062 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:52.683022 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/682a764a-19ca-49e8-a1d4-1258b8d74bf4-tmp-dir\") pod \"distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk\" (UID: \"682a764a-19ca-49e8-a1d4-1258b8d74bf4\") " pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:03:52.753630 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:52.753579 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtf95\" (UniqueName: \"kubernetes.io/projected/682a764a-19ca-49e8-a1d4-1258b8d74bf4-kube-api-access-dtf95\") pod \"distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk\" (UID: \"682a764a-19ca-49e8-a1d4-1258b8d74bf4\") " pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:03:52.782592 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:52.782557 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/682a764a-19ca-49e8-a1d4-1258b8d74bf4-tls-certs\") pod \"distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk\" (UID: \"682a764a-19ca-49e8-a1d4-1258b8d74bf4\") " pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:03:52.783986 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:52.783954 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/682a764a-19ca-49e8-a1d4-1258b8d74bf4-dshm\") pod \"distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk\" (UID: \"682a764a-19ca-49e8-a1d4-1258b8d74bf4\") " pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:03:53.082658 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:53.082513 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:03:53.392799 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:53.392774 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk"] Oct 30 20:03:53.393100 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:03:53.393078 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod682a764a_19ca_49e8_a1d4_1258b8d74bf4.slice/crio-4572181a77380ac8a27d95ed5858169d915b823a20275780c6429bd52f81fa20 WatchSource:0}: Error finding container 4572181a77380ac8a27d95ed5858169d915b823a20275780c6429bd52f81fa20: Status 404 returned error can't find the container with id 4572181a77380ac8a27d95ed5858169d915b823a20275780c6429bd52f81fa20 Oct 30 20:03:53.795606 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:53.795570 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" event={"ID":"682a764a-19ca-49e8-a1d4-1258b8d74bf4","Type":"ContainerStarted","Data":"509d90b132ac33c180da539c5def59572e449b8c1dd393a8ba5dbb9f327b4c73"} Oct 30 20:03:53.796024 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:53.795645 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" event={"ID":"682a764a-19ca-49e8-a1d4-1258b8d74bf4","Type":"ContainerStarted","Data":"4572181a77380ac8a27d95ed5858169d915b823a20275780c6429bd52f81fa20"} Oct 30 20:03:53.843066 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:53.843013 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podStartSLOduration=1.842997677 podStartE2EDuration="1.842997677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:53.841308206 +0000 UTC m=+995.254021985" watchObservedRunningTime="2026-10-30 20:03:53.842997677 +0000 UTC m=+995.255711456" Oct 30 20:03:54.083204 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:54.083115 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:03:55.079361 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:55.079332 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn"] Oct 30 20:03:55.083586 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:55.083565 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" Oct 30 20:03:55.109799 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:55.109773 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn"] Oct 30 20:03:55.227210 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:55.227167 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-istio-token\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn\" (UID: \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" Oct 30 20:03:55.227411 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:55.227225 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-istio-envoy\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn\" (UID: \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" Oct 30 20:03:55.227411 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:55.227313 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-istio-data\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn\" (UID: \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" Oct 30 20:03:55.227520 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:55.227431 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-istiod-ca-cert\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn\" (UID: \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" Oct 30 20:03:55.227520 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:55.227509 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-workload-certs\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn\" (UID: \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" Oct 30 20:03:55.227593 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:55.227534 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp5fz\" (UniqueName: \"kubernetes.io/projected/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-kube-api-access-mp5fz\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn\" (UID: \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" Oct 30 20:03:55.227593 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:55.227569 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-workload-socket\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn\" (UID: \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" Oct 30 20:03:55.227739 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:55.227722 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-istio-podinfo\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn\" (UID: \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" Oct 30 20:03:55.227776 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:55.227758 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-credential-socket\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn\" (UID: \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" Oct 30 20:03:55.328950 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:55.328918 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-istiod-ca-cert\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn\" (UID: \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" Oct 30 20:03:55.329156 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:55.328964 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-workload-certs\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn\" (UID: \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" Oct 30 20:03:55.329156 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:55.328984 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp5fz\" (UniqueName: \"kubernetes.io/projected/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-kube-api-access-mp5fz\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn\" (UID: \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" Oct 30 20:03:55.329156 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:55.329014 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-workload-socket\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn\" (UID: \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" Oct 30 20:03:55.329156 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:55.329072 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-istio-podinfo\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn\" (UID: \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" Oct 30 20:03:55.329156 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:55.329098 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-credential-socket\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn\" (UID: \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" Oct 30 20:03:55.329461 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:55.329228 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-istio-token\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn\" (UID: \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" Oct 30 20:03:55.329461 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:55.329279 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-istio-envoy\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn\" (UID: \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" Oct 30 20:03:55.329461 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:55.329312 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-istio-data\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn\" (UID: \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" Oct 30 20:03:55.329461 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:55.329410 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-workload-certs\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn\" (UID: \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" Oct 30 20:03:55.329713 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:55.329473 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-workload-socket\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn\" (UID: \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" Oct 30 20:03:55.329713 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:55.329573 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-credential-socket\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn\" (UID: \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" Oct 30 20:03:55.329831 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:55.329764 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-istio-data\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn\" (UID: \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" Oct 30 20:03:55.329930 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:55.329910 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-istiod-ca-cert\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn\" (UID: \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" Oct 30 20:03:55.331461 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:55.331443 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-istio-envoy\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn\" (UID: \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" Oct 30 20:03:55.331696 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:55.331674 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-istio-podinfo\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn\" (UID: \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" Oct 30 20:03:55.355272 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:55.355238 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-istio-token\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn\" (UID: \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" Oct 30 20:03:55.355388 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:55.355305 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp5fz\" (UniqueName: \"kubernetes.io/projected/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-kube-api-access-mp5fz\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn\" (UID: \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" Oct 30 20:03:55.398667 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:55.398637 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" Oct 30 20:03:55.721869 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:55.721840 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn"] Oct 30 20:03:55.722675 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:03:55.722649 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc3ed9fa_b7f0_418f_8ac9_422f9e765027.slice/crio-9c9e83ff2799a95ce0050664bd68844e66ec5196f8b92985e703b89ffa636026 WatchSource:0}: Error finding container 9c9e83ff2799a95ce0050664bd68844e66ec5196f8b92985e703b89ffa636026: Status 404 returned error can't find the container with id 9c9e83ff2799a95ce0050664bd68844e66ec5196f8b92985e703b89ffa636026 Oct 30 20:03:55.725066 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:55.725025 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:03:55.725142 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:55.725094 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:03:55.725142 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:55.725128 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:03:55.813215 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:55.813170 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" event={"ID":"fc3ed9fa-b7f0-418f-8ac9-422f9e765027","Type":"ContainerStarted","Data":"ec0589382cf960ccf68133b378f1efc8178b0d114b31e0ab49aabf9a1e03f11a"} Oct 30 20:03:55.813402 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:55.813242 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" event={"ID":"fc3ed9fa-b7f0-418f-8ac9-422f9e765027","Type":"ContainerStarted","Data":"9c9e83ff2799a95ce0050664bd68844e66ec5196f8b92985e703b89ffa636026"} Oct 30 20:03:55.867999 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:55.867937 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" podStartSLOduration=0.86792082 podStartE2EDuration="867.92082ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:55.864971436 +0000 UTC m=+997.277685242" watchObservedRunningTime="2026-10-30 20:03:55.86792082 +0000 UTC m=+997.280634600" Oct 30 20:03:56.401730 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:56.401686 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" Oct 30 20:03:56.405783 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:56.405749 5088 patch_prober.go:29] interesting pod/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 20:03:56.406054 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:56.406016 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" podUID="fc3ed9fa-b7f0-418f-8ac9-422f9e765027" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 20:03:57.408639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:57.405075 5088 patch_prober.go:29] interesting pod/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 20:03:57.408639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:57.405157 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" podUID="fc3ed9fa-b7f0-418f-8ac9-422f9e765027" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 20:03:58.401642 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:58.401001 5088 patch_prober.go:29] interesting pod/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 20:03:58.401642 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:58.401089 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" podUID="fc3ed9fa-b7f0-418f-8ac9-422f9e765027" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 20:03:59.406126 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:59.406089 5088 patch_prober.go:29] interesting pod/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 20:03:59.407831 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:03:59.407797 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" podUID="fc3ed9fa-b7f0-418f-8ac9-422f9e765027" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 20:04:00.402637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:00.400030 5088 patch_prober.go:29] interesting pod/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 20:04:00.402637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:00.400104 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" podUID="fc3ed9fa-b7f0-418f-8ac9-422f9e765027" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 20:04:01.409097 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:01.409049 5088 patch_prober.go:29] interesting pod/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 20:04:01.409508 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:01.409120 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" podUID="fc3ed9fa-b7f0-418f-8ac9-422f9e765027" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 20:04:02.401033 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:02.400992 5088 patch_prober.go:29] interesting pod/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 20:04:02.401329 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:02.401301 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" podUID="fc3ed9fa-b7f0-418f-8ac9-422f9e765027" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 20:04:03.082874 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:03.082838 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:04:03.103254 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:03.103208 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:04:03.404391 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:03.404302 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" Oct 30 20:04:03.404757 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:03.404733 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" Oct 30 20:04:03.405531 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:03.405512 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" Oct 30 20:04:03.880349 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:03.880317 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:04:11.286390 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:11.286356 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:04:22.437768 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:04:22.437727 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:04:23.936501 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:23.936468 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de4e2008-r28vt"] Oct 30 20:04:23.941715 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:23.941696 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de4e2008-r28vt" Oct 30 20:04:23.961802 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:23.961772 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de4e2008-r28vt"] Oct 30 20:04:23.970841 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:23.970813 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:04:23.971008 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:23.970979 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:04:24.065922 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:24.065890 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8eacb59-ba64-450b-b1bb-3bc6309d0b75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de4e2008-r28vt\" (UID: \"f8eacb59-ba64-450b-b1bb-3bc6309d0b75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de4e2008-r28vt" Oct 30 20:04:24.066105 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:24.065943 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cjw9\" (UniqueName: \"kubernetes.io/projected/f8eacb59-ba64-450b-b1bb-3bc6309d0b75-kube-api-access-4cjw9\") pod \"maas-api-revoke-sub-de4e2008-r28vt\" (UID: \"f8eacb59-ba64-450b-b1bb-3bc6309d0b75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de4e2008-r28vt" Oct 30 20:04:24.066147 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:24.066096 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8eacb59-ba64-450b-b1bb-3bc6309d0b75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de4e2008-r28vt\" (UID: \"f8eacb59-ba64-450b-b1bb-3bc6309d0b75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de4e2008-r28vt" Oct 30 20:04:24.167312 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:24.167269 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8eacb59-ba64-450b-b1bb-3bc6309d0b75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de4e2008-r28vt\" (UID: \"f8eacb59-ba64-450b-b1bb-3bc6309d0b75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de4e2008-r28vt" Oct 30 20:04:24.167519 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:24.167347 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cjw9\" (UniqueName: \"kubernetes.io/projected/f8eacb59-ba64-450b-b1bb-3bc6309d0b75-kube-api-access-4cjw9\") pod \"maas-api-revoke-sub-de4e2008-r28vt\" (UID: \"f8eacb59-ba64-450b-b1bb-3bc6309d0b75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de4e2008-r28vt" Oct 30 20:04:24.167519 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:24.167451 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8eacb59-ba64-450b-b1bb-3bc6309d0b75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de4e2008-r28vt\" (UID: \"f8eacb59-ba64-450b-b1bb-3bc6309d0b75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de4e2008-r28vt" Oct 30 20:04:24.168187 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:24.168161 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8eacb59-ba64-450b-b1bb-3bc6309d0b75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de4e2008-r28vt\" (UID: \"f8eacb59-ba64-450b-b1bb-3bc6309d0b75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de4e2008-r28vt" Oct 30 20:04:24.169731 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:24.169713 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8eacb59-ba64-450b-b1bb-3bc6309d0b75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de4e2008-r28vt\" (UID: \"f8eacb59-ba64-450b-b1bb-3bc6309d0b75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de4e2008-r28vt" Oct 30 20:04:24.194195 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:24.194123 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cjw9\" (UniqueName: \"kubernetes.io/projected/f8eacb59-ba64-450b-b1bb-3bc6309d0b75-kube-api-access-4cjw9\") pod \"maas-api-revoke-sub-de4e2008-r28vt\" (UID: \"f8eacb59-ba64-450b-b1bb-3bc6309d0b75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de4e2008-r28vt" Oct 30 20:04:24.254899 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:24.254861 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de4e2008-r28vt" Oct 30 20:04:24.570006 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:04:24.569971 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8eacb59_ba64_450b_b1bb_3bc6309d0b75.slice/crio-c6ad4236fe97a712dc743b583ab83af80490eb200ffa39eab7b6609b79ca5778 WatchSource:0}: Error finding container c6ad4236fe97a712dc743b583ab83af80490eb200ffa39eab7b6609b79ca5778: Status 404 returned error can't find the container with id c6ad4236fe97a712dc743b583ab83af80490eb200ffa39eab7b6609b79ca5778 Oct 30 20:04:24.571100 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:24.571068 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de4e2008-r28vt"] Oct 30 20:04:25.012450 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.012414 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de4e2008-r28vt" event={"ID":"f8eacb59-ba64-450b-b1bb-3bc6309d0b75","Type":"ContainerStarted","Data":"7b3f67c30439751b9f4993f2dcacdba42057ed92629c3944acdce861e40a79eb"} Oct 30 20:04:25.012889 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.012471 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de4e2008-r28vt" event={"ID":"f8eacb59-ba64-450b-b1bb-3bc6309d0b75","Type":"ContainerStarted","Data":"c6ad4236fe97a712dc743b583ab83af80490eb200ffa39eab7b6609b79ca5778"} Oct 30 20:04:25.067361 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.067293 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de4e2008-r28vt" podStartSLOduration=2.067279213 podStartE2EDuration="2.067279213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:25.063591567 +0000 UTC m=+1026.476305346" watchObservedRunningTime="2026-10-30 20:04:25.067279213 +0000 UTC m=+1026.479992996" Oct 30 20:04:25.220230 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.220141 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae2ba856-r8kv6"] Oct 30 20:04:25.224730 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.224704 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae2ba856-r8kv6" Oct 30 20:04:25.235842 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.235814 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp"] Oct 30 20:04:25.240898 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.240879 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" Oct 30 20:04:25.256370 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.254786 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae2ba856-r8kv6"] Oct 30 20:04:25.272688 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.272651 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp"] Oct 30 20:04:25.281221 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.281185 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m2lv\" (UniqueName: \"kubernetes.io/projected/d5d7b01c-5886-461d-aff5-c6646d23b78a-kube-api-access-6m2lv\") pod \"maas-api-revoke-sub-ae2ba856-r8kv6\" (UID: \"d5d7b01c-5886-461d-aff5-c6646d23b78a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae2ba856-r8kv6" Oct 30 20:04:25.281452 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.281434 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5d7b01c-5886-461d-aff5-c6646d23b78a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae2ba856-r8kv6\" (UID: \"d5d7b01c-5886-461d-aff5-c6646d23b78a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae2ba856-r8kv6" Oct 30 20:04:25.281601 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.281585 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5d7b01c-5886-461d-aff5-c6646d23b78a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae2ba856-r8kv6\" (UID: \"d5d7b01c-5886-461d-aff5-c6646d23b78a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae2ba856-r8kv6" Oct 30 20:04:25.383220 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.383189 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d209c20c-000f-4cc4-9792-5a71eeb0a773-istiod-ca-cert\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp\" (UID: \"d209c20c-000f-4cc4-9792-5a71eeb0a773\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" Oct 30 20:04:25.383416 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.383313 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5d7b01c-5886-461d-aff5-c6646d23b78a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae2ba856-r8kv6\" (UID: \"d5d7b01c-5886-461d-aff5-c6646d23b78a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae2ba856-r8kv6" Oct 30 20:04:25.383416 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.383376 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d209c20c-000f-4cc4-9792-5a71eeb0a773-credential-socket\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp\" (UID: \"d209c20c-000f-4cc4-9792-5a71eeb0a773\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" Oct 30 20:04:25.383539 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.383479 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m2lv\" (UniqueName: \"kubernetes.io/projected/d5d7b01c-5886-461d-aff5-c6646d23b78a-kube-api-access-6m2lv\") pod \"maas-api-revoke-sub-ae2ba856-r8kv6\" (UID: \"d5d7b01c-5886-461d-aff5-c6646d23b78a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae2ba856-r8kv6" Oct 30 20:04:25.383539 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.383511 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d209c20c-000f-4cc4-9792-5a71eeb0a773-istio-data\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp\" (UID: \"d209c20c-000f-4cc4-9792-5a71eeb0a773\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" Oct 30 20:04:25.383680 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.383567 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrm5m\" (UniqueName: \"kubernetes.io/projected/d209c20c-000f-4cc4-9792-5a71eeb0a773-kube-api-access-mrm5m\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp\" (UID: \"d209c20c-000f-4cc4-9792-5a71eeb0a773\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" Oct 30 20:04:25.383738 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.383702 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d209c20c-000f-4cc4-9792-5a71eeb0a773-istio-envoy\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp\" (UID: \"d209c20c-000f-4cc4-9792-5a71eeb0a773\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" Oct 30 20:04:25.383796 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.383742 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d209c20c-000f-4cc4-9792-5a71eeb0a773-istio-podinfo\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp\" (UID: \"d209c20c-000f-4cc4-9792-5a71eeb0a773\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" Oct 30 20:04:25.383796 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.383787 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d209c20c-000f-4cc4-9792-5a71eeb0a773-istio-token\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp\" (UID: \"d209c20c-000f-4cc4-9792-5a71eeb0a773\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" Oct 30 20:04:25.383902 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.383870 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d209c20c-000f-4cc4-9792-5a71eeb0a773-workload-certs\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp\" (UID: \"d209c20c-000f-4cc4-9792-5a71eeb0a773\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" Oct 30 20:04:25.383943 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.383904 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d209c20c-000f-4cc4-9792-5a71eeb0a773-workload-socket\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp\" (UID: \"d209c20c-000f-4cc4-9792-5a71eeb0a773\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" Oct 30 20:04:25.383943 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.383929 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5d7b01c-5886-461d-aff5-c6646d23b78a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae2ba856-r8kv6\" (UID: \"d5d7b01c-5886-461d-aff5-c6646d23b78a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae2ba856-r8kv6" Oct 30 20:04:25.384651 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.384597 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5d7b01c-5886-461d-aff5-c6646d23b78a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae2ba856-r8kv6\" (UID: \"d5d7b01c-5886-461d-aff5-c6646d23b78a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae2ba856-r8kv6" Oct 30 20:04:25.385988 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.385966 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5d7b01c-5886-461d-aff5-c6646d23b78a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae2ba856-r8kv6\" (UID: \"d5d7b01c-5886-461d-aff5-c6646d23b78a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae2ba856-r8kv6" Oct 30 20:04:25.412934 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.412905 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m2lv\" (UniqueName: \"kubernetes.io/projected/d5d7b01c-5886-461d-aff5-c6646d23b78a-kube-api-access-6m2lv\") pod \"maas-api-revoke-sub-ae2ba856-r8kv6\" (UID: \"d5d7b01c-5886-461d-aff5-c6646d23b78a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae2ba856-r8kv6" Oct 30 20:04:25.432864 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.432828 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj"] Oct 30 20:04:25.439431 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.439399 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" Oct 30 20:04:25.458624 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.458575 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj"] Oct 30 20:04:25.485220 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.485185 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrm5m\" (UniqueName: \"kubernetes.io/projected/d209c20c-000f-4cc4-9792-5a71eeb0a773-kube-api-access-mrm5m\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp\" (UID: \"d209c20c-000f-4cc4-9792-5a71eeb0a773\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" Oct 30 20:04:25.485429 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.485264 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d209c20c-000f-4cc4-9792-5a71eeb0a773-istio-envoy\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp\" (UID: \"d209c20c-000f-4cc4-9792-5a71eeb0a773\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" Oct 30 20:04:25.485429 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.485294 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d209c20c-000f-4cc4-9792-5a71eeb0a773-istio-podinfo\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp\" (UID: \"d209c20c-000f-4cc4-9792-5a71eeb0a773\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" Oct 30 20:04:25.485429 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.485322 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d209c20c-000f-4cc4-9792-5a71eeb0a773-istio-token\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp\" (UID: \"d209c20c-000f-4cc4-9792-5a71eeb0a773\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" Oct 30 20:04:25.485429 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.485358 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d209c20c-000f-4cc4-9792-5a71eeb0a773-workload-certs\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp\" (UID: \"d209c20c-000f-4cc4-9792-5a71eeb0a773\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" Oct 30 20:04:25.485429 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.485399 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d209c20c-000f-4cc4-9792-5a71eeb0a773-workload-socket\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp\" (UID: \"d209c20c-000f-4cc4-9792-5a71eeb0a773\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" Oct 30 20:04:25.485718 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.485459 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d209c20c-000f-4cc4-9792-5a71eeb0a773-istiod-ca-cert\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp\" (UID: \"d209c20c-000f-4cc4-9792-5a71eeb0a773\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" Oct 30 20:04:25.485718 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.485600 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d209c20c-000f-4cc4-9792-5a71eeb0a773-credential-socket\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp\" (UID: \"d209c20c-000f-4cc4-9792-5a71eeb0a773\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" Oct 30 20:04:25.485718 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.485665 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d209c20c-000f-4cc4-9792-5a71eeb0a773-istio-data\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp\" (UID: \"d209c20c-000f-4cc4-9792-5a71eeb0a773\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" Oct 30 20:04:25.485877 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.485842 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d209c20c-000f-4cc4-9792-5a71eeb0a773-workload-certs\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp\" (UID: \"d209c20c-000f-4cc4-9792-5a71eeb0a773\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" Oct 30 20:04:25.485934 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.485868 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d209c20c-000f-4cc4-9792-5a71eeb0a773-workload-socket\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp\" (UID: \"d209c20c-000f-4cc4-9792-5a71eeb0a773\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" Oct 30 20:04:25.486051 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.486032 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d209c20c-000f-4cc4-9792-5a71eeb0a773-istio-data\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp\" (UID: \"d209c20c-000f-4cc4-9792-5a71eeb0a773\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" Oct 30 20:04:25.486125 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.486105 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d209c20c-000f-4cc4-9792-5a71eeb0a773-credential-socket\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp\" (UID: \"d209c20c-000f-4cc4-9792-5a71eeb0a773\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" Oct 30 20:04:25.486485 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.486462 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d209c20c-000f-4cc4-9792-5a71eeb0a773-istiod-ca-cert\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp\" (UID: \"d209c20c-000f-4cc4-9792-5a71eeb0a773\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" Oct 30 20:04:25.496654 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.488924 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d209c20c-000f-4cc4-9792-5a71eeb0a773-istio-envoy\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp\" (UID: \"d209c20c-000f-4cc4-9792-5a71eeb0a773\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" Oct 30 20:04:25.496654 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.489167 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d209c20c-000f-4cc4-9792-5a71eeb0a773-istio-podinfo\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp\" (UID: \"d209c20c-000f-4cc4-9792-5a71eeb0a773\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" Oct 30 20:04:25.516756 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.516726 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d209c20c-000f-4cc4-9792-5a71eeb0a773-istio-token\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp\" (UID: \"d209c20c-000f-4cc4-9792-5a71eeb0a773\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" Oct 30 20:04:25.516994 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.516968 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrm5m\" (UniqueName: \"kubernetes.io/projected/d209c20c-000f-4cc4-9792-5a71eeb0a773-kube-api-access-mrm5m\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp\" (UID: \"d209c20c-000f-4cc4-9792-5a71eeb0a773\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" Oct 30 20:04:25.542108 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.542072 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae2ba856-r8kv6" Oct 30 20:04:25.571248 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.571220 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" Oct 30 20:04:25.587062 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.587032 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b55ec6e2-9392-4d8d-8340-ef12f674970c-istio-data\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj\" (UID: \"b55ec6e2-9392-4d8d-8340-ef12f674970c\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" Oct 30 20:04:25.587182 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.587094 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b55ec6e2-9392-4d8d-8340-ef12f674970c-credential-socket\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj\" (UID: \"b55ec6e2-9392-4d8d-8340-ef12f674970c\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" Oct 30 20:04:25.587233 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.587197 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b55ec6e2-9392-4d8d-8340-ef12f674970c-istio-podinfo\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj\" (UID: \"b55ec6e2-9392-4d8d-8340-ef12f674970c\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" Oct 30 20:04:25.587272 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.587256 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xtt5\" (UniqueName: \"kubernetes.io/projected/b55ec6e2-9392-4d8d-8340-ef12f674970c-kube-api-access-5xtt5\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj\" (UID: \"b55ec6e2-9392-4d8d-8340-ef12f674970c\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" Oct 30 20:04:25.587385 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.587360 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b55ec6e2-9392-4d8d-8340-ef12f674970c-istio-token\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj\" (UID: \"b55ec6e2-9392-4d8d-8340-ef12f674970c\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" Oct 30 20:04:25.587588 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.587566 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b55ec6e2-9392-4d8d-8340-ef12f674970c-workload-socket\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj\" (UID: \"b55ec6e2-9392-4d8d-8340-ef12f674970c\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" Oct 30 20:04:25.587745 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.587670 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b55ec6e2-9392-4d8d-8340-ef12f674970c-istio-envoy\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj\" (UID: \"b55ec6e2-9392-4d8d-8340-ef12f674970c\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" Oct 30 20:04:25.587823 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.587777 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b55ec6e2-9392-4d8d-8340-ef12f674970c-istiod-ca-cert\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj\" (UID: \"b55ec6e2-9392-4d8d-8340-ef12f674970c\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" Oct 30 20:04:25.587908 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.587888 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b55ec6e2-9392-4d8d-8340-ef12f674970c-workload-certs\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj\" (UID: \"b55ec6e2-9392-4d8d-8340-ef12f674970c\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" Oct 30 20:04:25.689038 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.688999 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b55ec6e2-9392-4d8d-8340-ef12f674970c-istiod-ca-cert\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj\" (UID: \"b55ec6e2-9392-4d8d-8340-ef12f674970c\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" Oct 30 20:04:25.689243 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.689124 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b55ec6e2-9392-4d8d-8340-ef12f674970c-workload-certs\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj\" (UID: \"b55ec6e2-9392-4d8d-8340-ef12f674970c\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" Oct 30 20:04:25.689243 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.689191 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b55ec6e2-9392-4d8d-8340-ef12f674970c-istio-data\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj\" (UID: \"b55ec6e2-9392-4d8d-8340-ef12f674970c\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" Oct 30 20:04:25.689364 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.689252 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b55ec6e2-9392-4d8d-8340-ef12f674970c-credential-socket\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj\" (UID: \"b55ec6e2-9392-4d8d-8340-ef12f674970c\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" Oct 30 20:04:25.689364 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.689281 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b55ec6e2-9392-4d8d-8340-ef12f674970c-istio-podinfo\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj\" (UID: \"b55ec6e2-9392-4d8d-8340-ef12f674970c\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" Oct 30 20:04:25.689364 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.689308 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xtt5\" (UniqueName: \"kubernetes.io/projected/b55ec6e2-9392-4d8d-8340-ef12f674970c-kube-api-access-5xtt5\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj\" (UID: \"b55ec6e2-9392-4d8d-8340-ef12f674970c\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" Oct 30 20:04:25.689364 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.689345 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b55ec6e2-9392-4d8d-8340-ef12f674970c-istio-token\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj\" (UID: \"b55ec6e2-9392-4d8d-8340-ef12f674970c\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" Oct 30 20:04:25.689564 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.689500 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b55ec6e2-9392-4d8d-8340-ef12f674970c-workload-socket\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj\" (UID: \"b55ec6e2-9392-4d8d-8340-ef12f674970c\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" Oct 30 20:04:25.689564 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.689547 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b55ec6e2-9392-4d8d-8340-ef12f674970c-istio-envoy\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj\" (UID: \"b55ec6e2-9392-4d8d-8340-ef12f674970c\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" Oct 30 20:04:25.690022 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.689862 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b55ec6e2-9392-4d8d-8340-ef12f674970c-istiod-ca-cert\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj\" (UID: \"b55ec6e2-9392-4d8d-8340-ef12f674970c\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" Oct 30 20:04:25.690533 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.690470 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b55ec6e2-9392-4d8d-8340-ef12f674970c-credential-socket\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj\" (UID: \"b55ec6e2-9392-4d8d-8340-ef12f674970c\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" Oct 30 20:04:25.690656 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.690532 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b55ec6e2-9392-4d8d-8340-ef12f674970c-istio-data\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj\" (UID: \"b55ec6e2-9392-4d8d-8340-ef12f674970c\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" Oct 30 20:04:25.691077 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.691053 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b55ec6e2-9392-4d8d-8340-ef12f674970c-workload-certs\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj\" (UID: \"b55ec6e2-9392-4d8d-8340-ef12f674970c\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" Oct 30 20:04:25.691205 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.691185 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b55ec6e2-9392-4d8d-8340-ef12f674970c-workload-socket\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj\" (UID: \"b55ec6e2-9392-4d8d-8340-ef12f674970c\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" Oct 30 20:04:25.692519 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.692477 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b55ec6e2-9392-4d8d-8340-ef12f674970c-istio-envoy\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj\" (UID: \"b55ec6e2-9392-4d8d-8340-ef12f674970c\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" Oct 30 20:04:25.693109 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.693086 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b55ec6e2-9392-4d8d-8340-ef12f674970c-istio-podinfo\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj\" (UID: \"b55ec6e2-9392-4d8d-8340-ef12f674970c\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" Oct 30 20:04:25.717687 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.717646 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b55ec6e2-9392-4d8d-8340-ef12f674970c-istio-token\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj\" (UID: \"b55ec6e2-9392-4d8d-8340-ef12f674970c\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" Oct 30 20:04:25.718048 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.718026 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xtt5\" (UniqueName: \"kubernetes.io/projected/b55ec6e2-9392-4d8d-8340-ef12f674970c-kube-api-access-5xtt5\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj\" (UID: \"b55ec6e2-9392-4d8d-8340-ef12f674970c\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" Oct 30 20:04:25.763093 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.763010 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" Oct 30 20:04:25.866887 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:04:25.866755 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d7b01c_5886_461d_aff5_c6646d23b78a.slice/crio-9ef18cb5a612c5534d641e76d8cad7b4159920b4bba149a7414137b809616759 WatchSource:0}: Error finding container 9ef18cb5a612c5534d641e76d8cad7b4159920b4bba149a7414137b809616759: Status 404 returned error can't find the container with id 9ef18cb5a612c5534d641e76d8cad7b4159920b4bba149a7414137b809616759 Oct 30 20:04:25.872842 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.872814 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae2ba856-r8kv6"] Oct 30 20:04:25.906906 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.906878 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp"] Oct 30 20:04:25.911537 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:04:25.911503 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd209c20c_000f_4cc4_9792_5a71eeb0a773.slice/crio-05dee448ce51c2432d099153b6e1e14a8e6e397ee80ba7b921cd66dc3ced8cfb WatchSource:0}: Error finding container 05dee448ce51c2432d099153b6e1e14a8e6e397ee80ba7b921cd66dc3ced8cfb: Status 404 returned error can't find the container with id 05dee448ce51c2432d099153b6e1e14a8e6e397ee80ba7b921cd66dc3ced8cfb Oct 30 20:04:25.914107 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.914078 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:04:25.914183 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.914145 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:04:25.914183 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:25.914174 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:04:26.023399 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:26.023314 5088 generic.go:356] "Generic (PLEG): container finished" podID="f8eacb59-ba64-450b-b1bb-3bc6309d0b75" containerID="7b3f67c30439751b9f4993f2dcacdba42057ed92629c3944acdce861e40a79eb" exitCode=0 Oct 30 20:04:26.023855 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:26.023411 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de4e2008-r28vt" event={"ID":"f8eacb59-ba64-450b-b1bb-3bc6309d0b75","Type":"ContainerDied","Data":"7b3f67c30439751b9f4993f2dcacdba42057ed92629c3944acdce861e40a79eb"} Oct 30 20:04:26.025695 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:26.025671 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" event={"ID":"d209c20c-000f-4cc4-9792-5a71eeb0a773","Type":"ContainerStarted","Data":"1619335ad2605965c2c5a4f2c72a93cf39d2404688f2c83428c6b8155e763dc7"} Oct 30 20:04:26.025867 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:26.025720 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" event={"ID":"d209c20c-000f-4cc4-9792-5a71eeb0a773","Type":"ContainerStarted","Data":"05dee448ce51c2432d099153b6e1e14a8e6e397ee80ba7b921cd66dc3ced8cfb"} Oct 30 20:04:26.027155 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:26.027133 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae2ba856-r8kv6" event={"ID":"d5d7b01c-5886-461d-aff5-c6646d23b78a","Type":"ContainerStarted","Data":"9ef18cb5a612c5534d641e76d8cad7b4159920b4bba149a7414137b809616759"} Oct 30 20:04:26.079791 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:26.079754 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj"] Oct 30 20:04:26.081078 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:04:26.081038 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb55ec6e2_9392_4d8d_8340_ef12f674970c.slice/crio-0641400eeee93afd1646bb6f071446950ef57acbdd689d01a78b782efc03853a WatchSource:0}: Error finding container 0641400eeee93afd1646bb6f071446950ef57acbdd689d01a78b782efc03853a: Status 404 returned error can't find the container with id 0641400eeee93afd1646bb6f071446950ef57acbdd689d01a78b782efc03853a Oct 30 20:04:26.083793 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:26.083762 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:04:26.083888 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:26.083836 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:04:26.083888 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:26.083870 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:04:26.136819 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:26.136771 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" podStartSLOduration=1.136754289 podStartE2EDuration="1.136754289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:26.13043646 +0000 UTC m=+1027.543150275" watchObservedRunningTime="2026-10-30 20:04:26.136754289 +0000 UTC m=+1027.549468115" Oct 30 20:04:26.571975 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:26.571929 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" Oct 30 20:04:26.573751 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:26.573719 5088 patch_prober.go:29] interesting pod/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 20:04:26.573918 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:26.573777 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" podUID="d209c20c-000f-4cc4-9792-5a71eeb0a773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 20:04:27.064763 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:27.064728 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" event={"ID":"b55ec6e2-9392-4d8d-8340-ef12f674970c","Type":"ContainerStarted","Data":"5a1758a4fb019200747822b65a3e9e033faf2d04d992fa5b6cb39c925af93acd"} Oct 30 20:04:27.065332 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:27.065289 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" event={"ID":"b55ec6e2-9392-4d8d-8340-ef12f674970c","Type":"ContainerStarted","Data":"0641400eeee93afd1646bb6f071446950ef57acbdd689d01a78b782efc03853a"} Oct 30 20:04:27.067286 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:27.066640 5088 generic.go:356] "Generic (PLEG): container finished" podID="d5d7b01c-5886-461d-aff5-c6646d23b78a" containerID="a8b9dc33d1d1d99b8da8acb0302238282ffdfe5573ce59c8ffd52380eedfab24" exitCode=0 Oct 30 20:04:27.067286 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:27.067100 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae2ba856-r8kv6" event={"ID":"d5d7b01c-5886-461d-aff5-c6646d23b78a","Type":"ContainerDied","Data":"a8b9dc33d1d1d99b8da8acb0302238282ffdfe5573ce59c8ffd52380eedfab24"} Oct 30 20:04:27.141651 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:27.140902 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" podStartSLOduration=2.140879134 podStartE2EDuration="2.140879134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:27.120181315 +0000 UTC m=+1028.532895113" watchObservedRunningTime="2026-10-30 20:04:27.140879134 +0000 UTC m=+1028.553592915" Oct 30 20:04:27.377999 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:27.377977 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de4e2008-r28vt" Oct 30 20:04:27.546571 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:27.546542 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8eacb59-ba64-450b-b1bb-3bc6309d0b75-kube-api-access-4cjw9\" (UniqueName: \"kubernetes.io/projected/f8eacb59-ba64-450b-b1bb-3bc6309d0b75-kube-api-access-4cjw9\") pod \"f8eacb59-ba64-450b-b1bb-3bc6309d0b75\" (UID: \"f8eacb59-ba64-450b-b1bb-3bc6309d0b75\") " Oct 30 20:04:27.546803 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:27.546671 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8eacb59-ba64-450b-b1bb-3bc6309d0b75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8eacb59-ba64-450b-b1bb-3bc6309d0b75-maas-api-xks-ca\") pod \"f8eacb59-ba64-450b-b1bb-3bc6309d0b75\" (UID: \"f8eacb59-ba64-450b-b1bb-3bc6309d0b75\") " Oct 30 20:04:27.547227 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:27.547184 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8eacb59-ba64-450b-b1bb-3bc6309d0b75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8eacb59-ba64-450b-b1bb-3bc6309d0b75-maas-api-openshift-ca\") pod \"f8eacb59-ba64-450b-b1bb-3bc6309d0b75\" (UID: \"f8eacb59-ba64-450b-b1bb-3bc6309d0b75\") " Oct 30 20:04:27.547645 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:27.547578 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8eacb59-ba64-450b-b1bb-3bc6309d0b75-maas-api-openshift-ca" pod "f8eacb59-ba64-450b-b1bb-3bc6309d0b75" (UID: "f8eacb59-ba64-450b-b1bb-3bc6309d0b75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:27.547948 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:27.547927 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8eacb59-ba64-450b-b1bb-3bc6309d0b75-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:04:27.548656 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:27.548635 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8eacb59-ba64-450b-b1bb-3bc6309d0b75-kube-api-access-4cjw9" pod "f8eacb59-ba64-450b-b1bb-3bc6309d0b75" (UID: "f8eacb59-ba64-450b-b1bb-3bc6309d0b75"). InnerVolumeSpecName "kube-api-access-4cjw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:27.549956 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:27.549934 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8eacb59-ba64-450b-b1bb-3bc6309d0b75-maas-api-xks-ca" pod "f8eacb59-ba64-450b-b1bb-3bc6309d0b75" (UID: "f8eacb59-ba64-450b-b1bb-3bc6309d0b75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:04:27.572510 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:27.572479 5088 patch_prober.go:29] interesting pod/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 20:04:27.572641 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:27.572528 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" podUID="d209c20c-000f-4cc4-9792-5a71eeb0a773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 20:04:27.649058 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:27.648963 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cjw9\" (UniqueName: \"kubernetes.io/projected/f8eacb59-ba64-450b-b1bb-3bc6309d0b75-kube-api-access-4cjw9\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:04:27.649058 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:27.648996 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8eacb59-ba64-450b-b1bb-3bc6309d0b75-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:04:27.763494 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:27.763461 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" Oct 30 20:04:27.765056 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:27.765028 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 20:04:27.765144 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:27.765082 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" podUID="b55ec6e2-9392-4d8d-8340-ef12f674970c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 20:04:28.074577 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:28.074540 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de4e2008-r28vt" Oct 30 20:04:28.074577 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:28.074551 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de4e2008-r28vt" event={"ID":"f8eacb59-ba64-450b-b1bb-3bc6309d0b75","Type":"ContainerDied","Data":"c6ad4236fe97a712dc743b583ab83af80490eb200ffa39eab7b6609b79ca5778"} Oct 30 20:04:28.075254 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:28.074653 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6ad4236fe97a712dc743b583ab83af80490eb200ffa39eab7b6609b79ca5778" Oct 30 20:04:28.480071 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:28.480044 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae2ba856-r8kv6" Oct 30 20:04:28.572479 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:28.572445 5088 patch_prober.go:29] interesting pod/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 20:04:28.572688 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:28.572495 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" podUID="d209c20c-000f-4cc4-9792-5a71eeb0a773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 20:04:28.664042 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:28.663989 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5d7b01c-5886-461d-aff5-c6646d23b78a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5d7b01c-5886-461d-aff5-c6646d23b78a-maas-api-openshift-ca\") pod \"d5d7b01c-5886-461d-aff5-c6646d23b78a\" (UID: \"d5d7b01c-5886-461d-aff5-c6646d23b78a\") " Oct 30 20:04:28.664231 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:28.664050 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5d7b01c-5886-461d-aff5-c6646d23b78a-kube-api-access-6m2lv\" (UniqueName: \"kubernetes.io/projected/d5d7b01c-5886-461d-aff5-c6646d23b78a-kube-api-access-6m2lv\") pod \"d5d7b01c-5886-461d-aff5-c6646d23b78a\" (UID: \"d5d7b01c-5886-461d-aff5-c6646d23b78a\") " Oct 30 20:04:28.664231 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:28.664209 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5d7b01c-5886-461d-aff5-c6646d23b78a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5d7b01c-5886-461d-aff5-c6646d23b78a-maas-api-xks-ca\") pod \"d5d7b01c-5886-461d-aff5-c6646d23b78a\" (UID: \"d5d7b01c-5886-461d-aff5-c6646d23b78a\") " Oct 30 20:04:28.664773 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:28.664735 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5d7b01c-5886-461d-aff5-c6646d23b78a-maas-api-openshift-ca" pod "d5d7b01c-5886-461d-aff5-c6646d23b78a" (UID: "d5d7b01c-5886-461d-aff5-c6646d23b78a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:28.664903 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:28.664828 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5d7b01c-5886-461d-aff5-c6646d23b78a-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:04:28.666882 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:28.666789 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5d7b01c-5886-461d-aff5-c6646d23b78a-maas-api-xks-ca" pod "d5d7b01c-5886-461d-aff5-c6646d23b78a" (UID: "d5d7b01c-5886-461d-aff5-c6646d23b78a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:04:28.667293 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:28.667270 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5d7b01c-5886-461d-aff5-c6646d23b78a-kube-api-access-6m2lv" pod "d5d7b01c-5886-461d-aff5-c6646d23b78a" (UID: "d5d7b01c-5886-461d-aff5-c6646d23b78a"). InnerVolumeSpecName "kube-api-access-6m2lv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:28.764437 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:28.764410 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 20:04:28.764563 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:28.764461 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" podUID="b55ec6e2-9392-4d8d-8340-ef12f674970c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 20:04:28.766600 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:28.766579 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m2lv\" (UniqueName: \"kubernetes.io/projected/d5d7b01c-5886-461d-aff5-c6646d23b78a-kube-api-access-6m2lv\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:04:28.766663 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:28.766606 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5d7b01c-5886-461d-aff5-c6646d23b78a-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:04:29.151515 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:29.151442 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae2ba856-r8kv6" Oct 30 20:04:29.151941 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:29.151687 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae2ba856-r8kv6" event={"ID":"d5d7b01c-5886-461d-aff5-c6646d23b78a","Type":"ContainerDied","Data":"9ef18cb5a612c5534d641e76d8cad7b4159920b4bba149a7414137b809616759"} Oct 30 20:04:29.151941 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:29.151740 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ef18cb5a612c5534d641e76d8cad7b4159920b4bba149a7414137b809616759" Oct 30 20:04:29.571582 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:29.571548 5088 patch_prober.go:29] interesting pod/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 20:04:29.571765 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:29.571603 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" podUID="d209c20c-000f-4cc4-9792-5a71eeb0a773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 20:04:29.763475 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:29.763435 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 20:04:29.763686 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:29.763488 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" podUID="b55ec6e2-9392-4d8d-8340-ef12f674970c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 20:04:29.978193 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:29.978113 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55f6e9a1-znm9t"] Oct 30 20:04:29.978761 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:29.978742 5088 cpu_manager.go:414] "containerMap: removing container" podUID="d5d7b01c-5886-461d-aff5-c6646d23b78a" containerName="revoke-keys" Oct 30 20:04:29.978761 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:29.978760 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5d7b01c-5886-461d-aff5-c6646d23b78a" containerName="revoke-keys" Oct 30 20:04:29.978878 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:29.978796 5088 cpu_manager.go:414] "containerMap: removing container" podUID="f8eacb59-ba64-450b-b1bb-3bc6309d0b75" containerName="revoke-keys" Oct 30 20:04:29.978878 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:29.978801 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8eacb59-ba64-450b-b1bb-3bc6309d0b75" containerName="revoke-keys" Oct 30 20:04:29.978946 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:29.978885 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5d7b01c-5886-461d-aff5-c6646d23b78a" containerName="revoke-keys" Oct 30 20:04:29.978946 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:29.978899 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8eacb59-ba64-450b-b1bb-3bc6309d0b75" containerName="revoke-keys" Oct 30 20:04:29.982656 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:29.982641 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f6e9a1-znm9t" Oct 30 20:04:30.006637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:30.005039 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:04:30.026637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:30.011406 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:04:30.051253 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:30.050682 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55f6e9a1-znm9t"] Oct 30 20:04:30.135635 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:30.135013 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55d65e3f-e3a8-4836-9d57-50c3dfee5a0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55f6e9a1-znm9t\" (UID: \"55d65e3f-e3a8-4836-9d57-50c3dfee5a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f6e9a1-znm9t" Oct 30 20:04:30.135635 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:30.135142 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55d65e3f-e3a8-4836-9d57-50c3dfee5a0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55f6e9a1-znm9t\" (UID: \"55d65e3f-e3a8-4836-9d57-50c3dfee5a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f6e9a1-znm9t" Oct 30 20:04:30.135635 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:30.135221 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5n4d\" (UniqueName: \"kubernetes.io/projected/55d65e3f-e3a8-4836-9d57-50c3dfee5a0a-kube-api-access-h5n4d\") pod \"maas-api-revoke-sub-55f6e9a1-znm9t\" (UID: \"55d65e3f-e3a8-4836-9d57-50c3dfee5a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f6e9a1-znm9t" Oct 30 20:04:30.236014 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:30.235985 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55d65e3f-e3a8-4836-9d57-50c3dfee5a0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55f6e9a1-znm9t\" (UID: \"55d65e3f-e3a8-4836-9d57-50c3dfee5a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f6e9a1-znm9t" Oct 30 20:04:30.236416 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:30.236055 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5n4d\" (UniqueName: \"kubernetes.io/projected/55d65e3f-e3a8-4836-9d57-50c3dfee5a0a-kube-api-access-h5n4d\") pod \"maas-api-revoke-sub-55f6e9a1-znm9t\" (UID: \"55d65e3f-e3a8-4836-9d57-50c3dfee5a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f6e9a1-znm9t" Oct 30 20:04:30.236416 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:30.236149 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55d65e3f-e3a8-4836-9d57-50c3dfee5a0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55f6e9a1-znm9t\" (UID: \"55d65e3f-e3a8-4836-9d57-50c3dfee5a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f6e9a1-znm9t" Oct 30 20:04:30.236702 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:30.236676 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55d65e3f-e3a8-4836-9d57-50c3dfee5a0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55f6e9a1-znm9t\" (UID: \"55d65e3f-e3a8-4836-9d57-50c3dfee5a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f6e9a1-znm9t" Oct 30 20:04:30.238636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:30.238591 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55d65e3f-e3a8-4836-9d57-50c3dfee5a0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55f6e9a1-znm9t\" (UID: \"55d65e3f-e3a8-4836-9d57-50c3dfee5a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f6e9a1-znm9t" Oct 30 20:04:30.262979 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:30.262924 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5n4d\" (UniqueName: \"kubernetes.io/projected/55d65e3f-e3a8-4836-9d57-50c3dfee5a0a-kube-api-access-h5n4d\") pod \"maas-api-revoke-sub-55f6e9a1-znm9t\" (UID: \"55d65e3f-e3a8-4836-9d57-50c3dfee5a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f6e9a1-znm9t" Oct 30 20:04:30.335637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:30.335270 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f6e9a1-znm9t" Oct 30 20:04:30.572382 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:30.572288 5088 patch_prober.go:29] interesting pod/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 20:04:30.572382 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:30.572341 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" podUID="d209c20c-000f-4cc4-9792-5a71eeb0a773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 20:04:30.747791 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:30.745784 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55f6e9a1-znm9t"] Oct 30 20:04:30.764790 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:30.764760 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 20:04:30.764891 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:30.764814 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" podUID="b55ec6e2-9392-4d8d-8340-ef12f674970c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 20:04:31.175161 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:31.175124 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f6e9a1-znm9t" event={"ID":"55d65e3f-e3a8-4836-9d57-50c3dfee5a0a","Type":"ContainerStarted","Data":"a075bc844ce526b73ea161dbb634c0376055def669926fe19de10d1fae182038"} Oct 30 20:04:31.175321 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:31.175211 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f6e9a1-znm9t" event={"ID":"55d65e3f-e3a8-4836-9d57-50c3dfee5a0a","Type":"ContainerStarted","Data":"f867531522f6f3600685fb01abfe56749d06e6f73838bfb438ffbfd985cfd311"} Oct 30 20:04:31.258788 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:31.258721 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f6e9a1-znm9t" podStartSLOduration=2.258698327 podStartE2EDuration="2.258698327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:31.244953389 +0000 UTC m=+1032.657667169" watchObservedRunningTime="2026-10-30 20:04:31.258698327 +0000 UTC m=+1032.671412113" Oct 30 20:04:31.572495 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:31.572415 5088 patch_prober.go:29] interesting pod/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 20:04:31.572495 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:31.572469 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" podUID="d209c20c-000f-4cc4-9792-5a71eeb0a773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 20:04:31.765639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:31.765590 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 20:04:31.765779 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:31.765686 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" podUID="b55ec6e2-9392-4d8d-8340-ef12f674970c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 20:04:32.189905 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:32.189864 5088 generic.go:356] "Generic (PLEG): container finished" podID="55d65e3f-e3a8-4836-9d57-50c3dfee5a0a" containerID="a075bc844ce526b73ea161dbb634c0376055def669926fe19de10d1fae182038" exitCode=0 Oct 30 20:04:32.190137 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:32.189938 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f6e9a1-znm9t" event={"ID":"55d65e3f-e3a8-4836-9d57-50c3dfee5a0a","Type":"ContainerDied","Data":"a075bc844ce526b73ea161dbb634c0376055def669926fe19de10d1fae182038"} Oct 30 20:04:32.572450 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:32.572412 5088 patch_prober.go:29] interesting pod/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 20:04:32.572863 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:32.572468 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" podUID="d209c20c-000f-4cc4-9792-5a71eeb0a773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 20:04:32.764051 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:32.764011 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 20:04:32.764228 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:32.764081 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" podUID="b55ec6e2-9392-4d8d-8340-ef12f674970c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 20:04:33.307660 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:33.307476 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:04:33.436180 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:33.436150 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f6e9a1-znm9t" Oct 30 20:04:33.495489 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:33.495455 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55d65e3f-e3a8-4836-9d57-50c3dfee5a0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55d65e3f-e3a8-4836-9d57-50c3dfee5a0a-maas-api-openshift-ca\") pod \"55d65e3f-e3a8-4836-9d57-50c3dfee5a0a\" (UID: \"55d65e3f-e3a8-4836-9d57-50c3dfee5a0a\") " Oct 30 20:04:33.496120 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:33.496099 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55d65e3f-e3a8-4836-9d57-50c3dfee5a0a-kube-api-access-h5n4d\" (UniqueName: \"kubernetes.io/projected/55d65e3f-e3a8-4836-9d57-50c3dfee5a0a-kube-api-access-h5n4d\") pod \"55d65e3f-e3a8-4836-9d57-50c3dfee5a0a\" (UID: \"55d65e3f-e3a8-4836-9d57-50c3dfee5a0a\") " Oct 30 20:04:33.496633 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:33.496599 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55d65e3f-e3a8-4836-9d57-50c3dfee5a0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55d65e3f-e3a8-4836-9d57-50c3dfee5a0a-maas-api-xks-ca\") pod \"55d65e3f-e3a8-4836-9d57-50c3dfee5a0a\" (UID: \"55d65e3f-e3a8-4836-9d57-50c3dfee5a0a\") " Oct 30 20:04:33.497037 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:33.496036 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55d65e3f-e3a8-4836-9d57-50c3dfee5a0a-maas-api-openshift-ca" pod "55d65e3f-e3a8-4836-9d57-50c3dfee5a0a" (UID: "55d65e3f-e3a8-4836-9d57-50c3dfee5a0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:33.497381 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:33.497358 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55d65e3f-e3a8-4836-9d57-50c3dfee5a0a-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:04:33.500908 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:33.500884 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55d65e3f-e3a8-4836-9d57-50c3dfee5a0a-kube-api-access-h5n4d" pod "55d65e3f-e3a8-4836-9d57-50c3dfee5a0a" (UID: "55d65e3f-e3a8-4836-9d57-50c3dfee5a0a"). InnerVolumeSpecName "kube-api-access-h5n4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:33.502859 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:33.502834 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55d65e3f-e3a8-4836-9d57-50c3dfee5a0a-maas-api-xks-ca" pod "55d65e3f-e3a8-4836-9d57-50c3dfee5a0a" (UID: "55d65e3f-e3a8-4836-9d57-50c3dfee5a0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:04:33.598926 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:33.598882 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5n4d\" (UniqueName: \"kubernetes.io/projected/55d65e3f-e3a8-4836-9d57-50c3dfee5a0a-kube-api-access-h5n4d\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:04:33.598926 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:33.598921 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55d65e3f-e3a8-4836-9d57-50c3dfee5a0a-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:04:33.703800 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:33.702951 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" Oct 30 20:04:33.703800 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:33.703744 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" Oct 30 20:04:33.705079 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:33.705029 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" Oct 30 20:04:33.866048 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:33.865955 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" Oct 30 20:04:33.867680 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:33.867656 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" Oct 30 20:04:33.874560 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:33.874536 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" Oct 30 20:04:34.204128 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:34.204053 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f6e9a1-znm9t" Oct 30 20:04:34.204128 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:34.204093 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f6e9a1-znm9t" event={"ID":"55d65e3f-e3a8-4836-9d57-50c3dfee5a0a","Type":"ContainerDied","Data":"f867531522f6f3600685fb01abfe56749d06e6f73838bfb438ffbfd985cfd311"} Oct 30 20:04:34.204317 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:34.204143 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f867531522f6f3600685fb01abfe56749d06e6f73838bfb438ffbfd985cfd311" Oct 30 20:04:35.023373 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:35.023338 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5148e465-4pb95"] Oct 30 20:04:35.024348 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:35.024326 5088 cpu_manager.go:414] "containerMap: removing container" podUID="55d65e3f-e3a8-4836-9d57-50c3dfee5a0a" containerName="revoke-keys" Oct 30 20:04:35.024400 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:35.024353 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55d65e3f-e3a8-4836-9d57-50c3dfee5a0a" containerName="revoke-keys" Oct 30 20:04:35.024520 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:35.024504 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="55d65e3f-e3a8-4836-9d57-50c3dfee5a0a" containerName="revoke-keys" Oct 30 20:04:35.029031 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:35.029010 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5148e465-4pb95" Oct 30 20:04:35.040177 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:35.040150 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bae93c7f-9009-40d5-8c0f-1157e00a46dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5148e465-4pb95\" (UID: \"bae93c7f-9009-40d5-8c0f-1157e00a46dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5148e465-4pb95" Oct 30 20:04:35.040516 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:35.040493 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bae93c7f-9009-40d5-8c0f-1157e00a46dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5148e465-4pb95\" (UID: \"bae93c7f-9009-40d5-8c0f-1157e00a46dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5148e465-4pb95" Oct 30 20:04:35.041006 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:35.040982 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4llg\" (UniqueName: \"kubernetes.io/projected/bae93c7f-9009-40d5-8c0f-1157e00a46dc-kube-api-access-j4llg\") pod \"maas-api-revoke-sub-5148e465-4pb95\" (UID: \"bae93c7f-9009-40d5-8c0f-1157e00a46dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5148e465-4pb95" Oct 30 20:04:35.054639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:35.054511 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5148e465-4pb95"] Oct 30 20:04:35.061195 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:35.061170 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:04:35.061443 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:35.061422 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:04:35.142185 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:35.142153 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bae93c7f-9009-40d5-8c0f-1157e00a46dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5148e465-4pb95\" (UID: \"bae93c7f-9009-40d5-8c0f-1157e00a46dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5148e465-4pb95" Oct 30 20:04:35.142388 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:35.142300 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bae93c7f-9009-40d5-8c0f-1157e00a46dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5148e465-4pb95\" (UID: \"bae93c7f-9009-40d5-8c0f-1157e00a46dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5148e465-4pb95" Oct 30 20:04:35.142458 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:35.142441 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4llg\" (UniqueName: \"kubernetes.io/projected/bae93c7f-9009-40d5-8c0f-1157e00a46dc-kube-api-access-j4llg\") pod \"maas-api-revoke-sub-5148e465-4pb95\" (UID: \"bae93c7f-9009-40d5-8c0f-1157e00a46dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5148e465-4pb95" Oct 30 20:04:35.143133 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:35.143092 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bae93c7f-9009-40d5-8c0f-1157e00a46dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5148e465-4pb95\" (UID: \"bae93c7f-9009-40d5-8c0f-1157e00a46dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5148e465-4pb95" Oct 30 20:04:35.144649 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:35.144626 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bae93c7f-9009-40d5-8c0f-1157e00a46dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5148e465-4pb95\" (UID: \"bae93c7f-9009-40d5-8c0f-1157e00a46dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5148e465-4pb95" Oct 30 20:04:35.169040 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:35.169010 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4llg\" (UniqueName: \"kubernetes.io/projected/bae93c7f-9009-40d5-8c0f-1157e00a46dc-kube-api-access-j4llg\") pod \"maas-api-revoke-sub-5148e465-4pb95\" (UID: \"bae93c7f-9009-40d5-8c0f-1157e00a46dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5148e465-4pb95" Oct 30 20:04:35.352215 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:35.352140 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5148e465-4pb95" Oct 30 20:04:35.870453 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:35.870425 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5148e465-4pb95"] Oct 30 20:04:35.871105 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:04:35.871079 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae93c7f_9009_40d5_8c0f_1157e00a46dc.slice/crio-60be54e2604e50ab64c24a362170b305fc22af3abea34908dbc92a2aa2da3d14 WatchSource:0}: Error finding container 60be54e2604e50ab64c24a362170b305fc22af3abea34908dbc92a2aa2da3d14: Status 404 returned error can't find the container with id 60be54e2604e50ab64c24a362170b305fc22af3abea34908dbc92a2aa2da3d14 Oct 30 20:04:36.222924 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:36.222887 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5148e465-4pb95" event={"ID":"bae93c7f-9009-40d5-8c0f-1157e00a46dc","Type":"ContainerStarted","Data":"60be54e2604e50ab64c24a362170b305fc22af3abea34908dbc92a2aa2da3d14"} Oct 30 20:04:36.790094 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:36.790051 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2757fd15-pbhr2"] Oct 30 20:04:36.794339 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:36.794317 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2757fd15-pbhr2" Oct 30 20:04:36.817772 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:36.817742 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2757fd15-pbhr2"] Oct 30 20:04:36.931677 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:36.931646 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghtrq\" (UniqueName: \"kubernetes.io/projected/784739f5-87b2-40e2-829e-95e3cab8148d-kube-api-access-ghtrq\") pod \"maas-api-revoke-sub-2757fd15-pbhr2\" (UID: \"784739f5-87b2-40e2-829e-95e3cab8148d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2757fd15-pbhr2" Oct 30 20:04:36.931766 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:36.931726 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/784739f5-87b2-40e2-829e-95e3cab8148d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2757fd15-pbhr2\" (UID: \"784739f5-87b2-40e2-829e-95e3cab8148d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2757fd15-pbhr2" Oct 30 20:04:36.931811 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:36.931778 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/784739f5-87b2-40e2-829e-95e3cab8148d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2757fd15-pbhr2\" (UID: \"784739f5-87b2-40e2-829e-95e3cab8148d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2757fd15-pbhr2" Oct 30 20:04:37.033586 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:37.033553 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghtrq\" (UniqueName: \"kubernetes.io/projected/784739f5-87b2-40e2-829e-95e3cab8148d-kube-api-access-ghtrq\") pod \"maas-api-revoke-sub-2757fd15-pbhr2\" (UID: \"784739f5-87b2-40e2-829e-95e3cab8148d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2757fd15-pbhr2" Oct 30 20:04:37.033777 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:37.033591 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/784739f5-87b2-40e2-829e-95e3cab8148d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2757fd15-pbhr2\" (UID: \"784739f5-87b2-40e2-829e-95e3cab8148d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2757fd15-pbhr2" Oct 30 20:04:37.033777 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:37.033656 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/784739f5-87b2-40e2-829e-95e3cab8148d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2757fd15-pbhr2\" (UID: \"784739f5-87b2-40e2-829e-95e3cab8148d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2757fd15-pbhr2" Oct 30 20:04:37.034301 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:37.034271 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/784739f5-87b2-40e2-829e-95e3cab8148d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2757fd15-pbhr2\" (UID: \"784739f5-87b2-40e2-829e-95e3cab8148d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2757fd15-pbhr2" Oct 30 20:04:37.036002 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:37.035981 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/784739f5-87b2-40e2-829e-95e3cab8148d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2757fd15-pbhr2\" (UID: \"784739f5-87b2-40e2-829e-95e3cab8148d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2757fd15-pbhr2" Oct 30 20:04:37.062587 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:37.062527 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghtrq\" (UniqueName: \"kubernetes.io/projected/784739f5-87b2-40e2-829e-95e3cab8148d-kube-api-access-ghtrq\") pod \"maas-api-revoke-sub-2757fd15-pbhr2\" (UID: \"784739f5-87b2-40e2-829e-95e3cab8148d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2757fd15-pbhr2" Oct 30 20:04:37.107692 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:37.107665 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2757fd15-pbhr2" Oct 30 20:04:37.244969 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:37.244016 5088 generic.go:356] "Generic (PLEG): container finished" podID="bae93c7f-9009-40d5-8c0f-1157e00a46dc" containerID="d5a49f9350f40f63b0f7f4562bc4374c91f2d77dccb5ded6d804cb5a8591faf3" exitCode=0 Oct 30 20:04:37.245662 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:37.245607 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5148e465-4pb95" event={"ID":"bae93c7f-9009-40d5-8c0f-1157e00a46dc","Type":"ContainerDied","Data":"d5a49f9350f40f63b0f7f4562bc4374c91f2d77dccb5ded6d804cb5a8591faf3"} Oct 30 20:04:37.419358 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:37.419332 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2757fd15-pbhr2"] Oct 30 20:04:37.419870 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:04:37.419842 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod784739f5_87b2_40e2_829e_95e3cab8148d.slice/crio-9751241ddf9466dfcffb571992dad571311813b4f9bfcc9536b264cffe547cf3 WatchSource:0}: Error finding container 9751241ddf9466dfcffb571992dad571311813b4f9bfcc9536b264cffe547cf3: Status 404 returned error can't find the container with id 9751241ddf9466dfcffb571992dad571311813b4f9bfcc9536b264cffe547cf3 Oct 30 20:04:38.253265 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:38.253234 5088 generic.go:356] "Generic (PLEG): container finished" podID="784739f5-87b2-40e2-829e-95e3cab8148d" containerID="6110fd54edcf054e9561252d08db05b212c4b3292529b849022ef776b6dc56f4" exitCode=0 Oct 30 20:04:38.253597 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:38.253325 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2757fd15-pbhr2" event={"ID":"784739f5-87b2-40e2-829e-95e3cab8148d","Type":"ContainerDied","Data":"6110fd54edcf054e9561252d08db05b212c4b3292529b849022ef776b6dc56f4"} Oct 30 20:04:38.253597 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:38.253403 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2757fd15-pbhr2" event={"ID":"784739f5-87b2-40e2-829e-95e3cab8148d","Type":"ContainerStarted","Data":"9751241ddf9466dfcffb571992dad571311813b4f9bfcc9536b264cffe547cf3"} Oct 30 20:04:38.409254 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:38.409228 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5148e465-4pb95" Oct 30 20:04:38.553709 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:38.553675 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bae93c7f-9009-40d5-8c0f-1157e00a46dc-kube-api-access-j4llg\" (UniqueName: \"kubernetes.io/projected/bae93c7f-9009-40d5-8c0f-1157e00a46dc-kube-api-access-j4llg\") pod \"bae93c7f-9009-40d5-8c0f-1157e00a46dc\" (UID: \"bae93c7f-9009-40d5-8c0f-1157e00a46dc\") " Oct 30 20:04:38.553903 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:38.553887 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bae93c7f-9009-40d5-8c0f-1157e00a46dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bae93c7f-9009-40d5-8c0f-1157e00a46dc-maas-api-xks-ca\") pod \"bae93c7f-9009-40d5-8c0f-1157e00a46dc\" (UID: \"bae93c7f-9009-40d5-8c0f-1157e00a46dc\") " Oct 30 20:04:38.554049 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:38.554031 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bae93c7f-9009-40d5-8c0f-1157e00a46dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bae93c7f-9009-40d5-8c0f-1157e00a46dc-maas-api-openshift-ca\") pod \"bae93c7f-9009-40d5-8c0f-1157e00a46dc\" (UID: \"bae93c7f-9009-40d5-8c0f-1157e00a46dc\") " Oct 30 20:04:38.554410 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:38.554383 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bae93c7f-9009-40d5-8c0f-1157e00a46dc-maas-api-openshift-ca" pod "bae93c7f-9009-40d5-8c0f-1157e00a46dc" (UID: "bae93c7f-9009-40d5-8c0f-1157e00a46dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:38.554514 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:38.554490 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bae93c7f-9009-40d5-8c0f-1157e00a46dc-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:04:38.555959 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:38.555931 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bae93c7f-9009-40d5-8c0f-1157e00a46dc-maas-api-xks-ca" pod "bae93c7f-9009-40d5-8c0f-1157e00a46dc" (UID: "bae93c7f-9009-40d5-8c0f-1157e00a46dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:04:38.556042 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:38.555933 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bae93c7f-9009-40d5-8c0f-1157e00a46dc-kube-api-access-j4llg" pod "bae93c7f-9009-40d5-8c0f-1157e00a46dc" (UID: "bae93c7f-9009-40d5-8c0f-1157e00a46dc"). InnerVolumeSpecName "kube-api-access-j4llg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:38.655910 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:38.655872 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4llg\" (UniqueName: \"kubernetes.io/projected/bae93c7f-9009-40d5-8c0f-1157e00a46dc-kube-api-access-j4llg\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:04:38.655910 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:38.655904 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bae93c7f-9009-40d5-8c0f-1157e00a46dc-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:04:39.261300 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:39.261267 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5148e465-4pb95" Oct 30 20:04:39.261737 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:39.261296 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5148e465-4pb95" event={"ID":"bae93c7f-9009-40d5-8c0f-1157e00a46dc","Type":"ContainerDied","Data":"60be54e2604e50ab64c24a362170b305fc22af3abea34908dbc92a2aa2da3d14"} Oct 30 20:04:39.261737 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:39.261344 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60be54e2604e50ab64c24a362170b305fc22af3abea34908dbc92a2aa2da3d14" Oct 30 20:04:39.459037 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:39.459015 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2757fd15-pbhr2" Oct 30 20:04:39.576235 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:39.576149 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/784739f5-87b2-40e2-829e-95e3cab8148d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/784739f5-87b2-40e2-829e-95e3cab8148d-maas-api-xks-ca\") pod \"784739f5-87b2-40e2-829e-95e3cab8148d\" (UID: \"784739f5-87b2-40e2-829e-95e3cab8148d\") " Oct 30 20:04:39.576403 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:39.576371 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/784739f5-87b2-40e2-829e-95e3cab8148d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/784739f5-87b2-40e2-829e-95e3cab8148d-maas-api-openshift-ca\") pod \"784739f5-87b2-40e2-829e-95e3cab8148d\" (UID: \"784739f5-87b2-40e2-829e-95e3cab8148d\") " Oct 30 20:04:39.576480 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:39.576420 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/784739f5-87b2-40e2-829e-95e3cab8148d-kube-api-access-ghtrq\" (UniqueName: \"kubernetes.io/projected/784739f5-87b2-40e2-829e-95e3cab8148d-kube-api-access-ghtrq\") pod \"784739f5-87b2-40e2-829e-95e3cab8148d\" (UID: \"784739f5-87b2-40e2-829e-95e3cab8148d\") " Oct 30 20:04:39.576757 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:39.576731 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/784739f5-87b2-40e2-829e-95e3cab8148d-maas-api-openshift-ca" pod "784739f5-87b2-40e2-829e-95e3cab8148d" (UID: "784739f5-87b2-40e2-829e-95e3cab8148d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:39.576955 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:39.576939 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/784739f5-87b2-40e2-829e-95e3cab8148d-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:04:39.578311 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:39.578291 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/784739f5-87b2-40e2-829e-95e3cab8148d-maas-api-xks-ca" pod "784739f5-87b2-40e2-829e-95e3cab8148d" (UID: "784739f5-87b2-40e2-829e-95e3cab8148d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:04:39.578436 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:39.578418 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/784739f5-87b2-40e2-829e-95e3cab8148d-kube-api-access-ghtrq" pod "784739f5-87b2-40e2-829e-95e3cab8148d" (UID: "784739f5-87b2-40e2-829e-95e3cab8148d"). InnerVolumeSpecName "kube-api-access-ghtrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:39.678409 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:39.678374 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/784739f5-87b2-40e2-829e-95e3cab8148d-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:04:39.678409 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:39.678408 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghtrq\" (UniqueName: \"kubernetes.io/projected/784739f5-87b2-40e2-829e-95e3cab8148d-kube-api-access-ghtrq\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:04:40.270321 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:40.270291 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2757fd15-pbhr2" Oct 30 20:04:40.270737 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:40.270286 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2757fd15-pbhr2" event={"ID":"784739f5-87b2-40e2-829e-95e3cab8148d","Type":"ContainerDied","Data":"9751241ddf9466dfcffb571992dad571311813b4f9bfcc9536b264cffe547cf3"} Oct 30 20:04:40.270737 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:40.270412 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9751241ddf9466dfcffb571992dad571311813b4f9bfcc9536b264cffe547cf3" Oct 30 20:04:41.648072 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:41.648040 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-320efee7-2tctv"] Oct 30 20:04:41.648697 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:41.648672 5088 cpu_manager.go:414] "containerMap: removing container" podUID="bae93c7f-9009-40d5-8c0f-1157e00a46dc" containerName="revoke-keys" Oct 30 20:04:41.648697 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:41.648697 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bae93c7f-9009-40d5-8c0f-1157e00a46dc" containerName="revoke-keys" Oct 30 20:04:41.648812 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:41.648738 5088 cpu_manager.go:414] "containerMap: removing container" podUID="784739f5-87b2-40e2-829e-95e3cab8148d" containerName="revoke-keys" Oct 30 20:04:41.648812 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:41.648743 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="784739f5-87b2-40e2-829e-95e3cab8148d" containerName="revoke-keys" Oct 30 20:04:41.648884 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:41.648840 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="bae93c7f-9009-40d5-8c0f-1157e00a46dc" containerName="revoke-keys" Oct 30 20:04:41.648884 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:41.648857 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="784739f5-87b2-40e2-829e-95e3cab8148d" containerName="revoke-keys" Oct 30 20:04:41.652461 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:41.652444 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-320efee7-2tctv" Oct 30 20:04:41.682740 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:41.682704 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-320efee7-2tctv"] Oct 30 20:04:41.689982 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:41.689954 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:04:41.690123 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:41.690035 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:04:41.805115 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:41.805077 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3c58e20-4219-4650-af7e-bdf6795c8cf9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-320efee7-2tctv\" (UID: \"b3c58e20-4219-4650-af7e-bdf6795c8cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-320efee7-2tctv" Oct 30 20:04:41.805115 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:41.805118 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqmpq\" (UniqueName: \"kubernetes.io/projected/b3c58e20-4219-4650-af7e-bdf6795c8cf9-kube-api-access-gqmpq\") pod \"maas-api-revoke-sub-320efee7-2tctv\" (UID: \"b3c58e20-4219-4650-af7e-bdf6795c8cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-320efee7-2tctv" Oct 30 20:04:41.805354 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:41.805241 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3c58e20-4219-4650-af7e-bdf6795c8cf9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-320efee7-2tctv\" (UID: \"b3c58e20-4219-4650-af7e-bdf6795c8cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-320efee7-2tctv" Oct 30 20:04:41.906308 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:41.906228 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3c58e20-4219-4650-af7e-bdf6795c8cf9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-320efee7-2tctv\" (UID: \"b3c58e20-4219-4650-af7e-bdf6795c8cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-320efee7-2tctv" Oct 30 20:04:41.906308 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:41.906266 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqmpq\" (UniqueName: \"kubernetes.io/projected/b3c58e20-4219-4650-af7e-bdf6795c8cf9-kube-api-access-gqmpq\") pod \"maas-api-revoke-sub-320efee7-2tctv\" (UID: \"b3c58e20-4219-4650-af7e-bdf6795c8cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-320efee7-2tctv" Oct 30 20:04:41.906486 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:41.906325 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3c58e20-4219-4650-af7e-bdf6795c8cf9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-320efee7-2tctv\" (UID: \"b3c58e20-4219-4650-af7e-bdf6795c8cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-320efee7-2tctv" Oct 30 20:04:41.907142 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:41.907123 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3c58e20-4219-4650-af7e-bdf6795c8cf9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-320efee7-2tctv\" (UID: \"b3c58e20-4219-4650-af7e-bdf6795c8cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-320efee7-2tctv" Oct 30 20:04:41.908552 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:41.908529 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3c58e20-4219-4650-af7e-bdf6795c8cf9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-320efee7-2tctv\" (UID: \"b3c58e20-4219-4650-af7e-bdf6795c8cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-320efee7-2tctv" Oct 30 20:04:41.932183 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:41.932120 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqmpq\" (UniqueName: \"kubernetes.io/projected/b3c58e20-4219-4650-af7e-bdf6795c8cf9-kube-api-access-gqmpq\") pod \"maas-api-revoke-sub-320efee7-2tctv\" (UID: \"b3c58e20-4219-4650-af7e-bdf6795c8cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-320efee7-2tctv" Oct 30 20:04:41.967745 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:41.967722 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-320efee7-2tctv" Oct 30 20:04:42.272997 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:04:42.272961 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3c58e20_4219_4650_af7e_bdf6795c8cf9.slice/crio-f5d1d59dde020cd57fe5f7e6ef62cb278f5d66bc2af2bd93daf5c597451a8d56 WatchSource:0}: Error finding container f5d1d59dde020cd57fe5f7e6ef62cb278f5d66bc2af2bd93daf5c597451a8d56: Status 404 returned error can't find the container with id f5d1d59dde020cd57fe5f7e6ef62cb278f5d66bc2af2bd93daf5c597451a8d56 Oct 30 20:04:42.273145 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:42.273018 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-320efee7-2tctv"] Oct 30 20:04:42.287947 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:42.287872 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-320efee7-2tctv" event={"ID":"b3c58e20-4219-4650-af7e-bdf6795c8cf9","Type":"ContainerStarted","Data":"f5d1d59dde020cd57fe5f7e6ef62cb278f5d66bc2af2bd93daf5c597451a8d56"} Oct 30 20:04:43.297625 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:43.297571 5088 generic.go:356] "Generic (PLEG): container finished" podID="b3c58e20-4219-4650-af7e-bdf6795c8cf9" containerID="2874047ba764f4224752e90c08cf1915db184a73416b93b8fee618511f18df69" exitCode=0 Oct 30 20:04:43.298052 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:43.297643 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-320efee7-2tctv" event={"ID":"b3c58e20-4219-4650-af7e-bdf6795c8cf9","Type":"ContainerDied","Data":"2874047ba764f4224752e90c08cf1915db184a73416b93b8fee618511f18df69"} Oct 30 20:04:44.498635 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:44.497133 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-320efee7-2tctv" Oct 30 20:04:44.645774 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:44.645696 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3c58e20-4219-4650-af7e-bdf6795c8cf9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3c58e20-4219-4650-af7e-bdf6795c8cf9-maas-api-openshift-ca\") pod \"b3c58e20-4219-4650-af7e-bdf6795c8cf9\" (UID: \"b3c58e20-4219-4650-af7e-bdf6795c8cf9\") " Oct 30 20:04:44.645774 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:44.645759 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3c58e20-4219-4650-af7e-bdf6795c8cf9-kube-api-access-gqmpq\" (UniqueName: \"kubernetes.io/projected/b3c58e20-4219-4650-af7e-bdf6795c8cf9-kube-api-access-gqmpq\") pod \"b3c58e20-4219-4650-af7e-bdf6795c8cf9\" (UID: \"b3c58e20-4219-4650-af7e-bdf6795c8cf9\") " Oct 30 20:04:44.645969 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:44.645873 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3c58e20-4219-4650-af7e-bdf6795c8cf9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3c58e20-4219-4650-af7e-bdf6795c8cf9-maas-api-xks-ca\") pod \"b3c58e20-4219-4650-af7e-bdf6795c8cf9\" (UID: \"b3c58e20-4219-4650-af7e-bdf6795c8cf9\") " Oct 30 20:04:44.646089 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:44.646065 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3c58e20-4219-4650-af7e-bdf6795c8cf9-maas-api-openshift-ca" pod "b3c58e20-4219-4650-af7e-bdf6795c8cf9" (UID: "b3c58e20-4219-4650-af7e-bdf6795c8cf9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:44.646405 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:44.646385 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3c58e20-4219-4650-af7e-bdf6795c8cf9-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:04:44.647858 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:44.647832 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3c58e20-4219-4650-af7e-bdf6795c8cf9-maas-api-xks-ca" pod "b3c58e20-4219-4650-af7e-bdf6795c8cf9" (UID: "b3c58e20-4219-4650-af7e-bdf6795c8cf9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:04:44.647858 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:44.647843 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3c58e20-4219-4650-af7e-bdf6795c8cf9-kube-api-access-gqmpq" pod "b3c58e20-4219-4650-af7e-bdf6795c8cf9" (UID: "b3c58e20-4219-4650-af7e-bdf6795c8cf9"). InnerVolumeSpecName "kube-api-access-gqmpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:44.748101 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:44.747939 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3c58e20-4219-4650-af7e-bdf6795c8cf9-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:04:44.748101 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:44.747959 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqmpq\" (UniqueName: \"kubernetes.io/projected/b3c58e20-4219-4650-af7e-bdf6795c8cf9-kube-api-access-gqmpq\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:04:45.313430 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:45.313386 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-320efee7-2tctv" event={"ID":"b3c58e20-4219-4650-af7e-bdf6795c8cf9","Type":"ContainerDied","Data":"f5d1d59dde020cd57fe5f7e6ef62cb278f5d66bc2af2bd93daf5c597451a8d56"} Oct 30 20:04:45.313641 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:45.313445 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5d1d59dde020cd57fe5f7e6ef62cb278f5d66bc2af2bd93daf5c597451a8d56" Oct 30 20:04:45.313641 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:45.313402 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-320efee7-2tctv" Oct 30 20:04:48.297636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:48.296953 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:04:48.577694 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:48.576800 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 20:04:48.743669 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:48.740707 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:04:48.884705 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:48.882820 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:04:48.984637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:48.981771 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:04:49.067662 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:49.056707 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:04:49.574662 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:49.503405 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 20:04:49.574662 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:49.518402 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" Oct 30 20:04:49.574662 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:49.537094 5088 patch_prober.go:29] interesting pod/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:04:49.574662 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:49.537159 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" podUID="cf9a8162-01f0-4226-b8f4-783125c277a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:04:49.627717 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:49.600121 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 20:04:49.627717 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:49.600489 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:04:49.681199 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:49.681164 5088 patch_prober.go:29] interesting pod/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:04:49.681474 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:49.681444 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" podUID="cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:04:49.895648 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:49.881194 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:04:49.896004 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:49.895952 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" podUID="419aecf7-a250-42ad-995b-e7ebd9630a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:04:49.896534 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:49.896502 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:04:49.917798 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:49.917753 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:04:49.922661 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:49.918182 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:04:50.046665 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:50.046603 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 30 20:04:50.079815 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:50.079763 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:04:50.090094 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:50.090063 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 20:04:50.114638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:50.090361 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:04:50.438938 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:50.438885 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:04:50.439240 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:50.439194 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" podUID="b55ec6e2-9392-4d8d-8340-ef12f674970c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:04:50.536838 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:50.536777 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:04:50.642893 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:50.642849 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 20:04:51.065237 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:51.065191 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wtlm7" podUID="419ffdb0-c223-4426-872a-50e3073812e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded" Oct 30 20:04:51.066093 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:51.065929 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 30 20:04:51.136191 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:51.136142 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:04:51.228642 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:51.221719 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:04:51.548718 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:51.543971 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 20:04:51.650357 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:51.650316 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:04:51.682002 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:51.681965 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:04:51.685176 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:51.685152 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:04:51.818325 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:51.818241 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" Oct 30 20:04:51.826835 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:51.820846 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 20:04:51.826835 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:51.820928 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:04:51.835352 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:51.835326 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-381315a4-w9c7h"] Oct 30 20:04:51.836507 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:51.836483 5088 cpu_manager.go:414] "containerMap: removing container" podUID="b3c58e20-4219-4650-af7e-bdf6795c8cf9" containerName="revoke-keys" Oct 30 20:04:51.836657 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:51.836643 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3c58e20-4219-4650-af7e-bdf6795c8cf9" containerName="revoke-keys" Oct 30 20:04:51.837596 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:51.837562 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3c58e20-4219-4650-af7e-bdf6795c8cf9" containerName="revoke-keys" Oct 30 20:04:51.843084 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:51.843065 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-381315a4-w9c7h" Oct 30 20:04:51.860451 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:51.860417 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-381315a4-w9c7h"] Oct 30 20:04:51.878483 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:51.877429 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:04:51.878483 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:51.877569 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:04:51.956819 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:51.956786 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzd2n\" (UniqueName: \"kubernetes.io/projected/ded4b273-9ae0-4ddf-bc70-154f4a67a31c-kube-api-access-rzd2n\") pod \"maas-api-revoke-sub-381315a4-w9c7h\" (UID: \"ded4b273-9ae0-4ddf-bc70-154f4a67a31c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-381315a4-w9c7h" Oct 30 20:04:51.957296 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:51.957273 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ded4b273-9ae0-4ddf-bc70-154f4a67a31c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-381315a4-w9c7h\" (UID: \"ded4b273-9ae0-4ddf-bc70-154f4a67a31c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-381315a4-w9c7h" Oct 30 20:04:51.957788 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:51.957763 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ded4b273-9ae0-4ddf-bc70-154f4a67a31c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-381315a4-w9c7h\" (UID: \"ded4b273-9ae0-4ddf-bc70-154f4a67a31c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-381315a4-w9c7h" Oct 30 20:04:52.059078 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:52.059042 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ded4b273-9ae0-4ddf-bc70-154f4a67a31c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-381315a4-w9c7h\" (UID: \"ded4b273-9ae0-4ddf-bc70-154f4a67a31c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-381315a4-w9c7h" Oct 30 20:04:52.059278 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:52.059161 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzd2n\" (UniqueName: \"kubernetes.io/projected/ded4b273-9ae0-4ddf-bc70-154f4a67a31c-kube-api-access-rzd2n\") pod \"maas-api-revoke-sub-381315a4-w9c7h\" (UID: \"ded4b273-9ae0-4ddf-bc70-154f4a67a31c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-381315a4-w9c7h" Oct 30 20:04:52.059278 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:52.059213 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ded4b273-9ae0-4ddf-bc70-154f4a67a31c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-381315a4-w9c7h\" (UID: \"ded4b273-9ae0-4ddf-bc70-154f4a67a31c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-381315a4-w9c7h" Oct 30 20:04:52.060175 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:52.060151 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ded4b273-9ae0-4ddf-bc70-154f4a67a31c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-381315a4-w9c7h\" (UID: \"ded4b273-9ae0-4ddf-bc70-154f4a67a31c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-381315a4-w9c7h" Oct 30 20:04:52.062058 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:52.062030 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ded4b273-9ae0-4ddf-bc70-154f4a67a31c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-381315a4-w9c7h\" (UID: \"ded4b273-9ae0-4ddf-bc70-154f4a67a31c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-381315a4-w9c7h" Oct 30 20:04:52.085803 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:52.085735 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzd2n\" (UniqueName: \"kubernetes.io/projected/ded4b273-9ae0-4ddf-bc70-154f4a67a31c-kube-api-access-rzd2n\") pod \"maas-api-revoke-sub-381315a4-w9c7h\" (UID: \"ded4b273-9ae0-4ddf-bc70-154f4a67a31c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-381315a4-w9c7h" Oct 30 20:04:52.228463 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:52.228431 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-381315a4-w9c7h" Oct 30 20:04:52.439288 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:04:52.439204 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:04:52.535463 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:04:52.535436 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podded4b273_9ae0_4ddf_bc70_154f4a67a31c.slice/crio-300af7555811aeecb3bd2361083f4ad0aa613cef36c43ec69ac4b4cc782e0233 WatchSource:0}: Error finding container 300af7555811aeecb3bd2361083f4ad0aa613cef36c43ec69ac4b4cc782e0233: Status 404 returned error can't find the container with id 300af7555811aeecb3bd2361083f4ad0aa613cef36c43ec69ac4b4cc782e0233 Oct 30 20:04:52.535724 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:52.535699 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-381315a4-w9c7h"] Oct 30 20:04:53.137780 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:53.137740 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-381315a4-w9c7h" event={"ID":"ded4b273-9ae0-4ddf-bc70-154f4a67a31c","Type":"ContainerStarted","Data":"0b117f14dd7c3c281855b300156ecb9442c3c45994511bf02127cb8204f079f0"} Oct 30 20:04:53.138193 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:53.137803 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-381315a4-w9c7h" event={"ID":"ded4b273-9ae0-4ddf-bc70-154f4a67a31c","Type":"ContainerStarted","Data":"300af7555811aeecb3bd2361083f4ad0aa613cef36c43ec69ac4b4cc782e0233"} Oct 30 20:04:53.186109 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:53.186052 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-381315a4-w9c7h" podStartSLOduration=2.186039455 podStartE2EDuration="2.186039455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:53.18178411 +0000 UTC m=+1054.594497908" watchObservedRunningTime="2026-10-30 20:04:53.186039455 +0000 UTC m=+1054.598753234" Oct 30 20:04:54.146505 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:54.146471 5088 generic.go:356] "Generic (PLEG): container finished" podID="ded4b273-9ae0-4ddf-bc70-154f4a67a31c" containerID="0b117f14dd7c3c281855b300156ecb9442c3c45994511bf02127cb8204f079f0" exitCode=0 Oct 30 20:04:54.146994 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:54.146561 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-381315a4-w9c7h" event={"ID":"ded4b273-9ae0-4ddf-bc70-154f4a67a31c","Type":"ContainerDied","Data":"0b117f14dd7c3c281855b300156ecb9442c3c45994511bf02127cb8204f079f0"} Oct 30 20:04:55.306067 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.306043 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-381315a4-w9c7h" Oct 30 20:04:55.413300 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.413201 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ded4b273-9ae0-4ddf-bc70-154f4a67a31c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ded4b273-9ae0-4ddf-bc70-154f4a67a31c-maas-api-openshift-ca\") pod \"ded4b273-9ae0-4ddf-bc70-154f4a67a31c\" (UID: \"ded4b273-9ae0-4ddf-bc70-154f4a67a31c\") " Oct 30 20:04:55.413440 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.413378 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ded4b273-9ae0-4ddf-bc70-154f4a67a31c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ded4b273-9ae0-4ddf-bc70-154f4a67a31c-maas-api-xks-ca\") pod \"ded4b273-9ae0-4ddf-bc70-154f4a67a31c\" (UID: \"ded4b273-9ae0-4ddf-bc70-154f4a67a31c\") " Oct 30 20:04:55.413514 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.413495 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ded4b273-9ae0-4ddf-bc70-154f4a67a31c-kube-api-access-rzd2n\" (UniqueName: \"kubernetes.io/projected/ded4b273-9ae0-4ddf-bc70-154f4a67a31c-kube-api-access-rzd2n\") pod \"ded4b273-9ae0-4ddf-bc70-154f4a67a31c\" (UID: \"ded4b273-9ae0-4ddf-bc70-154f4a67a31c\") " Oct 30 20:04:55.413580 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.413546 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ded4b273-9ae0-4ddf-bc70-154f4a67a31c-maas-api-openshift-ca" pod "ded4b273-9ae0-4ddf-bc70-154f4a67a31c" (UID: "ded4b273-9ae0-4ddf-bc70-154f4a67a31c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:55.413895 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.413879 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ded4b273-9ae0-4ddf-bc70-154f4a67a31c-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:04:55.415479 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.415459 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ded4b273-9ae0-4ddf-bc70-154f4a67a31c-maas-api-xks-ca" pod "ded4b273-9ae0-4ddf-bc70-154f4a67a31c" (UID: "ded4b273-9ae0-4ddf-bc70-154f4a67a31c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:04:55.415656 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.415606 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ded4b273-9ae0-4ddf-bc70-154f4a67a31c-kube-api-access-rzd2n" pod "ded4b273-9ae0-4ddf-bc70-154f4a67a31c" (UID: "ded4b273-9ae0-4ddf-bc70-154f4a67a31c"). InnerVolumeSpecName "kube-api-access-rzd2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:55.515534 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.515495 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ded4b273-9ae0-4ddf-bc70-154f4a67a31c-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:04:55.515534 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.515526 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzd2n\" (UniqueName: \"kubernetes.io/projected/ded4b273-9ae0-4ddf-bc70-154f4a67a31c-kube-api-access-rzd2n\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:04:55.633635 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.619702 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m"] Oct 30 20:04:55.633635 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.630582 5088 cpu_manager.go:414] "containerMap: removing container" podUID="ded4b273-9ae0-4ddf-bc70-154f4a67a31c" containerName="revoke-keys" Oct 30 20:04:55.633635 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.630633 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ded4b273-9ae0-4ddf-bc70-154f4a67a31c" containerName="revoke-keys" Oct 30 20:04:55.633635 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.630818 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="ded4b273-9ae0-4ddf-bc70-154f4a67a31c" containerName="revoke-keys" Oct 30 20:04:55.680604 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.680528 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" Oct 30 20:04:55.683917 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.683664 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk"] Oct 30 20:04:55.688421 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.688401 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" Oct 30 20:04:55.689410 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.689389 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk"] Oct 30 20:04:55.693754 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.693573 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m"] Oct 30 20:04:55.820445 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.820409 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b56fca64-1150-4198-945e-7e830300c11d-istio-token\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m\" (UID: \"b56fca64-1150-4198-945e-7e830300c11d\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" Oct 30 20:04:55.820604 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.820542 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b36276d-bc5b-4415-8cf9-461aee572f3a-istiod-ca-cert\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk\" (UID: \"0b36276d-bc5b-4415-8cf9-461aee572f3a\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" Oct 30 20:04:55.820703 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.820603 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b36276d-bc5b-4415-8cf9-461aee572f3a-istio-envoy\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk\" (UID: \"0b36276d-bc5b-4415-8cf9-461aee572f3a\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" Oct 30 20:04:55.820703 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.820643 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmqdl\" (UniqueName: \"kubernetes.io/projected/b56fca64-1150-4198-945e-7e830300c11d-kube-api-access-wmqdl\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m\" (UID: \"b56fca64-1150-4198-945e-7e830300c11d\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" Oct 30 20:04:55.820703 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.820668 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b36276d-bc5b-4415-8cf9-461aee572f3a-istio-podinfo\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk\" (UID: \"0b36276d-bc5b-4415-8cf9-461aee572f3a\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" Oct 30 20:04:55.820910 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.820890 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56fca64-1150-4198-945e-7e830300c11d-workload-certs\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m\" (UID: \"b56fca64-1150-4198-945e-7e830300c11d\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" Oct 30 20:04:55.820978 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.820959 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b36276d-bc5b-4415-8cf9-461aee572f3a-workload-socket\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk\" (UID: \"0b36276d-bc5b-4415-8cf9-461aee572f3a\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" Oct 30 20:04:55.821024 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.821010 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b36276d-bc5b-4415-8cf9-461aee572f3a-istio-data\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk\" (UID: \"0b36276d-bc5b-4415-8cf9-461aee572f3a\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" Oct 30 20:04:55.821070 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.821043 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56fca64-1150-4198-945e-7e830300c11d-credential-socket\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m\" (UID: \"b56fca64-1150-4198-945e-7e830300c11d\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" Oct 30 20:04:55.821145 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.821129 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56fca64-1150-4198-945e-7e830300c11d-istio-envoy\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m\" (UID: \"b56fca64-1150-4198-945e-7e830300c11d\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" Oct 30 20:04:55.821181 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.821152 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56fca64-1150-4198-945e-7e830300c11d-istio-podinfo\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m\" (UID: \"b56fca64-1150-4198-945e-7e830300c11d\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" Oct 30 20:04:55.821181 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.821174 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56fca64-1150-4198-945e-7e830300c11d-workload-socket\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m\" (UID: \"b56fca64-1150-4198-945e-7e830300c11d\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" Oct 30 20:04:55.821288 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.821273 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8csc\" (UniqueName: \"kubernetes.io/projected/0b36276d-bc5b-4415-8cf9-461aee572f3a-kube-api-access-d8csc\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk\" (UID: \"0b36276d-bc5b-4415-8cf9-461aee572f3a\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" Oct 30 20:04:55.821354 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.821342 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56fca64-1150-4198-945e-7e830300c11d-istiod-ca-cert\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m\" (UID: \"b56fca64-1150-4198-945e-7e830300c11d\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" Oct 30 20:04:55.821396 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.821372 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b36276d-bc5b-4415-8cf9-461aee572f3a-credential-socket\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk\" (UID: \"0b36276d-bc5b-4415-8cf9-461aee572f3a\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" Oct 30 20:04:55.821523 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.821506 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b36276d-bc5b-4415-8cf9-461aee572f3a-workload-certs\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk\" (UID: \"0b36276d-bc5b-4415-8cf9-461aee572f3a\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" Oct 30 20:04:55.821601 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.821529 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b36276d-bc5b-4415-8cf9-461aee572f3a-istio-token\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk\" (UID: \"0b36276d-bc5b-4415-8cf9-461aee572f3a\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" Oct 30 20:04:55.821707 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.821668 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56fca64-1150-4198-945e-7e830300c11d-istio-data\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m\" (UID: \"b56fca64-1150-4198-945e-7e830300c11d\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" Oct 30 20:04:55.922884 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.922843 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56fca64-1150-4198-945e-7e830300c11d-istio-data\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m\" (UID: \"b56fca64-1150-4198-945e-7e830300c11d\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" Oct 30 20:04:55.922884 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.922892 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b56fca64-1150-4198-945e-7e830300c11d-istio-token\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m\" (UID: \"b56fca64-1150-4198-945e-7e830300c11d\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" Oct 30 20:04:55.923116 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.922959 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b36276d-bc5b-4415-8cf9-461aee572f3a-istiod-ca-cert\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk\" (UID: \"0b36276d-bc5b-4415-8cf9-461aee572f3a\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" Oct 30 20:04:55.923116 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.922994 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b36276d-bc5b-4415-8cf9-461aee572f3a-istio-envoy\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk\" (UID: \"0b36276d-bc5b-4415-8cf9-461aee572f3a\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" Oct 30 20:04:55.923116 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.923013 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmqdl\" (UniqueName: \"kubernetes.io/projected/b56fca64-1150-4198-945e-7e830300c11d-kube-api-access-wmqdl\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m\" (UID: \"b56fca64-1150-4198-945e-7e830300c11d\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" Oct 30 20:04:55.923116 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.923031 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b36276d-bc5b-4415-8cf9-461aee572f3a-istio-podinfo\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk\" (UID: \"0b36276d-bc5b-4415-8cf9-461aee572f3a\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" Oct 30 20:04:55.923116 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.923066 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56fca64-1150-4198-945e-7e830300c11d-workload-certs\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m\" (UID: \"b56fca64-1150-4198-945e-7e830300c11d\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" Oct 30 20:04:55.923116 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.923082 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b36276d-bc5b-4415-8cf9-461aee572f3a-workload-socket\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk\" (UID: \"0b36276d-bc5b-4415-8cf9-461aee572f3a\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" Oct 30 20:04:55.923463 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.923127 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b36276d-bc5b-4415-8cf9-461aee572f3a-istio-data\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk\" (UID: \"0b36276d-bc5b-4415-8cf9-461aee572f3a\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" Oct 30 20:04:55.923463 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.923169 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56fca64-1150-4198-945e-7e830300c11d-credential-socket\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m\" (UID: \"b56fca64-1150-4198-945e-7e830300c11d\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" Oct 30 20:04:55.923463 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.923226 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56fca64-1150-4198-945e-7e830300c11d-istio-envoy\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m\" (UID: \"b56fca64-1150-4198-945e-7e830300c11d\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" Oct 30 20:04:55.923463 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.923251 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56fca64-1150-4198-945e-7e830300c11d-istio-podinfo\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m\" (UID: \"b56fca64-1150-4198-945e-7e830300c11d\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" Oct 30 20:04:55.923463 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.923286 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56fca64-1150-4198-945e-7e830300c11d-workload-socket\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m\" (UID: \"b56fca64-1150-4198-945e-7e830300c11d\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" Oct 30 20:04:55.923463 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.923330 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8csc\" (UniqueName: \"kubernetes.io/projected/0b36276d-bc5b-4415-8cf9-461aee572f3a-kube-api-access-d8csc\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk\" (UID: \"0b36276d-bc5b-4415-8cf9-461aee572f3a\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" Oct 30 20:04:55.923463 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.923362 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56fca64-1150-4198-945e-7e830300c11d-istio-data\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m\" (UID: \"b56fca64-1150-4198-945e-7e830300c11d\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" Oct 30 20:04:55.923463 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.923407 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56fca64-1150-4198-945e-7e830300c11d-workload-certs\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m\" (UID: \"b56fca64-1150-4198-945e-7e830300c11d\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" Oct 30 20:04:55.923463 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.923434 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56fca64-1150-4198-945e-7e830300c11d-istiod-ca-cert\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m\" (UID: \"b56fca64-1150-4198-945e-7e830300c11d\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" Oct 30 20:04:55.923993 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.923480 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b36276d-bc5b-4415-8cf9-461aee572f3a-credential-socket\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk\" (UID: \"0b36276d-bc5b-4415-8cf9-461aee572f3a\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" Oct 30 20:04:55.923993 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.923526 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b36276d-bc5b-4415-8cf9-461aee572f3a-workload-certs\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk\" (UID: \"0b36276d-bc5b-4415-8cf9-461aee572f3a\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" Oct 30 20:04:55.923993 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.923542 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b36276d-bc5b-4415-8cf9-461aee572f3a-workload-socket\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk\" (UID: \"0b36276d-bc5b-4415-8cf9-461aee572f3a\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" Oct 30 20:04:55.923993 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.923552 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b36276d-bc5b-4415-8cf9-461aee572f3a-istio-token\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk\" (UID: \"0b36276d-bc5b-4415-8cf9-461aee572f3a\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" Oct 30 20:04:55.923993 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.923571 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56fca64-1150-4198-945e-7e830300c11d-credential-socket\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m\" (UID: \"b56fca64-1150-4198-945e-7e830300c11d\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" Oct 30 20:04:55.923993 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.923759 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b36276d-bc5b-4415-8cf9-461aee572f3a-istio-data\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk\" (UID: \"0b36276d-bc5b-4415-8cf9-461aee572f3a\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" Oct 30 20:04:55.923993 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.923863 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b36276d-bc5b-4415-8cf9-461aee572f3a-credential-socket\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk\" (UID: \"0b36276d-bc5b-4415-8cf9-461aee572f3a\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" Oct 30 20:04:55.924359 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.924148 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b36276d-bc5b-4415-8cf9-461aee572f3a-istiod-ca-cert\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk\" (UID: \"0b36276d-bc5b-4415-8cf9-461aee572f3a\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" Oct 30 20:04:55.924359 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.924207 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b36276d-bc5b-4415-8cf9-461aee572f3a-workload-certs\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk\" (UID: \"0b36276d-bc5b-4415-8cf9-461aee572f3a\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" Oct 30 20:04:55.924359 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.924256 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56fca64-1150-4198-945e-7e830300c11d-workload-socket\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m\" (UID: \"b56fca64-1150-4198-945e-7e830300c11d\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" Oct 30 20:04:55.924359 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.924350 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56fca64-1150-4198-945e-7e830300c11d-istiod-ca-cert\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m\" (UID: \"b56fca64-1150-4198-945e-7e830300c11d\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" Oct 30 20:04:55.925667 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.925640 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b36276d-bc5b-4415-8cf9-461aee572f3a-istio-envoy\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk\" (UID: \"0b36276d-bc5b-4415-8cf9-461aee572f3a\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" Oct 30 20:04:55.925941 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.925917 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b36276d-bc5b-4415-8cf9-461aee572f3a-istio-podinfo\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk\" (UID: \"0b36276d-bc5b-4415-8cf9-461aee572f3a\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" Oct 30 20:04:55.926217 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.926200 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56fca64-1150-4198-945e-7e830300c11d-istio-envoy\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m\" (UID: \"b56fca64-1150-4198-945e-7e830300c11d\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" Oct 30 20:04:55.926326 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.926305 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56fca64-1150-4198-945e-7e830300c11d-istio-podinfo\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m\" (UID: \"b56fca64-1150-4198-945e-7e830300c11d\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" Oct 30 20:04:55.950344 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.950283 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b36276d-bc5b-4415-8cf9-461aee572f3a-istio-token\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk\" (UID: \"0b36276d-bc5b-4415-8cf9-461aee572f3a\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" Oct 30 20:04:55.950344 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.950292 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b56fca64-1150-4198-945e-7e830300c11d-istio-token\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m\" (UID: \"b56fca64-1150-4198-945e-7e830300c11d\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" Oct 30 20:04:55.950344 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.950298 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmqdl\" (UniqueName: \"kubernetes.io/projected/b56fca64-1150-4198-945e-7e830300c11d-kube-api-access-wmqdl\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m\" (UID: \"b56fca64-1150-4198-945e-7e830300c11d\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" Oct 30 20:04:55.950510 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:55.950388 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8csc\" (UniqueName: \"kubernetes.io/projected/0b36276d-bc5b-4415-8cf9-461aee572f3a-kube-api-access-d8csc\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk\" (UID: \"0b36276d-bc5b-4415-8cf9-461aee572f3a\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" Oct 30 20:04:56.004874 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:56.004846 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" Oct 30 20:04:56.012216 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:56.012194 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" Oct 30 20:04:56.165645 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:56.165571 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-381315a4-w9c7h" event={"ID":"ded4b273-9ae0-4ddf-bc70-154f4a67a31c","Type":"ContainerDied","Data":"300af7555811aeecb3bd2361083f4ad0aa613cef36c43ec69ac4b4cc782e0233"} Oct 30 20:04:56.165805 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:56.165655 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="300af7555811aeecb3bd2361083f4ad0aa613cef36c43ec69ac4b4cc782e0233" Oct 30 20:04:56.165805 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:56.165710 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-381315a4-w9c7h" Oct 30 20:04:56.336427 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:56.336354 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m"] Oct 30 20:04:56.336762 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:04:56.336467 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb56fca64_1150_4198_945e_7e830300c11d.slice/crio-241e1db8b27b2fc77a301bbac6d896975f8f1324b718a69b6736dcea185f8e19 WatchSource:0}: Error finding container 241e1db8b27b2fc77a301bbac6d896975f8f1324b718a69b6736dcea185f8e19: Status 404 returned error can't find the container with id 241e1db8b27b2fc77a301bbac6d896975f8f1324b718a69b6736dcea185f8e19 Oct 30 20:04:56.339041 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:56.339011 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:04:56.339107 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:56.339075 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:04:56.339144 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:56.339106 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:04:56.352671 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:04:56.352636 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b36276d_bc5b_4415_8cf9_461aee572f3a.slice/crio-e9b6e0bfaa6bd0ba4363efce496ebd01f6506c5a6c2b8f1a1de9f5b36106c990 WatchSource:0}: Error finding container e9b6e0bfaa6bd0ba4363efce496ebd01f6506c5a6c2b8f1a1de9f5b36106c990: Status 404 returned error can't find the container with id e9b6e0bfaa6bd0ba4363efce496ebd01f6506c5a6c2b8f1a1de9f5b36106c990 Oct 30 20:04:56.353508 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:56.353484 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk"] Oct 30 20:04:56.355081 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:56.355051 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:04:56.355180 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:56.355110 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:04:56.355180 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:56.355137 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:04:57.175182 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:57.175140 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" event={"ID":"0b36276d-bc5b-4415-8cf9-461aee572f3a","Type":"ContainerStarted","Data":"d53c6f364c7241cc108097bf75f911b385021a7733320a1ccf7a4a977d146d8d"} Oct 30 20:04:57.175376 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:57.175199 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" event={"ID":"0b36276d-bc5b-4415-8cf9-461aee572f3a","Type":"ContainerStarted","Data":"e9b6e0bfaa6bd0ba4363efce496ebd01f6506c5a6c2b8f1a1de9f5b36106c990"} Oct 30 20:04:57.177109 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:57.177074 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" event={"ID":"b56fca64-1150-4198-945e-7e830300c11d","Type":"ContainerStarted","Data":"9eaf99979ceb984cf41aa039dda144ebbde0304a942ce7350ed64c04c8537d33"} Oct 30 20:04:57.177210 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:57.177137 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" event={"ID":"b56fca64-1150-4198-945e-7e830300c11d","Type":"ContainerStarted","Data":"241e1db8b27b2fc77a301bbac6d896975f8f1324b718a69b6736dcea185f8e19"} Oct 30 20:04:57.271871 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:57.271813 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" podStartSLOduration=2.2717999 podStartE2EDuration="2.2717999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:57.257259401 +0000 UTC m=+1058.669973201" watchObservedRunningTime="2026-10-30 20:04:57.2717999 +0000 UTC m=+1058.684513673" Oct 30 20:04:57.339396 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:57.339329 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" podStartSLOduration=2.339309131 podStartE2EDuration="2.339309131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:57.331013473 +0000 UTC m=+1058.743727254" watchObservedRunningTime="2026-10-30 20:04:57.339309131 +0000 UTC m=+1058.752022910" Oct 30 20:04:57.684637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:57.663507 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b"] Oct 30 20:04:57.749643 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:57.747873 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b"] Oct 30 20:04:57.749643 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:57.748722 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" Oct 30 20:04:57.771238 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:57.769857 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-51dd62\"/\"distinct-w6-51dd62-kserve-self-signed-certs\"" Oct 30 20:04:57.897656 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:57.894627 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/659701e4-4e82-4c4a-b651-4ad8641f7b65-tls-certs\") pod \"distinct-w6-51dd62-kserve-54bffd8b69-v8k2b\" (UID: \"659701e4-4e82-4c4a-b651-4ad8641f7b65\") " pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" Oct 30 20:04:57.897656 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:57.894820 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk9pt\" (UniqueName: \"kubernetes.io/projected/659701e4-4e82-4c4a-b651-4ad8641f7b65-kube-api-access-mk9pt\") pod \"distinct-w6-51dd62-kserve-54bffd8b69-v8k2b\" (UID: \"659701e4-4e82-4c4a-b651-4ad8641f7b65\") " pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" Oct 30 20:04:57.897656 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:57.894914 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/659701e4-4e82-4c4a-b651-4ad8641f7b65-model-cache\") pod \"distinct-w6-51dd62-kserve-54bffd8b69-v8k2b\" (UID: \"659701e4-4e82-4c4a-b651-4ad8641f7b65\") " pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" Oct 30 20:04:57.897656 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:57.894953 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/659701e4-4e82-4c4a-b651-4ad8641f7b65-dshm\") pod \"distinct-w6-51dd62-kserve-54bffd8b69-v8k2b\" (UID: \"659701e4-4e82-4c4a-b651-4ad8641f7b65\") " pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" Oct 30 20:04:57.897656 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:57.895049 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/659701e4-4e82-4c4a-b651-4ad8641f7b65-home\") pod \"distinct-w6-51dd62-kserve-54bffd8b69-v8k2b\" (UID: \"659701e4-4e82-4c4a-b651-4ad8641f7b65\") " pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" Oct 30 20:04:57.897656 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:57.895158 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/659701e4-4e82-4c4a-b651-4ad8641f7b65-tmp-dir\") pod \"distinct-w6-51dd62-kserve-54bffd8b69-v8k2b\" (UID: \"659701e4-4e82-4c4a-b651-4ad8641f7b65\") " pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" Oct 30 20:04:58.002179 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:58.002143 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/659701e4-4e82-4c4a-b651-4ad8641f7b65-tls-certs\") pod \"distinct-w6-51dd62-kserve-54bffd8b69-v8k2b\" (UID: \"659701e4-4e82-4c4a-b651-4ad8641f7b65\") " pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" Oct 30 20:04:58.002570 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:58.002528 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk9pt\" (UniqueName: \"kubernetes.io/projected/659701e4-4e82-4c4a-b651-4ad8641f7b65-kube-api-access-mk9pt\") pod \"distinct-w6-51dd62-kserve-54bffd8b69-v8k2b\" (UID: \"659701e4-4e82-4c4a-b651-4ad8641f7b65\") " pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" Oct 30 20:04:58.006931 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:58.006903 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/659701e4-4e82-4c4a-b651-4ad8641f7b65-model-cache\") pod \"distinct-w6-51dd62-kserve-54bffd8b69-v8k2b\" (UID: \"659701e4-4e82-4c4a-b651-4ad8641f7b65\") " pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" Oct 30 20:04:58.007198 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:58.007178 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/659701e4-4e82-4c4a-b651-4ad8641f7b65-dshm\") pod \"distinct-w6-51dd62-kserve-54bffd8b69-v8k2b\" (UID: \"659701e4-4e82-4c4a-b651-4ad8641f7b65\") " pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" Oct 30 20:04:58.007405 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:58.007389 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/659701e4-4e82-4c4a-b651-4ad8641f7b65-home\") pod \"distinct-w6-51dd62-kserve-54bffd8b69-v8k2b\" (UID: \"659701e4-4e82-4c4a-b651-4ad8641f7b65\") " pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" Oct 30 20:04:58.007731 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:58.007710 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/659701e4-4e82-4c4a-b651-4ad8641f7b65-tmp-dir\") pod \"distinct-w6-51dd62-kserve-54bffd8b69-v8k2b\" (UID: \"659701e4-4e82-4c4a-b651-4ad8641f7b65\") " pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" Oct 30 20:04:58.019896 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:58.019847 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/659701e4-4e82-4c4a-b651-4ad8641f7b65-model-cache\") pod \"distinct-w6-51dd62-kserve-54bffd8b69-v8k2b\" (UID: \"659701e4-4e82-4c4a-b651-4ad8641f7b65\") " pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" Oct 30 20:04:58.027903 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:58.022602 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/659701e4-4e82-4c4a-b651-4ad8641f7b65-tmp-dir\") pod \"distinct-w6-51dd62-kserve-54bffd8b69-v8k2b\" (UID: \"659701e4-4e82-4c4a-b651-4ad8641f7b65\") " pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" Oct 30 20:04:58.027903 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:58.022964 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/659701e4-4e82-4c4a-b651-4ad8641f7b65-home\") pod \"distinct-w6-51dd62-kserve-54bffd8b69-v8k2b\" (UID: \"659701e4-4e82-4c4a-b651-4ad8641f7b65\") " pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" Oct 30 20:04:58.027903 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:58.024943 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" Oct 30 20:04:58.027903 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:58.025066 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 20:04:58.027903 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:58.025110 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" podUID="b56fca64-1150-4198-945e-7e830300c11d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 20:04:58.027903 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:58.025827 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" Oct 30 20:04:58.027903 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:58.027201 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 20:04:58.027903 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:58.027246 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" podUID="0b36276d-bc5b-4415-8cf9-461aee572f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 20:04:58.080647 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:58.065542 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk9pt\" (UniqueName: \"kubernetes.io/projected/659701e4-4e82-4c4a-b651-4ad8641f7b65-kube-api-access-mk9pt\") pod \"distinct-w6-51dd62-kserve-54bffd8b69-v8k2b\" (UID: \"659701e4-4e82-4c4a-b651-4ad8641f7b65\") " pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" Oct 30 20:04:58.094654 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:58.094602 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/659701e4-4e82-4c4a-b651-4ad8641f7b65-tls-certs\") pod \"distinct-w6-51dd62-kserve-54bffd8b69-v8k2b\" (UID: \"659701e4-4e82-4c4a-b651-4ad8641f7b65\") " pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" Oct 30 20:04:58.105025 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:58.104976 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/659701e4-4e82-4c4a-b651-4ad8641f7b65-dshm\") pod \"distinct-w6-51dd62-kserve-54bffd8b69-v8k2b\" (UID: \"659701e4-4e82-4c4a-b651-4ad8641f7b65\") " pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" Oct 30 20:04:58.368310 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:58.368211 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" Oct 30 20:04:58.741363 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:04:58.741288 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod659701e4_4e82_4c4a_b651_4ad8641f7b65.slice/crio-3989bee64b116e24e78201adce2b4155d02e5ae841f0e326c4b0cc5f0ca01556 WatchSource:0}: Error finding container 3989bee64b116e24e78201adce2b4155d02e5ae841f0e326c4b0cc5f0ca01556: Status 404 returned error can't find the container with id 3989bee64b116e24e78201adce2b4155d02e5ae841f0e326c4b0cc5f0ca01556 Oct 30 20:04:58.746211 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:58.746191 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b"] Oct 30 20:04:59.019701 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:59.019486 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 20:04:59.019701 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:59.019552 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" podUID="0b36276d-bc5b-4415-8cf9-461aee572f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 20:04:59.039386 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:59.038728 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 20:04:59.039386 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:59.038785 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" podUID="b56fca64-1150-4198-945e-7e830300c11d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 20:04:59.233731 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:59.233681 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" event={"ID":"659701e4-4e82-4c4a-b651-4ad8641f7b65","Type":"ContainerStarted","Data":"e41fb20108e4fc754dc7d7e0a0263fd9ea78afdba73dee850bce2bc8b933fe02"} Oct 30 20:04:59.234104 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:59.234083 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" event={"ID":"659701e4-4e82-4c4a-b651-4ad8641f7b65","Type":"ContainerStarted","Data":"3989bee64b116e24e78201adce2b4155d02e5ae841f0e326c4b0cc5f0ca01556"} Oct 30 20:04:59.281946 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:59.281848 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podStartSLOduration=2.281833228 podStartE2EDuration="2.281833228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:59.280801482 +0000 UTC m=+1060.693515261" watchObservedRunningTime="2026-10-30 20:04:59.281833228 +0000 UTC m=+1060.694547007" Oct 30 20:04:59.368925 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:04:59.368890 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" Oct 30 20:05:00.010103 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:00.010065 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 20:05:00.010389 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:00.010358 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" podUID="b56fca64-1150-4198-945e-7e830300c11d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 20:05:00.014740 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:00.014716 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 20:05:00.016381 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:00.016350 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" podUID="0b36276d-bc5b-4415-8cf9-461aee572f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 20:05:01.007634 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:01.007063 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 20:05:01.007634 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:01.007129 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" podUID="b56fca64-1150-4198-945e-7e830300c11d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 20:05:01.030638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:01.019952 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 20:05:01.030638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:01.020010 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" podUID="0b36276d-bc5b-4415-8cf9-461aee572f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 20:05:01.431690 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:01.425058 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6035ee0c-wphdw"] Oct 30 20:05:01.432064 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:01.432032 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035ee0c-wphdw" Oct 30 20:05:01.439012 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:01.438983 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6035ee0c-wphdw"] Oct 30 20:05:01.446308 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:01.444125 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb4bf3de-ea87-4425-80d2-e76a3354d23d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6035ee0c-wphdw\" (UID: \"fb4bf3de-ea87-4425-80d2-e76a3354d23d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035ee0c-wphdw" Oct 30 20:05:01.446308 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:01.444209 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb4bf3de-ea87-4425-80d2-e76a3354d23d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6035ee0c-wphdw\" (UID: \"fb4bf3de-ea87-4425-80d2-e76a3354d23d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035ee0c-wphdw" Oct 30 20:05:01.446308 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:01.444342 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njmnv\" (UniqueName: \"kubernetes.io/projected/fb4bf3de-ea87-4425-80d2-e76a3354d23d-kube-api-access-njmnv\") pod \"maas-api-revoke-sub-6035ee0c-wphdw\" (UID: \"fb4bf3de-ea87-4425-80d2-e76a3354d23d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035ee0c-wphdw" Oct 30 20:05:01.453742 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:01.453719 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:05:01.453988 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:01.453971 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:05:01.546107 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:01.546073 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njmnv\" (UniqueName: \"kubernetes.io/projected/fb4bf3de-ea87-4425-80d2-e76a3354d23d-kube-api-access-njmnv\") pod \"maas-api-revoke-sub-6035ee0c-wphdw\" (UID: \"fb4bf3de-ea87-4425-80d2-e76a3354d23d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035ee0c-wphdw" Oct 30 20:05:01.546722 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:01.546702 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb4bf3de-ea87-4425-80d2-e76a3354d23d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6035ee0c-wphdw\" (UID: \"fb4bf3de-ea87-4425-80d2-e76a3354d23d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035ee0c-wphdw" Oct 30 20:05:01.547423 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:01.547409 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb4bf3de-ea87-4425-80d2-e76a3354d23d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6035ee0c-wphdw\" (UID: \"fb4bf3de-ea87-4425-80d2-e76a3354d23d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035ee0c-wphdw" Oct 30 20:05:01.547939 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:01.547338 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb4bf3de-ea87-4425-80d2-e76a3354d23d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6035ee0c-wphdw\" (UID: \"fb4bf3de-ea87-4425-80d2-e76a3354d23d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035ee0c-wphdw" Oct 30 20:05:01.551409 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:01.551385 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb4bf3de-ea87-4425-80d2-e76a3354d23d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6035ee0c-wphdw\" (UID: \"fb4bf3de-ea87-4425-80d2-e76a3354d23d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035ee0c-wphdw" Oct 30 20:05:01.640642 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:01.640076 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njmnv\" (UniqueName: \"kubernetes.io/projected/fb4bf3de-ea87-4425-80d2-e76a3354d23d-kube-api-access-njmnv\") pod \"maas-api-revoke-sub-6035ee0c-wphdw\" (UID: \"fb4bf3de-ea87-4425-80d2-e76a3354d23d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035ee0c-wphdw" Oct 30 20:05:01.756165 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:01.756140 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035ee0c-wphdw" Oct 30 20:05:02.041442 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:02.038777 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 20:05:02.041442 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:02.038845 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" podUID="b56fca64-1150-4198-945e-7e830300c11d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 20:05:02.041442 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:02.038778 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 20:05:02.041442 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:02.040721 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" podUID="0b36276d-bc5b-4415-8cf9-461aee572f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 20:05:02.140142 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:02.140108 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6035ee0c-wphdw"] Oct 30 20:05:02.257775 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:02.257741 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035ee0c-wphdw" event={"ID":"fb4bf3de-ea87-4425-80d2-e76a3354d23d","Type":"ContainerStarted","Data":"61873747f72c2889b7ef8a505604abc8b88acf6819781c506ac93b193a5cb086"} Oct 30 20:05:03.006370 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:03.006328 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 20:05:03.006547 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:03.006398 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" podUID="b56fca64-1150-4198-945e-7e830300c11d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 20:05:03.012919 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:03.012887 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 20:05:03.013056 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:03.012956 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" podUID="0b36276d-bc5b-4415-8cf9-461aee572f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 20:05:03.269806 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:03.269716 5088 generic.go:356] "Generic (PLEG): container finished" podID="fb4bf3de-ea87-4425-80d2-e76a3354d23d" containerID="7c0f447c7ea99fa8edd0712214712a55332134e468229d48911850e098aa43db" exitCode=0 Oct 30 20:05:03.270164 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:03.269811 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035ee0c-wphdw" event={"ID":"fb4bf3de-ea87-4425-80d2-e76a3354d23d","Type":"ContainerDied","Data":"7c0f447c7ea99fa8edd0712214712a55332134e468229d48911850e098aa43db"} Oct 30 20:05:04.152001 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:04.151968 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" Oct 30 20:05:04.156295 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:04.156273 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" Oct 30 20:05:04.158727 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:04.158710 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" Oct 30 20:05:04.279875 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:04.279846 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" Oct 30 20:05:04.281121 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:04.281081 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" Oct 30 20:05:04.282139 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:04.281658 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" Oct 30 20:05:04.482869 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:04.482847 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035ee0c-wphdw" Oct 30 20:05:04.520995 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:04.519816 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb4bf3de-ea87-4425-80d2-e76a3354d23d-kube-api-access-njmnv\" (UniqueName: \"kubernetes.io/projected/fb4bf3de-ea87-4425-80d2-e76a3354d23d-kube-api-access-njmnv\") pod \"fb4bf3de-ea87-4425-80d2-e76a3354d23d\" (UID: \"fb4bf3de-ea87-4425-80d2-e76a3354d23d\") " Oct 30 20:05:04.520995 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:04.519904 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb4bf3de-ea87-4425-80d2-e76a3354d23d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb4bf3de-ea87-4425-80d2-e76a3354d23d-maas-api-openshift-ca\") pod \"fb4bf3de-ea87-4425-80d2-e76a3354d23d\" (UID: \"fb4bf3de-ea87-4425-80d2-e76a3354d23d\") " Oct 30 20:05:04.520995 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:04.520033 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb4bf3de-ea87-4425-80d2-e76a3354d23d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb4bf3de-ea87-4425-80d2-e76a3354d23d-maas-api-xks-ca\") pod \"fb4bf3de-ea87-4425-80d2-e76a3354d23d\" (UID: \"fb4bf3de-ea87-4425-80d2-e76a3354d23d\") " Oct 30 20:05:04.520995 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:04.520844 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb4bf3de-ea87-4425-80d2-e76a3354d23d-maas-api-openshift-ca" pod "fb4bf3de-ea87-4425-80d2-e76a3354d23d" (UID: "fb4bf3de-ea87-4425-80d2-e76a3354d23d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:05:04.522559 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:04.522533 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb4bf3de-ea87-4425-80d2-e76a3354d23d-kube-api-access-njmnv" pod "fb4bf3de-ea87-4425-80d2-e76a3354d23d" (UID: "fb4bf3de-ea87-4425-80d2-e76a3354d23d"). InnerVolumeSpecName "kube-api-access-njmnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:05:04.523290 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:04.523273 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb4bf3de-ea87-4425-80d2-e76a3354d23d-maas-api-xks-ca" pod "fb4bf3de-ea87-4425-80d2-e76a3354d23d" (UID: "fb4bf3de-ea87-4425-80d2-e76a3354d23d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:05:04.622473 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:04.622442 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njmnv\" (UniqueName: \"kubernetes.io/projected/fb4bf3de-ea87-4425-80d2-e76a3354d23d-kube-api-access-njmnv\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:05:04.622473 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:04.622470 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb4bf3de-ea87-4425-80d2-e76a3354d23d-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:05:04.622473 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:04.622480 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb4bf3de-ea87-4425-80d2-e76a3354d23d-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:05:05.288178 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:05.288141 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035ee0c-wphdw" Oct 30 20:05:05.288696 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:05.288174 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035ee0c-wphdw" event={"ID":"fb4bf3de-ea87-4425-80d2-e76a3354d23d","Type":"ContainerDied","Data":"61873747f72c2889b7ef8a505604abc8b88acf6819781c506ac93b193a5cb086"} Oct 30 20:05:05.288696 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:05.288224 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61873747f72c2889b7ef8a505604abc8b88acf6819781c506ac93b193a5cb086" Oct 30 20:05:06.706013 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:06.705979 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:05:08.378177 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:08.374179 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" Oct 30 20:05:09.146644 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:09.145039 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:09.240796 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:09.240713 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:09.396637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:09.395735 5088 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": context deadline exceeded" Oct 30 20:05:09.658637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:09.653904 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:09.837639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:09.820723 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:09.930698 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:09.928722 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:10.020638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:10.012652 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq"] Oct 30 20:05:10.020638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:10.013599 5088 cpu_manager.go:414] "containerMap: removing container" podUID="fb4bf3de-ea87-4425-80d2-e76a3354d23d" containerName="revoke-keys" Oct 30 20:05:10.028636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:10.025443 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb4bf3de-ea87-4425-80d2-e76a3354d23d" containerName="revoke-keys" Oct 30 20:05:10.028636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:10.025801 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb4bf3de-ea87-4425-80d2-e76a3354d23d" containerName="revoke-keys" Oct 30 20:05:10.030529 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:10.030502 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" Oct 30 20:05:10.082427 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:10.080205 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:10.124311 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:10.124259 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a255c4a5-c090-49d2-84ba-536dfb9b8a69-istiod-ca-cert\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq\" (UID: \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" Oct 30 20:05:10.124736 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:10.124711 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a255c4a5-c090-49d2-84ba-536dfb9b8a69-workload-socket\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq\" (UID: \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" Oct 30 20:05:10.125181 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:10.125071 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a255c4a5-c090-49d2-84ba-536dfb9b8a69-istio-token\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq\" (UID: \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" Oct 30 20:05:10.125501 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:10.125484 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a255c4a5-c090-49d2-84ba-536dfb9b8a69-istio-envoy\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq\" (UID: \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" Oct 30 20:05:10.125801 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:10.125782 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a255c4a5-c090-49d2-84ba-536dfb9b8a69-istio-data\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq\" (UID: \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" Oct 30 20:05:10.156721 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:10.156675 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a255c4a5-c090-49d2-84ba-536dfb9b8a69-credential-socket\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq\" (UID: \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" Oct 30 20:05:10.157125 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:10.157087 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zkdz\" (UniqueName: \"kubernetes.io/projected/a255c4a5-c090-49d2-84ba-536dfb9b8a69-kube-api-access-4zkdz\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq\" (UID: \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" Oct 30 20:05:10.157333 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:10.157317 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a255c4a5-c090-49d2-84ba-536dfb9b8a69-workload-certs\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq\" (UID: \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" Oct 30 20:05:10.157486 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:10.157469 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a255c4a5-c090-49d2-84ba-536dfb9b8a69-istio-podinfo\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq\" (UID: \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" Oct 30 20:05:10.157944 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:10.157901 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:10.188735 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:10.158488 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:05:10.284656 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:10.251598 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:10.284656 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:10.252449 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:05:10.287193 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:10.287160 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zkdz\" (UniqueName: \"kubernetes.io/projected/a255c4a5-c090-49d2-84ba-536dfb9b8a69-kube-api-access-4zkdz\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq\" (UID: \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" Oct 30 20:05:10.287438 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:10.287416 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a255c4a5-c090-49d2-84ba-536dfb9b8a69-workload-certs\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq\" (UID: \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" Oct 30 20:05:10.290715 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:10.290685 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a255c4a5-c090-49d2-84ba-536dfb9b8a69-istio-podinfo\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq\" (UID: \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" Oct 30 20:05:10.290930 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:10.290910 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a255c4a5-c090-49d2-84ba-536dfb9b8a69-istiod-ca-cert\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq\" (UID: \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" Oct 30 20:05:10.294780 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:10.294750 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a255c4a5-c090-49d2-84ba-536dfb9b8a69-workload-socket\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq\" (UID: \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" Oct 30 20:05:10.294995 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:10.294975 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a255c4a5-c090-49d2-84ba-536dfb9b8a69-istio-token\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq\" (UID: \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" Oct 30 20:05:10.295208 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:10.295191 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a255c4a5-c090-49d2-84ba-536dfb9b8a69-istio-envoy\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq\" (UID: \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" Oct 30 20:05:10.416691 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:10.416662 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a255c4a5-c090-49d2-84ba-536dfb9b8a69-istio-data\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq\" (UID: \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" Oct 30 20:05:10.417373 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:10.417344 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a255c4a5-c090-49d2-84ba-536dfb9b8a69-istio-data\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq\" (UID: \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" Oct 30 20:05:10.417482 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:10.354526 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a255c4a5-c090-49d2-84ba-536dfb9b8a69-istiod-ca-cert\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq\" (UID: \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" Oct 30 20:05:10.417482 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:10.321576 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a255c4a5-c090-49d2-84ba-536dfb9b8a69-workload-certs\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq\" (UID: \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" Oct 30 20:05:10.417663 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:10.353246 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a255c4a5-c090-49d2-84ba-536dfb9b8a69-workload-socket\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq\" (UID: \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" Oct 30 20:05:10.417937 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:10.417885 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a255c4a5-c090-49d2-84ba-536dfb9b8a69-istio-podinfo\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq\" (UID: \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" Oct 30 20:05:10.465734 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:10.418294 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a255c4a5-c090-49d2-84ba-536dfb9b8a69-credential-socket\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq\" (UID: \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" Oct 30 20:05:10.466265 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:10.466108 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a255c4a5-c090-49d2-84ba-536dfb9b8a69-credential-socket\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq\" (UID: \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" Oct 30 20:05:10.585745 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:10.549349 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a255c4a5-c090-49d2-84ba-536dfb9b8a69-istio-token\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq\" (UID: \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" Oct 30 20:05:10.668651 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:10.661831 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:10.668651 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:10.662651 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:05:10.801910 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:10.801862 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq"] Oct 30 20:05:10.845632 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:10.845562 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:10.984647 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:10.931982 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:10.984647 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:10.963631 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" Oct 30 20:05:10.984647 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:10.963793 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" Oct 30 20:05:10.984647 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:10.980521 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a255c4a5-c090-49d2-84ba-536dfb9b8a69-istio-envoy\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq\" (UID: \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" Oct 30 20:05:11.069822 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.010416 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zkdz\" (UniqueName: \"kubernetes.io/projected/a255c4a5-c090-49d2-84ba-536dfb9b8a69-kube-api-access-4zkdz\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq\" (UID: \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" Oct 30 20:05:11.089381 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.082448 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 30 20:05:11.089381 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.083927 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 20:05:11.111296 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.110821 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 20:05:11.162992 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.162939 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 20:05:11.192672 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.186415 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" Oct 30 20:05:11.225833 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.225799 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:05:11.259023 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.258983 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 20:05:11.349122 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.347523 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:05:11.544646 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.541667 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d"] Oct 30 20:05:11.612165 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.611720 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:11.692802 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.692753 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" Oct 30 20:05:11.709458 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.709408 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d"] Oct 30 20:05:11.791775 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.791745 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r"] Oct 30 20:05:11.805709 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.716909 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" Oct 30 20:05:11.872581 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.871785 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:05:11.872581 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.871848 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" Oct 30 20:05:11.874226 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.873089 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" Oct 30 20:05:11.874226 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.873478 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r"] Oct 30 20:05:11.911407 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.902171 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h"] Oct 30 20:05:11.947300 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.945922 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" Oct 30 20:05:11.959715 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.958753 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98275e84-2e20-46db-8264-27ec2ac2e2f2-istio-token\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r\" (UID: \"98275e84-2e20-46db-8264-27ec2ac2e2f2\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" Oct 30 20:05:11.959715 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.958803 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98275e84-2e20-46db-8264-27ec2ac2e2f2-credential-socket\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r\" (UID: \"98275e84-2e20-46db-8264-27ec2ac2e2f2\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" Oct 30 20:05:11.959715 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.958916 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-workload-certs\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h\" (UID: \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" Oct 30 20:05:11.959715 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.958961 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-istio-envoy\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h\" (UID: \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" Oct 30 20:05:11.959715 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.958986 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98275e84-2e20-46db-8264-27ec2ac2e2f2-istio-podinfo\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r\" (UID: \"98275e84-2e20-46db-8264-27ec2ac2e2f2\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" Oct 30 20:05:11.959715 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.959037 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-istiod-ca-cert\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h\" (UID: \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" Oct 30 20:05:11.959715 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.959070 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d44744f7-8f4c-4c79-978b-52649ce5a5ed-workload-socket\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d\" (UID: \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" Oct 30 20:05:11.959715 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.959136 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98275e84-2e20-46db-8264-27ec2ac2e2f2-workload-certs\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r\" (UID: \"98275e84-2e20-46db-8264-27ec2ac2e2f2\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" Oct 30 20:05:11.959715 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.959166 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98275e84-2e20-46db-8264-27ec2ac2e2f2-workload-socket\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r\" (UID: \"98275e84-2e20-46db-8264-27ec2ac2e2f2\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" Oct 30 20:05:11.959715 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.959196 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d44744f7-8f4c-4c79-978b-52649ce5a5ed-istio-envoy\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d\" (UID: \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" Oct 30 20:05:11.959715 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.959264 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-istio-podinfo\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h\" (UID: \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" Oct 30 20:05:11.959715 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.959311 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-istio-data\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h\" (UID: \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" Oct 30 20:05:11.959715 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.959363 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d44744f7-8f4c-4c79-978b-52649ce5a5ed-istio-podinfo\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d\" (UID: \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" Oct 30 20:05:11.959715 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.959435 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98275e84-2e20-46db-8264-27ec2ac2e2f2-istiod-ca-cert\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r\" (UID: \"98275e84-2e20-46db-8264-27ec2ac2e2f2\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" Oct 30 20:05:11.959715 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.959462 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98275e84-2e20-46db-8264-27ec2ac2e2f2-istio-envoy\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r\" (UID: \"98275e84-2e20-46db-8264-27ec2ac2e2f2\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" Oct 30 20:05:11.959715 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.959489 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d44744f7-8f4c-4c79-978b-52649ce5a5ed-istio-token\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d\" (UID: \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" Oct 30 20:05:11.959715 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.959573 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d44744f7-8f4c-4c79-978b-52649ce5a5ed-istio-data\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d\" (UID: \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" Oct 30 20:05:11.963517 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.959605 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d44744f7-8f4c-4c79-978b-52649ce5a5ed-credential-socket\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d\" (UID: \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" Oct 30 20:05:11.963517 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.963478 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxt7j\" (UniqueName: \"kubernetes.io/projected/d44744f7-8f4c-4c79-978b-52649ce5a5ed-kube-api-access-hxt7j\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d\" (UID: \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" Oct 30 20:05:11.963972 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.963931 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d44744f7-8f4c-4c79-978b-52649ce5a5ed-workload-certs\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d\" (UID: \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" Oct 30 20:05:11.964209 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.964151 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-istio-token\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h\" (UID: \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" Oct 30 20:05:11.966218 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.964666 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86cmb\" (UniqueName: \"kubernetes.io/projected/98275e84-2e20-46db-8264-27ec2ac2e2f2-kube-api-access-86cmb\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r\" (UID: \"98275e84-2e20-46db-8264-27ec2ac2e2f2\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" Oct 30 20:05:11.966218 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.964744 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-credential-socket\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h\" (UID: \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" Oct 30 20:05:11.966218 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.964771 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcvs2\" (UniqueName: \"kubernetes.io/projected/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-kube-api-access-rcvs2\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h\" (UID: \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" Oct 30 20:05:11.966218 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.964924 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98275e84-2e20-46db-8264-27ec2ac2e2f2-istio-data\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r\" (UID: \"98275e84-2e20-46db-8264-27ec2ac2e2f2\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" Oct 30 20:05:11.966218 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.964979 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d44744f7-8f4c-4c79-978b-52649ce5a5ed-istiod-ca-cert\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d\" (UID: \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" Oct 30 20:05:11.966218 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.965030 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-workload-socket\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h\" (UID: \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" Oct 30 20:05:11.967256 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:11.967177 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h"] Oct 30 20:05:12.068876 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.068844 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-istio-podinfo\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h\" (UID: \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" Oct 30 20:05:12.070952 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.070923 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-istio-data\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h\" (UID: \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" Oct 30 20:05:12.071566 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.071539 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d44744f7-8f4c-4c79-978b-52649ce5a5ed-istio-podinfo\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d\" (UID: \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" Oct 30 20:05:12.072744 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.072724 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98275e84-2e20-46db-8264-27ec2ac2e2f2-istiod-ca-cert\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r\" (UID: \"98275e84-2e20-46db-8264-27ec2ac2e2f2\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" Oct 30 20:05:12.073664 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.073638 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98275e84-2e20-46db-8264-27ec2ac2e2f2-istio-envoy\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r\" (UID: \"98275e84-2e20-46db-8264-27ec2ac2e2f2\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" Oct 30 20:05:12.074192 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.074156 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d44744f7-8f4c-4c79-978b-52649ce5a5ed-istio-token\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d\" (UID: \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" Oct 30 20:05:12.074359 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.074341 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d44744f7-8f4c-4c79-978b-52649ce5a5ed-istio-data\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d\" (UID: \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" Oct 30 20:05:12.076031 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.071460 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-istio-data\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h\" (UID: \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" Oct 30 20:05:12.076031 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.074483 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d44744f7-8f4c-4c79-978b-52649ce5a5ed-credential-socket\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d\" (UID: \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" Oct 30 20:05:12.076031 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.074530 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxt7j\" (UniqueName: \"kubernetes.io/projected/d44744f7-8f4c-4c79-978b-52649ce5a5ed-kube-api-access-hxt7j\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d\" (UID: \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" Oct 30 20:05:12.076031 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.074591 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d44744f7-8f4c-4c79-978b-52649ce5a5ed-workload-certs\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d\" (UID: \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" Oct 30 20:05:12.076031 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.074639 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-istio-token\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h\" (UID: \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" Oct 30 20:05:12.076031 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.074675 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86cmb\" (UniqueName: \"kubernetes.io/projected/98275e84-2e20-46db-8264-27ec2ac2e2f2-kube-api-access-86cmb\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r\" (UID: \"98275e84-2e20-46db-8264-27ec2ac2e2f2\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" Oct 30 20:05:12.076031 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.074741 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-credential-socket\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h\" (UID: \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" Oct 30 20:05:12.076031 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.074789 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcvs2\" (UniqueName: \"kubernetes.io/projected/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-kube-api-access-rcvs2\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h\" (UID: \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" Oct 30 20:05:12.076031 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.074895 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98275e84-2e20-46db-8264-27ec2ac2e2f2-istio-data\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r\" (UID: \"98275e84-2e20-46db-8264-27ec2ac2e2f2\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" Oct 30 20:05:12.076031 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.074934 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d44744f7-8f4c-4c79-978b-52649ce5a5ed-istiod-ca-cert\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d\" (UID: \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" Oct 30 20:05:12.076031 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.074997 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-workload-socket\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h\" (UID: \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" Oct 30 20:05:12.076031 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.075029 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98275e84-2e20-46db-8264-27ec2ac2e2f2-istio-token\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r\" (UID: \"98275e84-2e20-46db-8264-27ec2ac2e2f2\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" Oct 30 20:05:12.076031 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.075067 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98275e84-2e20-46db-8264-27ec2ac2e2f2-credential-socket\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r\" (UID: \"98275e84-2e20-46db-8264-27ec2ac2e2f2\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" Oct 30 20:05:12.076031 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.075156 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-workload-certs\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h\" (UID: \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" Oct 30 20:05:12.076031 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.075195 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-istio-envoy\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h\" (UID: \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" Oct 30 20:05:12.076031 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.075222 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98275e84-2e20-46db-8264-27ec2ac2e2f2-istio-podinfo\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r\" (UID: \"98275e84-2e20-46db-8264-27ec2ac2e2f2\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" Oct 30 20:05:12.076031 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.075285 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-istiod-ca-cert\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h\" (UID: \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" Oct 30 20:05:12.076031 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.075325 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d44744f7-8f4c-4c79-978b-52649ce5a5ed-workload-socket\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d\" (UID: \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" Oct 30 20:05:12.076031 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.075398 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98275e84-2e20-46db-8264-27ec2ac2e2f2-workload-certs\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r\" (UID: \"98275e84-2e20-46db-8264-27ec2ac2e2f2\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" Oct 30 20:05:12.076031 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.075446 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98275e84-2e20-46db-8264-27ec2ac2e2f2-workload-socket\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r\" (UID: \"98275e84-2e20-46db-8264-27ec2ac2e2f2\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" Oct 30 20:05:12.076031 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.075494 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d44744f7-8f4c-4c79-978b-52649ce5a5ed-istio-envoy\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d\" (UID: \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" Oct 30 20:05:12.076031 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.075720 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d44744f7-8f4c-4c79-978b-52649ce5a5ed-credential-socket\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d\" (UID: \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" Oct 30 20:05:12.076031 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.075975 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d44744f7-8f4c-4c79-978b-52649ce5a5ed-istio-data\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d\" (UID: \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" Oct 30 20:05:12.076031 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.073560 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98275e84-2e20-46db-8264-27ec2ac2e2f2-istiod-ca-cert\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r\" (UID: \"98275e84-2e20-46db-8264-27ec2ac2e2f2\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" Oct 30 20:05:12.077311 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.076277 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-workload-socket\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h\" (UID: \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" Oct 30 20:05:12.077311 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.076842 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d44744f7-8f4c-4c79-978b-52649ce5a5ed-workload-certs\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d\" (UID: \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" Oct 30 20:05:12.078443 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.077595 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-credential-socket\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h\" (UID: \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" Oct 30 20:05:12.078443 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.078045 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98275e84-2e20-46db-8264-27ec2ac2e2f2-istio-data\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r\" (UID: \"98275e84-2e20-46db-8264-27ec2ac2e2f2\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" Oct 30 20:05:12.078604 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.078474 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d44744f7-8f4c-4c79-978b-52649ce5a5ed-istiod-ca-cert\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d\" (UID: \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" Oct 30 20:05:12.078802 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.078779 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98275e84-2e20-46db-8264-27ec2ac2e2f2-credential-socket\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r\" (UID: \"98275e84-2e20-46db-8264-27ec2ac2e2f2\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" Oct 30 20:05:12.079418 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.079399 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-workload-certs\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h\" (UID: \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" Oct 30 20:05:12.079829 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.079805 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d44744f7-8f4c-4c79-978b-52649ce5a5ed-workload-socket\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d\" (UID: \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" Oct 30 20:05:12.080727 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.080708 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-istiod-ca-cert\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h\" (UID: \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" Oct 30 20:05:12.081316 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.081288 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98275e84-2e20-46db-8264-27ec2ac2e2f2-workload-certs\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r\" (UID: \"98275e84-2e20-46db-8264-27ec2ac2e2f2\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" Oct 30 20:05:12.082028 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.081824 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d44744f7-8f4c-4c79-978b-52649ce5a5ed-istio-podinfo\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d\" (UID: \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" Oct 30 20:05:12.082337 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.082287 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98275e84-2e20-46db-8264-27ec2ac2e2f2-workload-socket\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r\" (UID: \"98275e84-2e20-46db-8264-27ec2ac2e2f2\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" Oct 30 20:05:12.082715 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.082666 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d44744f7-8f4c-4c79-978b-52649ce5a5ed-istio-envoy\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d\" (UID: \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" Oct 30 20:05:12.112213 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.111692 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98275e84-2e20-46db-8264-27ec2ac2e2f2-istio-envoy\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r\" (UID: \"98275e84-2e20-46db-8264-27ec2ac2e2f2\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" Oct 30 20:05:12.122208 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.121663 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-istio-podinfo\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h\" (UID: \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" Oct 30 20:05:12.128943 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.127868 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-istio-envoy\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h\" (UID: \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" Oct 30 20:05:12.174928 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.174899 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86cmb\" (UniqueName: \"kubernetes.io/projected/98275e84-2e20-46db-8264-27ec2ac2e2f2-kube-api-access-86cmb\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r\" (UID: \"98275e84-2e20-46db-8264-27ec2ac2e2f2\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" Oct 30 20:05:12.182470 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.179972 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d44744f7-8f4c-4c79-978b-52649ce5a5ed-istio-token\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d\" (UID: \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" Oct 30 20:05:12.182470 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.180977 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98275e84-2e20-46db-8264-27ec2ac2e2f2-istio-podinfo\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r\" (UID: \"98275e84-2e20-46db-8264-27ec2ac2e2f2\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" Oct 30 20:05:12.182470 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.181249 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98275e84-2e20-46db-8264-27ec2ac2e2f2-istio-token\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r\" (UID: \"98275e84-2e20-46db-8264-27ec2ac2e2f2\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" Oct 30 20:05:12.182470 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.181300 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcvs2\" (UniqueName: \"kubernetes.io/projected/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-kube-api-access-rcvs2\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h\" (UID: \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" Oct 30 20:05:12.182470 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.181678 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-istio-token\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h\" (UID: \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" Oct 30 20:05:12.182470 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.182394 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxt7j\" (UniqueName: \"kubernetes.io/projected/d44744f7-8f4c-4c79-978b-52649ce5a5ed-kube-api-access-hxt7j\") pod \"e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d\" (UID: \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\") " pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" Oct 30 20:05:12.231868 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.231567 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" Oct 30 20:05:12.235869 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.235839 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" Oct 30 20:05:12.235981 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.235842 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq"] Oct 30 20:05:12.236925 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:05:12.236901 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda255c4a5_c090_49d2_84ba_536dfb9b8a69.slice/crio-e4a1b171f325420d687994df96ba8900e2b68544df4e5579ff279092b25c1fda WatchSource:0}: Error finding container e4a1b171f325420d687994df96ba8900e2b68544df4e5579ff279092b25c1fda: Status 404 returned error can't find the container with id e4a1b171f325420d687994df96ba8900e2b68544df4e5579ff279092b25c1fda Oct 30 20:05:12.242859 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.242821 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:05:12.242983 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.242905 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:05:12.242983 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.242947 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:05:12.365593 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.365561 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" Oct 30 20:05:12.600762 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.600736 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r"] Oct 30 20:05:12.601917 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:05:12.601889 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd44744f7_8f4c_4c79_978b_52649ce5a5ed.slice/crio-1778982336a81f990b393dcaec7f58cea3668b8d23b07095d2c9a546bbeb8462 WatchSource:0}: Error finding container 1778982336a81f990b393dcaec7f58cea3668b8d23b07095d2c9a546bbeb8462: Status 404 returned error can't find the container with id 1778982336a81f990b393dcaec7f58cea3668b8d23b07095d2c9a546bbeb8462 Oct 30 20:05:12.603225 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:05:12.602837 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98275e84_2e20_46db_8264_27ec2ac2e2f2.slice/crio-55f92264bb19aec87a9df7cbc8b68908112c59f40da1db7e9bfd362207a8e997 WatchSource:0}: Error finding container 55f92264bb19aec87a9df7cbc8b68908112c59f40da1db7e9bfd362207a8e997: Status 404 returned error can't find the container with id 55f92264bb19aec87a9df7cbc8b68908112c59f40da1db7e9bfd362207a8e997 Oct 30 20:05:12.606001 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.605970 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:05:12.606090 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.606048 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:05:12.606160 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.606091 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:05:12.606160 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.606091 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:05:12.606259 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.606166 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:05:12.606259 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.606210 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:05:12.607096 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.607071 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d"] Oct 30 20:05:12.706311 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.706282 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h"] Oct 30 20:05:12.768653 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.760984 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:05:12.768653 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.761145 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:05:12.768653 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.761200 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:05:12.892110 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.892028 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" event={"ID":"a255c4a5-c090-49d2-84ba-536dfb9b8a69","Type":"ContainerStarted","Data":"500b9e0dbc558d86839f26b86c64e2b7e81db3e1fda3e37f9dcbb093e05578d4"} Oct 30 20:05:12.892339 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.892320 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" event={"ID":"a255c4a5-c090-49d2-84ba-536dfb9b8a69","Type":"ContainerStarted","Data":"e4a1b171f325420d687994df96ba8900e2b68544df4e5579ff279092b25c1fda"} Oct 30 20:05:12.896286 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.896245 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" event={"ID":"98275e84-2e20-46db-8264-27ec2ac2e2f2","Type":"ContainerStarted","Data":"c3fb1272968557cba700fced30ca4bd585345d53ef20b3c596965d955230c051"} Oct 30 20:05:12.896769 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.896581 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" event={"ID":"98275e84-2e20-46db-8264-27ec2ac2e2f2","Type":"ContainerStarted","Data":"55f92264bb19aec87a9df7cbc8b68908112c59f40da1db7e9bfd362207a8e997"} Oct 30 20:05:12.897812 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.897790 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" event={"ID":"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90","Type":"ContainerStarted","Data":"694bc27926151698b194dd89499b5182f2cfe6ef4844635dca205016eea82ecb"} Oct 30 20:05:12.899564 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.899541 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" event={"ID":"d44744f7-8f4c-4c79-978b-52649ce5a5ed","Type":"ContainerStarted","Data":"314ac6568ace40ce03f7bf80c6852789d144db290701be78a5ba82f0f10dc195"} Oct 30 20:05:12.899701 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.899584 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" event={"ID":"d44744f7-8f4c-4c79-978b-52649ce5a5ed","Type":"ContainerStarted","Data":"1778982336a81f990b393dcaec7f58cea3668b8d23b07095d2c9a546bbeb8462"} Oct 30 20:05:12.947656 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:12.947591 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" podStartSLOduration=3.9475772449999997 podStartE2EDuration="3.947577245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:12.943069911 +0000 UTC m=+1074.355783690" watchObservedRunningTime="2026-10-30 20:05:12.947577245 +0000 UTC m=+1074.360291024" Oct 30 20:05:13.012664 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:13.002943 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" podStartSLOduration=3.002923835 podStartE2EDuration="3.002923835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:13.000420819 +0000 UTC m=+1074.413134612" watchObservedRunningTime="2026-10-30 20:05:13.002923835 +0000 UTC m=+1074.415637613" Oct 30 20:05:13.078203 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:13.078128 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" podStartSLOduration=3.078107393 podStartE2EDuration="3.078107393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:13.075224084 +0000 UTC m=+1074.487937876" watchObservedRunningTime="2026-10-30 20:05:13.078107393 +0000 UTC m=+1074.490821175" Oct 30 20:05:13.233661 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:13.233627 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" Oct 30 20:05:13.234002 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:13.233979 5088 patch_prober.go:29] interesting pod/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 20:05:13.234156 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:13.234130 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" podUID="d44744f7-8f4c-4c79-978b-52649ce5a5ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 20:05:13.237454 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:13.237430 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" Oct 30 20:05:13.237783 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:13.237764 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 20:05:13.237958 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:13.237934 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" podUID="98275e84-2e20-46db-8264-27ec2ac2e2f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 20:05:13.713701 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:13.712088 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" Oct 30 20:05:13.723824 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:13.723775 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 20:05:13.724176 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:13.724101 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" podUID="a255c4a5-c090-49d2-84ba-536dfb9b8a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 20:05:13.970487 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:13.970320 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" event={"ID":"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90","Type":"ContainerStarted","Data":"c681ce095c069d984ee329fe861fc5299eb161f9aeb2b45159bb8880bd808928"} Oct 30 20:05:14.077391 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:14.077323 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" podStartSLOduration=4.077299354 podStartE2EDuration="4.077299354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:14.067107961 +0000 UTC m=+1075.479821743" watchObservedRunningTime="2026-10-30 20:05:14.077299354 +0000 UTC m=+1075.490013137" Oct 30 20:05:14.232437 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:14.232036 5088 patch_prober.go:29] interesting pod/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 20:05:14.232437 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:14.232098 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" podUID="d44744f7-8f4c-4c79-978b-52649ce5a5ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 20:05:14.255638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:14.239372 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 20:05:14.255638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:14.239430 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" podUID="98275e84-2e20-46db-8264-27ec2ac2e2f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 20:05:14.374667 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:14.374195 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" Oct 30 20:05:14.374667 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:14.374350 5088 patch_prober.go:29] interesting pod/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 20:05:14.374667 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:14.374398 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" podUID="e402d580-e9a6-4e35-a2ff-e5ef2d78eb90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 20:05:14.715707 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:14.706829 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 20:05:14.715707 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:14.706934 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" podUID="a255c4a5-c090-49d2-84ba-536dfb9b8a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 20:05:15.194818 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:15.194764 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:15.245826 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:15.245765 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 20:05:15.246018 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:15.245847 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" podUID="98275e84-2e20-46db-8264-27ec2ac2e2f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 20:05:15.246576 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:15.246552 5088 patch_prober.go:29] interesting pod/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 20:05:15.246760 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:15.246737 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" podUID="d44744f7-8f4c-4c79-978b-52649ce5a5ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 20:05:15.368056 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:15.368016 5088 patch_prober.go:29] interesting pod/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 20:05:15.368308 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:15.368281 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" podUID="e402d580-e9a6-4e35-a2ff-e5ef2d78eb90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 20:05:15.652370 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:15.649713 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:15.652370 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:15.650114 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 20:05:15.709638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:15.705096 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 20:05:15.709638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:15.705191 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" podUID="a255c4a5-c090-49d2-84ba-536dfb9b8a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 20:05:16.144661 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:16.135708 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:16.174637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:16.167839 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 20:05:16.233066 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:16.233015 5088 patch_prober.go:29] interesting pod/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 20:05:16.233357 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:16.233324 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" podUID="d44744f7-8f4c-4c79-978b-52649ce5a5ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 20:05:16.238034 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:16.238003 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 20:05:16.238265 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:16.238238 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" podUID="98275e84-2e20-46db-8264-27ec2ac2e2f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 20:05:16.244779 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:16.244736 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:16.245188 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:16.245155 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" Oct 30 20:05:16.355240 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:16.355144 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:16.396975 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:16.385254 5088 patch_prober.go:29] interesting pod/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 20:05:16.396975 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:16.385324 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" podUID="e402d580-e9a6-4e35-a2ff-e5ef2d78eb90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 20:05:16.568634 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:16.565723 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:16.671921 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:16.671789 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 20:05:16.672659 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:16.672177 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 20:05:16.674067 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:16.674039 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" Oct 30 20:05:16.674268 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:16.674242 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:05:16.712819 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:16.712693 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 20:05:16.712819 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:16.712765 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" podUID="a255c4a5-c090-49d2-84ba-536dfb9b8a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 20:05:17.212726 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:17.172798 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:17.240050 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:17.240012 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 20:05:17.240423 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:17.240389 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" podUID="98275e84-2e20-46db-8264-27ec2ac2e2f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 20:05:17.258684 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:17.252335 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:17.326786 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:17.325024 5088 patch_prober.go:29] interesting pod/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 20:05:17.326786 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:17.325107 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" podUID="d44744f7-8f4c-4c79-978b-52649ce5a5ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 20:05:17.384661 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:17.364029 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:17.384661 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:17.370097 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:05:17.384661 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:17.370154 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:05:17.393642 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:17.390552 5088 patch_prober.go:29] interesting pod/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 20:05:17.393642 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:17.390653 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" podUID="e402d580-e9a6-4e35-a2ff-e5ef2d78eb90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 20:05:17.681809 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:17.681759 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:17.686418 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:17.682039 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:17.713879 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:17.713844 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 20:05:17.714116 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:17.714091 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" podUID="a255c4a5-c090-49d2-84ba-536dfb9b8a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 20:05:18.150277 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:18.149699 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 30 20:05:18.176774 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:18.176722 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 20:05:18.235182 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:18.235133 5088 patch_prober.go:29] interesting pod/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 20:05:18.235837 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:18.235785 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" podUID="d44744f7-8f4c-4c79-978b-52649ce5a5ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 20:05:18.236798 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:18.236776 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 20:05:18.236972 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:18.236951 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" podUID="98275e84-2e20-46db-8264-27ec2ac2e2f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 20:05:18.255387 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:18.255328 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:18.366894 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:18.366747 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:18.371238 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:18.371198 5088 patch_prober.go:29] interesting pod/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 20:05:18.371462 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:18.371436 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" podUID="e402d580-e9a6-4e35-a2ff-e5ef2d78eb90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 20:05:18.688451 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:18.688401 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:18.954690 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:18.690818 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:18.954690 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:18.847300 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:19.023651 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:18.998052 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 20:05:19.023651 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:18.998117 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" podUID="a255c4a5-c090-49d2-84ba-536dfb9b8a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 20:05:19.193317 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:19.193256 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 30 20:05:19.247221 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:19.236726 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:19.314637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:19.302751 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 20:05:19.314637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:19.302835 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" podUID="98275e84-2e20-46db-8264-27ec2ac2e2f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 20:05:19.338637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:19.336109 5088 patch_prober.go:29] interesting pod/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 20:05:19.338637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:19.336181 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" podUID="d44744f7-8f4c-4c79-978b-52649ce5a5ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 20:05:19.374640 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:19.370979 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 20:05:19.374640 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:19.373404 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 20:05:19.378635 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:19.376799 5088 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:19.378635 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:19.377350 5088 patch_prober.go:29] interesting pod/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 20:05:19.378635 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:19.377396 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" podUID="e402d580-e9a6-4e35-a2ff-e5ef2d78eb90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 20:05:19.646707 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:19.604483 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 20:05:19.768761 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:19.723781 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 20:05:19.797096 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:19.797034 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" podUID="a255c4a5-c090-49d2-84ba-536dfb9b8a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 20:05:19.879961 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:19.879902 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 20:05:19.975729 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:19.958630 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:20.205654 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:20.196829 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:20.235714 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:20.232032 5088 patch_prober.go:29] interesting pod/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 20:05:20.235714 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:20.232106 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" podUID="d44744f7-8f4c-4c79-978b-52649ce5a5ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 20:05:20.240250 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:20.240202 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:20.384637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:20.381488 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:20.435193 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:20.435145 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 20:05:20.478648 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:20.450690 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 20:05:20.553910 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:20.553775 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" podUID="98275e84-2e20-46db-8264-27ec2ac2e2f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 20:05:20.574642 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:20.505276 5088 patch_prober.go:29] interesting pod/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 20:05:20.574929 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:20.574883 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" podUID="e402d580-e9a6-4e35-a2ff-e5ef2d78eb90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 20:05:20.578570 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:20.577115 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" Oct 30 20:05:20.581182 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:20.581164 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:05:20.611347 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:20.586875 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:05:20.611347 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:20.591877 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" Oct 30 20:05:20.619126 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:20.619098 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 20:05:20.736836 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:20.734497 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:05:20.739147 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:20.739113 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 20:05:20.739275 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:20.739170 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" podUID="a255c4a5-c090-49d2-84ba-536dfb9b8a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 20:05:21.232901 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:21.232861 5088 patch_prober.go:29] interesting pod/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 20:05:21.233133 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:21.232925 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" podUID="d44744f7-8f4c-4c79-978b-52649ce5a5ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 20:05:21.236728 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:21.236695 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 20:05:21.236872 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:21.236753 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" podUID="98275e84-2e20-46db-8264-27ec2ac2e2f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 20:05:21.366876 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:21.366833 5088 patch_prober.go:29] interesting pod/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 20:05:21.367049 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:21.366896 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" podUID="e402d580-e9a6-4e35-a2ff-e5ef2d78eb90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 20:05:21.710710 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:21.710605 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" Oct 30 20:05:21.711402 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:21.711376 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" Oct 30 20:05:21.712472 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:21.712444 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" Oct 30 20:05:22.443404 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:05:22.443366 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:05:22.516596 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:22.515637 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" Oct 30 20:05:22.516596 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:22.516540 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" Oct 30 20:05:22.518983 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:22.518956 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" Oct 30 20:05:22.584161 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:22.584131 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" Oct 30 20:05:22.584839 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:22.584809 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" Oct 30 20:05:22.585962 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:22.585940 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" Oct 30 20:05:22.662194 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:22.655100 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" Oct 30 20:05:22.662194 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:22.656412 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" Oct 30 20:05:22.662194 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:22.661091 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" Oct 30 20:05:23.018264 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:23.018229 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7db4898-8jr6z"] Oct 30 20:05:23.024367 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:23.024340 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7db4898-8jr6z" Oct 30 20:05:23.043664 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:23.043631 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7db4898-8jr6z"] Oct 30 20:05:23.043838 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:23.043747 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:05:23.044011 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:23.043986 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:05:23.122242 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:23.122206 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c1f9b70-c9a5-4f10-9fee-8c75f815378e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7db4898-8jr6z\" (UID: \"8c1f9b70-c9a5-4f10-9fee-8c75f815378e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7db4898-8jr6z" Oct 30 20:05:23.122416 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:23.122384 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c1f9b70-c9a5-4f10-9fee-8c75f815378e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7db4898-8jr6z\" (UID: \"8c1f9b70-c9a5-4f10-9fee-8c75f815378e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7db4898-8jr6z" Oct 30 20:05:23.122600 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:23.122583 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5bww\" (UniqueName: \"kubernetes.io/projected/8c1f9b70-c9a5-4f10-9fee-8c75f815378e-kube-api-access-c5bww\") pod \"maas-api-revoke-sub-d7db4898-8jr6z\" (UID: \"8c1f9b70-c9a5-4f10-9fee-8c75f815378e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7db4898-8jr6z" Oct 30 20:05:23.224008 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:23.223976 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c1f9b70-c9a5-4f10-9fee-8c75f815378e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7db4898-8jr6z\" (UID: \"8c1f9b70-c9a5-4f10-9fee-8c75f815378e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7db4898-8jr6z" Oct 30 20:05:23.224216 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:23.224079 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c1f9b70-c9a5-4f10-9fee-8c75f815378e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7db4898-8jr6z\" (UID: \"8c1f9b70-c9a5-4f10-9fee-8c75f815378e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7db4898-8jr6z" Oct 30 20:05:23.224290 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:23.224236 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5bww\" (UniqueName: \"kubernetes.io/projected/8c1f9b70-c9a5-4f10-9fee-8c75f815378e-kube-api-access-c5bww\") pod \"maas-api-revoke-sub-d7db4898-8jr6z\" (UID: \"8c1f9b70-c9a5-4f10-9fee-8c75f815378e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7db4898-8jr6z" Oct 30 20:05:23.224818 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:23.224794 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c1f9b70-c9a5-4f10-9fee-8c75f815378e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7db4898-8jr6z\" (UID: \"8c1f9b70-c9a5-4f10-9fee-8c75f815378e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7db4898-8jr6z" Oct 30 20:05:23.226501 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:23.226478 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c1f9b70-c9a5-4f10-9fee-8c75f815378e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7db4898-8jr6z\" (UID: \"8c1f9b70-c9a5-4f10-9fee-8c75f815378e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7db4898-8jr6z" Oct 30 20:05:23.251043 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:23.251012 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5bww\" (UniqueName: \"kubernetes.io/projected/8c1f9b70-c9a5-4f10-9fee-8c75f815378e-kube-api-access-c5bww\") pod \"maas-api-revoke-sub-d7db4898-8jr6z\" (UID: \"8c1f9b70-c9a5-4f10-9fee-8c75f815378e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7db4898-8jr6z" Oct 30 20:05:23.341777 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:23.341680 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7db4898-8jr6z" Oct 30 20:05:23.651516 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:05:23.651478 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c1f9b70_c9a5_4f10_9fee_8c75f815378e.slice/crio-ac4341fc9c050e726ea2a3ed1df5bb5837597e9495bb26934bd7266c5bc1be40 WatchSource:0}: Error finding container ac4341fc9c050e726ea2a3ed1df5bb5837597e9495bb26934bd7266c5bc1be40: Status 404 returned error can't find the container with id ac4341fc9c050e726ea2a3ed1df5bb5837597e9495bb26934bd7266c5bc1be40 Oct 30 20:05:23.651695 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:23.651673 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7db4898-8jr6z"] Oct 30 20:05:23.938188 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:23.938153 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7db4898-8jr6z" event={"ID":"8c1f9b70-c9a5-4f10-9fee-8c75f815378e","Type":"ContainerStarted","Data":"ac4341fc9c050e726ea2a3ed1df5bb5837597e9495bb26934bd7266c5bc1be40"} Oct 30 20:05:24.948769 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:24.948680 5088 generic.go:356] "Generic (PLEG): container finished" podID="8c1f9b70-c9a5-4f10-9fee-8c75f815378e" containerID="51662691140ba7074b7a7ec3d5cc198d59ee5eef0cae8c18126b5643bce0c917" exitCode=0 Oct 30 20:05:24.948769 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:24.948742 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7db4898-8jr6z" event={"ID":"8c1f9b70-c9a5-4f10-9fee-8c75f815378e","Type":"ContainerDied","Data":"51662691140ba7074b7a7ec3d5cc198d59ee5eef0cae8c18126b5643bce0c917"} Oct 30 20:05:25.676094 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:25.676057 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2"] Oct 30 20:05:25.691106 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:25.691072 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" Oct 30 20:05:25.711639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:25.711596 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2"] Oct 30 20:05:25.773516 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:25.773478 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0c9594e-8534-4538-a4bc-2d40d918e362-credential-socket\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2\" (UID: \"a0c9594e-8534-4538-a4bc-2d40d918e362\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" Oct 30 20:05:25.773516 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:25.773516 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqb82\" (UniqueName: \"kubernetes.io/projected/a0c9594e-8534-4538-a4bc-2d40d918e362-kube-api-access-mqb82\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2\" (UID: \"a0c9594e-8534-4538-a4bc-2d40d918e362\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" Oct 30 20:05:25.773748 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:25.773546 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0c9594e-8534-4538-a4bc-2d40d918e362-workload-socket\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2\" (UID: \"a0c9594e-8534-4538-a4bc-2d40d918e362\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" Oct 30 20:05:25.773748 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:25.773698 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0c9594e-8534-4538-a4bc-2d40d918e362-workload-certs\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2\" (UID: \"a0c9594e-8534-4538-a4bc-2d40d918e362\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" Oct 30 20:05:25.773888 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:25.773750 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0c9594e-8534-4538-a4bc-2d40d918e362-istiod-ca-cert\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2\" (UID: \"a0c9594e-8534-4538-a4bc-2d40d918e362\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" Oct 30 20:05:25.773888 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:25.773847 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0c9594e-8534-4538-a4bc-2d40d918e362-istio-token\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2\" (UID: \"a0c9594e-8534-4538-a4bc-2d40d918e362\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" Oct 30 20:05:25.774089 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:25.774061 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0c9594e-8534-4538-a4bc-2d40d918e362-istio-podinfo\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2\" (UID: \"a0c9594e-8534-4538-a4bc-2d40d918e362\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" Oct 30 20:05:25.774174 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:25.774161 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0c9594e-8534-4538-a4bc-2d40d918e362-istio-data\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2\" (UID: \"a0c9594e-8534-4538-a4bc-2d40d918e362\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" Oct 30 20:05:25.774287 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:25.774270 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0c9594e-8534-4538-a4bc-2d40d918e362-istio-envoy\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2\" (UID: \"a0c9594e-8534-4538-a4bc-2d40d918e362\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" Oct 30 20:05:25.876018 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:25.875980 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0c9594e-8534-4538-a4bc-2d40d918e362-istiod-ca-cert\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2\" (UID: \"a0c9594e-8534-4538-a4bc-2d40d918e362\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" Oct 30 20:05:25.876215 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:25.876027 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0c9594e-8534-4538-a4bc-2d40d918e362-istio-token\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2\" (UID: \"a0c9594e-8534-4538-a4bc-2d40d918e362\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" Oct 30 20:05:25.876215 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:25.876171 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0c9594e-8534-4538-a4bc-2d40d918e362-istio-podinfo\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2\" (UID: \"a0c9594e-8534-4538-a4bc-2d40d918e362\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" Oct 30 20:05:25.876332 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:25.876253 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0c9594e-8534-4538-a4bc-2d40d918e362-istio-data\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2\" (UID: \"a0c9594e-8534-4538-a4bc-2d40d918e362\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" Oct 30 20:05:25.876383 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:25.876361 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0c9594e-8534-4538-a4bc-2d40d918e362-istio-envoy\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2\" (UID: \"a0c9594e-8534-4538-a4bc-2d40d918e362\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" Oct 30 20:05:25.876813 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:25.876443 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0c9594e-8534-4538-a4bc-2d40d918e362-credential-socket\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2\" (UID: \"a0c9594e-8534-4538-a4bc-2d40d918e362\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" Oct 30 20:05:25.876813 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:25.876470 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqb82\" (UniqueName: \"kubernetes.io/projected/a0c9594e-8534-4538-a4bc-2d40d918e362-kube-api-access-mqb82\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2\" (UID: \"a0c9594e-8534-4538-a4bc-2d40d918e362\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" Oct 30 20:05:25.876813 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:25.876513 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0c9594e-8534-4538-a4bc-2d40d918e362-workload-socket\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2\" (UID: \"a0c9594e-8534-4538-a4bc-2d40d918e362\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" Oct 30 20:05:25.876813 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:25.876546 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0c9594e-8534-4538-a4bc-2d40d918e362-workload-certs\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2\" (UID: \"a0c9594e-8534-4538-a4bc-2d40d918e362\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" Oct 30 20:05:25.876813 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:25.876759 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0c9594e-8534-4538-a4bc-2d40d918e362-istio-data\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2\" (UID: \"a0c9594e-8534-4538-a4bc-2d40d918e362\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" Oct 30 20:05:25.877101 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:25.876920 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0c9594e-8534-4538-a4bc-2d40d918e362-istiod-ca-cert\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2\" (UID: \"a0c9594e-8534-4538-a4bc-2d40d918e362\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" Oct 30 20:05:25.877101 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:25.876954 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0c9594e-8534-4538-a4bc-2d40d918e362-workload-certs\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2\" (UID: \"a0c9594e-8534-4538-a4bc-2d40d918e362\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" Oct 30 20:05:25.877101 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:25.877031 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0c9594e-8534-4538-a4bc-2d40d918e362-workload-socket\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2\" (UID: \"a0c9594e-8534-4538-a4bc-2d40d918e362\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" Oct 30 20:05:25.877255 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:25.877227 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0c9594e-8534-4538-a4bc-2d40d918e362-credential-socket\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2\" (UID: \"a0c9594e-8534-4538-a4bc-2d40d918e362\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" Oct 30 20:05:25.878980 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:25.878951 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0c9594e-8534-4538-a4bc-2d40d918e362-istio-podinfo\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2\" (UID: \"a0c9594e-8534-4538-a4bc-2d40d918e362\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" Oct 30 20:05:25.879061 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:25.879046 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0c9594e-8534-4538-a4bc-2d40d918e362-istio-envoy\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2\" (UID: \"a0c9594e-8534-4538-a4bc-2d40d918e362\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" Oct 30 20:05:25.880041 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:25.880017 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g"] Oct 30 20:05:25.886390 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:25.886366 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" Oct 30 20:05:25.902777 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:25.902744 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0c9594e-8534-4538-a4bc-2d40d918e362-istio-token\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2\" (UID: \"a0c9594e-8534-4538-a4bc-2d40d918e362\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" Oct 30 20:05:25.908386 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:25.908043 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqb82\" (UniqueName: \"kubernetes.io/projected/a0c9594e-8534-4538-a4bc-2d40d918e362-kube-api-access-mqb82\") pod \"e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2\" (UID: \"a0c9594e-8534-4538-a4bc-2d40d918e362\") " pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" Oct 30 20:05:25.915182 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:25.915147 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g"] Oct 30 20:05:25.978481 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:25.978439 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f2b960b-8db8-4b84-b6d1-9ac12823e278-istio-envoy\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g\" (UID: \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" Oct 30 20:05:25.979076 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:25.978591 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f2b960b-8db8-4b84-b6d1-9ac12823e278-istiod-ca-cert\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g\" (UID: \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" Oct 30 20:05:25.979076 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:25.978698 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f2b960b-8db8-4b84-b6d1-9ac12823e278-istio-podinfo\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g\" (UID: \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" Oct 30 20:05:25.979076 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:25.978745 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f2b960b-8db8-4b84-b6d1-9ac12823e278-istio-data\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g\" (UID: \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" Oct 30 20:05:25.979076 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:25.978840 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f2b960b-8db8-4b84-b6d1-9ac12823e278-workload-certs\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g\" (UID: \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" Oct 30 20:05:25.979076 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:25.978881 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrcb9\" (UniqueName: \"kubernetes.io/projected/1f2b960b-8db8-4b84-b6d1-9ac12823e278-kube-api-access-zrcb9\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g\" (UID: \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" Oct 30 20:05:25.979076 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:25.978910 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f2b960b-8db8-4b84-b6d1-9ac12823e278-credential-socket\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g\" (UID: \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" Oct 30 20:05:25.979076 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:25.979014 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f2b960b-8db8-4b84-b6d1-9ac12823e278-workload-socket\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g\" (UID: \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" Oct 30 20:05:25.979458 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:25.979116 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f2b960b-8db8-4b84-b6d1-9ac12823e278-istio-token\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g\" (UID: \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" Oct 30 20:05:26.015072 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.014939 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" Oct 30 20:05:26.080652 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.080575 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f2b960b-8db8-4b84-b6d1-9ac12823e278-workload-certs\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g\" (UID: \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" Oct 30 20:05:26.080652 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.080642 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrcb9\" (UniqueName: \"kubernetes.io/projected/1f2b960b-8db8-4b84-b6d1-9ac12823e278-kube-api-access-zrcb9\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g\" (UID: \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" Oct 30 20:05:26.080906 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.080665 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f2b960b-8db8-4b84-b6d1-9ac12823e278-credential-socket\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g\" (UID: \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" Oct 30 20:05:26.080906 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.080710 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f2b960b-8db8-4b84-b6d1-9ac12823e278-workload-socket\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g\" (UID: \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" Oct 30 20:05:26.080906 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.080771 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f2b960b-8db8-4b84-b6d1-9ac12823e278-istio-token\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g\" (UID: \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" Oct 30 20:05:26.080906 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.080831 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f2b960b-8db8-4b84-b6d1-9ac12823e278-istio-envoy\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g\" (UID: \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" Oct 30 20:05:26.080906 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.080883 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f2b960b-8db8-4b84-b6d1-9ac12823e278-istiod-ca-cert\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g\" (UID: \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" Oct 30 20:05:26.081171 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.080911 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f2b960b-8db8-4b84-b6d1-9ac12823e278-istio-podinfo\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g\" (UID: \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" Oct 30 20:05:26.081171 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.080936 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f2b960b-8db8-4b84-b6d1-9ac12823e278-istio-data\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g\" (UID: \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" Oct 30 20:05:26.081350 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.081331 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f2b960b-8db8-4b84-b6d1-9ac12823e278-istio-data\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g\" (UID: \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" Oct 30 20:05:26.081521 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.081506 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f2b960b-8db8-4b84-b6d1-9ac12823e278-workload-certs\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g\" (UID: \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" Oct 30 20:05:26.081993 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.081970 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f2b960b-8db8-4b84-b6d1-9ac12823e278-credential-socket\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g\" (UID: \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" Oct 30 20:05:26.082524 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.082491 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f2b960b-8db8-4b84-b6d1-9ac12823e278-istiod-ca-cert\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g\" (UID: \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" Oct 30 20:05:26.082646 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.082628 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f2b960b-8db8-4b84-b6d1-9ac12823e278-workload-socket\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g\" (UID: \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" Oct 30 20:05:26.085118 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.085067 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f2b960b-8db8-4b84-b6d1-9ac12823e278-istio-envoy\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g\" (UID: \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" Oct 30 20:05:26.086149 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.086125 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f2b960b-8db8-4b84-b6d1-9ac12823e278-istio-podinfo\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g\" (UID: \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" Oct 30 20:05:26.110208 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.110171 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f2b960b-8db8-4b84-b6d1-9ac12823e278-istio-token\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g\" (UID: \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" Oct 30 20:05:26.112098 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.111897 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrcb9\" (UniqueName: \"kubernetes.io/projected/1f2b960b-8db8-4b84-b6d1-9ac12823e278-kube-api-access-zrcb9\") pod \"e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g\" (UID: \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\") " pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" Oct 30 20:05:26.127159 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.127126 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7db4898-8jr6z" Oct 30 20:05:26.181916 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.181876 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c1f9b70-c9a5-4f10-9fee-8c75f815378e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c1f9b70-c9a5-4f10-9fee-8c75f815378e-maas-api-openshift-ca\") pod \"8c1f9b70-c9a5-4f10-9fee-8c75f815378e\" (UID: \"8c1f9b70-c9a5-4f10-9fee-8c75f815378e\") " Oct 30 20:05:26.182088 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.181952 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c1f9b70-c9a5-4f10-9fee-8c75f815378e-kube-api-access-c5bww\" (UniqueName: \"kubernetes.io/projected/8c1f9b70-c9a5-4f10-9fee-8c75f815378e-kube-api-access-c5bww\") pod \"8c1f9b70-c9a5-4f10-9fee-8c75f815378e\" (UID: \"8c1f9b70-c9a5-4f10-9fee-8c75f815378e\") " Oct 30 20:05:26.182158 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.182140 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c1f9b70-c9a5-4f10-9fee-8c75f815378e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c1f9b70-c9a5-4f10-9fee-8c75f815378e-maas-api-xks-ca\") pod \"8c1f9b70-c9a5-4f10-9fee-8c75f815378e\" (UID: \"8c1f9b70-c9a5-4f10-9fee-8c75f815378e\") " Oct 30 20:05:26.182515 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.182484 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c1f9b70-c9a5-4f10-9fee-8c75f815378e-maas-api-openshift-ca" pod "8c1f9b70-c9a5-4f10-9fee-8c75f815378e" (UID: "8c1f9b70-c9a5-4f10-9fee-8c75f815378e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:05:26.182835 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.182779 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c1f9b70-c9a5-4f10-9fee-8c75f815378e-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:05:26.184922 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.184879 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c1f9b70-c9a5-4f10-9fee-8c75f815378e-maas-api-xks-ca" pod "8c1f9b70-c9a5-4f10-9fee-8c75f815378e" (UID: "8c1f9b70-c9a5-4f10-9fee-8c75f815378e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:05:26.185499 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.185473 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c1f9b70-c9a5-4f10-9fee-8c75f815378e-kube-api-access-c5bww" pod "8c1f9b70-c9a5-4f10-9fee-8c75f815378e" (UID: "8c1f9b70-c9a5-4f10-9fee-8c75f815378e"). InnerVolumeSpecName "kube-api-access-c5bww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:05:26.252976 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.252945 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" Oct 30 20:05:26.284083 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.284048 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c1f9b70-c9a5-4f10-9fee-8c75f815378e-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:05:26.284083 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.284076 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5bww\" (UniqueName: \"kubernetes.io/projected/8c1f9b70-c9a5-4f10-9fee-8c75f815378e-kube-api-access-c5bww\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:05:26.379680 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.379651 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2"] Oct 30 20:05:26.380004 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:05:26.379970 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0c9594e_8534_4538_a4bc_2d40d918e362.slice/crio-a60e757d74802457065a4422e0679f5350f09a38c4a0a49d045e980d88d8514b WatchSource:0}: Error finding container a60e757d74802457065a4422e0679f5350f09a38c4a0a49d045e980d88d8514b: Status 404 returned error can't find the container with id a60e757d74802457065a4422e0679f5350f09a38c4a0a49d045e980d88d8514b Oct 30 20:05:26.384472 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.384430 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:05:26.384583 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.384515 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:05:26.384583 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.384560 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:05:26.569138 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:05:26.569104 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f2b960b_8db8_4b84_b6d1_9ac12823e278.slice/crio-3301c524dcd532234860794bb45de4e8819f1bec03fee43f69be6ef58fd55675 WatchSource:0}: Error finding container 3301c524dcd532234860794bb45de4e8819f1bec03fee43f69be6ef58fd55675: Status 404 returned error can't find the container with id 3301c524dcd532234860794bb45de4e8819f1bec03fee43f69be6ef58fd55675 Oct 30 20:05:26.569367 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.569344 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g"] Oct 30 20:05:26.571794 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.571758 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:05:26.571896 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.571837 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:05:26.571896 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.571887 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:05:26.984857 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.984810 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7db4898-8jr6z" event={"ID":"8c1f9b70-c9a5-4f10-9fee-8c75f815378e","Type":"ContainerDied","Data":"ac4341fc9c050e726ea2a3ed1df5bb5837597e9495bb26934bd7266c5bc1be40"} Oct 30 20:05:26.985334 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.984883 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac4341fc9c050e726ea2a3ed1df5bb5837597e9495bb26934bd7266c5bc1be40" Oct 30 20:05:26.985334 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.984832 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7db4898-8jr6z" Oct 30 20:05:26.987194 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.987163 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" event={"ID":"a0c9594e-8534-4538-a4bc-2d40d918e362","Type":"ContainerStarted","Data":"e2b0d690456a79235aab26cb2f341ef4af7ae03bad18f9d8222163d3b012ead0"} Oct 30 20:05:26.987392 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.987370 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" event={"ID":"a0c9594e-8534-4538-a4bc-2d40d918e362","Type":"ContainerStarted","Data":"a60e757d74802457065a4422e0679f5350f09a38c4a0a49d045e980d88d8514b"} Oct 30 20:05:26.989912 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.989888 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" event={"ID":"1f2b960b-8db8-4b84-b6d1-9ac12823e278","Type":"ContainerStarted","Data":"ac6ef7d6f2e1e074bfa2ccd91e5df5aacd6d4d99966f02c65d29e197fcc55de3"} Oct 30 20:05:26.990066 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:26.989949 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" event={"ID":"1f2b960b-8db8-4b84-b6d1-9ac12823e278","Type":"ContainerStarted","Data":"3301c524dcd532234860794bb45de4e8819f1bec03fee43f69be6ef58fd55675"} Oct 30 20:05:27.016778 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:27.015635 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" Oct 30 20:05:27.027775 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:27.027738 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 20:05:27.027936 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:27.027814 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" podUID="a0c9594e-8534-4538-a4bc-2d40d918e362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 20:05:27.049091 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:27.049037 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" podStartSLOduration=2.049021962 podStartE2EDuration="2.049021962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:27.043702393 +0000 UTC m=+1088.456416197" watchObservedRunningTime="2026-10-30 20:05:27.049021962 +0000 UTC m=+1088.461735737" Oct 30 20:05:27.097904 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:27.097839 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" podStartSLOduration=2.0978208130000002 podStartE2EDuration="2.097820813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:27.094082329 +0000 UTC m=+1088.506796119" watchObservedRunningTime="2026-10-30 20:05:27.097820813 +0000 UTC m=+1088.510534593" Oct 30 20:05:27.255815 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:27.255277 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" Oct 30 20:05:27.268120 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:27.267758 5088 patch_prober.go:29] interesting pod/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 20:05:27.268120 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:27.267823 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" podUID="1f2b960b-8db8-4b84-b6d1-9ac12823e278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 20:05:28.027636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:28.026680 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 20:05:28.027636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:28.026753 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" podUID="a0c9594e-8534-4538-a4bc-2d40d918e362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 20:05:28.253475 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:28.253436 5088 patch_prober.go:29] interesting pod/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 20:05:28.253640 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:28.253503 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" podUID="1f2b960b-8db8-4b84-b6d1-9ac12823e278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 20:05:28.381495 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:28.381405 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" Oct 30 20:05:28.448950 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:28.448913 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" Oct 30 20:05:29.019645 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:29.019050 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 20:05:29.019645 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:29.019117 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" podUID="a0c9594e-8534-4538-a4bc-2d40d918e362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 20:05:29.259548 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:29.256950 5088 patch_prober.go:29] interesting pod/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 20:05:29.259548 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:29.257030 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" podUID="1f2b960b-8db8-4b84-b6d1-9ac12823e278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 20:05:30.020636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:30.020300 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 20:05:30.020636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:30.020362 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" podUID="a0c9594e-8534-4538-a4bc-2d40d918e362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 20:05:30.253507 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:30.253469 5088 patch_prober.go:29] interesting pod/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 20:05:30.253730 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:30.253531 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" podUID="1f2b960b-8db8-4b84-b6d1-9ac12823e278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 20:05:31.033637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:31.026723 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 20:05:31.033637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:31.026785 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" podUID="a0c9594e-8534-4538-a4bc-2d40d918e362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 20:05:31.254529 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:31.254485 5088 patch_prober.go:29] interesting pod/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 20:05:31.254701 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:31.254552 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" podUID="1f2b960b-8db8-4b84-b6d1-9ac12823e278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 20:05:31.640682 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:31.634843 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0d7df91-4crlv"] Oct 30 20:05:31.641983 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:31.641956 5088 cpu_manager.go:414] "containerMap: removing container" podUID="8c1f9b70-c9a5-4f10-9fee-8c75f815378e" containerName="revoke-keys" Oct 30 20:05:31.641983 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:31.641979 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c1f9b70-c9a5-4f10-9fee-8c75f815378e" containerName="revoke-keys" Oct 30 20:05:31.642155 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:31.642116 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c1f9b70-c9a5-4f10-9fee-8c75f815378e" containerName="revoke-keys" Oct 30 20:05:31.647139 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:31.647118 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d7df91-4crlv" Oct 30 20:05:31.656576 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:31.656549 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0d7df91-4crlv"] Oct 30 20:05:31.672945 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:31.672920 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:05:31.673066 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:31.672967 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:05:31.816636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:31.807836 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb63c45f-e1d2-41b8-8723-55f67fa0f5a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0d7df91-4crlv\" (UID: \"bb63c45f-e1d2-41b8-8723-55f67fa0f5a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d7df91-4crlv" Oct 30 20:05:31.816636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:31.807974 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqc9k\" (UniqueName: \"kubernetes.io/projected/bb63c45f-e1d2-41b8-8723-55f67fa0f5a2-kube-api-access-lqc9k\") pod \"maas-api-revoke-sub-d0d7df91-4crlv\" (UID: \"bb63c45f-e1d2-41b8-8723-55f67fa0f5a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d7df91-4crlv" Oct 30 20:05:31.816636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:31.808137 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb63c45f-e1d2-41b8-8723-55f67fa0f5a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0d7df91-4crlv\" (UID: \"bb63c45f-e1d2-41b8-8723-55f67fa0f5a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d7df91-4crlv" Oct 30 20:05:31.918681 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:31.910793 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb63c45f-e1d2-41b8-8723-55f67fa0f5a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0d7df91-4crlv\" (UID: \"bb63c45f-e1d2-41b8-8723-55f67fa0f5a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d7df91-4crlv" Oct 30 20:05:31.918681 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:31.910986 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb63c45f-e1d2-41b8-8723-55f67fa0f5a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0d7df91-4crlv\" (UID: \"bb63c45f-e1d2-41b8-8723-55f67fa0f5a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d7df91-4crlv" Oct 30 20:05:31.918681 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:31.911125 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqc9k\" (UniqueName: \"kubernetes.io/projected/bb63c45f-e1d2-41b8-8723-55f67fa0f5a2-kube-api-access-lqc9k\") pod \"maas-api-revoke-sub-d0d7df91-4crlv\" (UID: \"bb63c45f-e1d2-41b8-8723-55f67fa0f5a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d7df91-4crlv" Oct 30 20:05:31.918681 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:31.917359 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb63c45f-e1d2-41b8-8723-55f67fa0f5a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0d7df91-4crlv\" (UID: \"bb63c45f-e1d2-41b8-8723-55f67fa0f5a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d7df91-4crlv" Oct 30 20:05:31.942317 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:31.940093 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb63c45f-e1d2-41b8-8723-55f67fa0f5a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0d7df91-4crlv\" (UID: \"bb63c45f-e1d2-41b8-8723-55f67fa0f5a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d7df91-4crlv" Oct 30 20:05:31.942486 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:31.942470 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqc9k\" (UniqueName: \"kubernetes.io/projected/bb63c45f-e1d2-41b8-8723-55f67fa0f5a2-kube-api-access-lqc9k\") pod \"maas-api-revoke-sub-d0d7df91-4crlv\" (UID: \"bb63c45f-e1d2-41b8-8723-55f67fa0f5a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d7df91-4crlv" Oct 30 20:05:31.964499 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:31.964468 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d7df91-4crlv" Oct 30 20:05:32.055653 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:32.050320 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 20:05:32.055653 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:32.050389 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" podUID="a0c9594e-8534-4538-a4bc-2d40d918e362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 20:05:32.322216 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:32.321736 5088 patch_prober.go:29] interesting pod/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 20:05:32.322216 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:32.321812 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" podUID="1f2b960b-8db8-4b84-b6d1-9ac12823e278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 20:05:33.031640 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:33.021753 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 20:05:33.031640 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:33.021837 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" podUID="a0c9594e-8534-4538-a4bc-2d40d918e362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 20:05:33.442432 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:33.441133 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 20:05:33.487036 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:33.486991 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:33.654561 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:33.654506 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 20:05:33.656889 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:33.656849 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 30 20:05:33.658403 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:33.658356 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": context deadline exceeded" Oct 30 20:05:33.746807 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:33.511778 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 30 20:05:33.746807 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:33.555033 5088 patch_prober.go:29] interesting pod/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 20:05:33.746807 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:33.742374 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" podUID="1f2b960b-8db8-4b84-b6d1-9ac12823e278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 20:05:33.898756 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:33.898708 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:33.917822 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:33.917773 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0d7df91-4crlv"] Oct 30 20:05:33.942586 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:33.942528 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded" Oct 30 20:05:34.033473 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:34.033434 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:34.139438 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:34.034839 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 20:05:34.139781 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:34.139713 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" podUID="a0c9594e-8534-4538-a4bc-2d40d918e362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 20:05:34.139923 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:34.095427 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:34.271148 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:34.271029 5088 patch_prober.go:29] interesting pod/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 20:05:34.271474 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:34.271434 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" podUID="1f2b960b-8db8-4b84-b6d1-9ac12823e278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 20:05:34.474985 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:34.474940 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 20:05:34.518809 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:34.518771 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:05:34.677691 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:34.667446 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:34.677691 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:34.668251 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:05:34.677691 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:34.670053 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:34.677691 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:34.670388 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 20:05:34.740540 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:34.711700 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:34.740540 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:34.712659 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" Oct 30 20:05:34.910639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:34.804737 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 30 20:05:34.910639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:34.806292 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" Oct 30 20:05:34.910639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:34.910163 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:34.998908 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:34.998701 5088 patch_prober.go:29] interesting pod/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:05:35.464738 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:35.464690 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" podUID="d209c20c-000f-4cc4-9792-5a71eeb0a773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:35.464963 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:35.421407 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:05:35.490665 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:35.254428 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 20:05:35.557663 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:35.556633 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:35.557909 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:35.253342 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 20:05:35.558073 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:35.558047 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" podUID="a0c9594e-8534-4538-a4bc-2d40d918e362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 20:05:35.558184 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:35.558159 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" Oct 30 20:05:35.572639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:35.265166 5088 patch_prober.go:29] interesting pod/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 20:05:35.572926 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:35.572886 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" podUID="1f2b960b-8db8-4b84-b6d1-9ac12823e278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 20:05:35.675386 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:35.675289 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 30 20:05:35.685727 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:35.676333 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 20:05:35.731922 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:35.716446 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:35.855811 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:35.809113 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:35.914638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:35.910704 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:36.045059 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:36.030985 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 20:05:36.045059 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:36.031044 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" podUID="a0c9594e-8534-4538-a4bc-2d40d918e362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 20:05:36.142652 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:36.140505 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d7df91-4crlv" event={"ID":"bb63c45f-e1d2-41b8-8723-55f67fa0f5a2","Type":"ContainerStarted","Data":"fa635a94ae6c6ae8f7efd9df011cc15133d98b9f0ef642d02d77799800c41b70"} Oct 30 20:05:36.342004 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:36.341906 5088 patch_prober.go:29] interesting pod/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 20:05:36.342281 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:36.342246 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" podUID="1f2b960b-8db8-4b84-b6d1-9ac12823e278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 20:05:36.536636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:36.535813 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:36.565638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:36.562753 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:36.565638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:36.565035 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:36.676929 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:36.676818 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:36.678232 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:36.678203 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 20:05:36.786344 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:36.759752 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 20:05:36.815900 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:36.814066 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:36.913726 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:36.913598 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:37.043079 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:37.043020 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 20:05:37.043376 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:37.043328 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" podUID="a0c9594e-8534-4538-a4bc-2d40d918e362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 20:05:37.329690 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:37.290765 5088 patch_prober.go:29] interesting pod/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 20:05:37.379171 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:37.377104 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" podUID="1f2b960b-8db8-4b84-b6d1-9ac12823e278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 20:05:37.619005 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:37.618880 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 20:05:37.839647 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:37.834134 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:37.843642 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:37.839950 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 20:05:37.905636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:37.860276 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 30 20:05:37.907002 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:37.860412 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 20:05:37.930899 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:37.930856 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 20:05:38.194693 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:38.165955 5088 generic.go:356] "Generic (PLEG): container finished" podID="bb63c45f-e1d2-41b8-8723-55f67fa0f5a2" containerID="148ed002f2b11c8794736f4511c4466813ee332e1411e6af730235e536896102" exitCode=0 Oct 30 20:05:38.323114 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:38.186188 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d7df91-4crlv" event={"ID":"bb63c45f-e1d2-41b8-8723-55f67fa0f5a2","Type":"ContainerDied","Data":"148ed002f2b11c8794736f4511c4466813ee332e1411e6af730235e536896102"} Oct 30 20:05:38.323439 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:38.292235 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 20:05:38.323632 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:38.323588 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" podUID="a0c9594e-8534-4538-a4bc-2d40d918e362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 20:05:38.343958 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:38.327222 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 20:05:38.437637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:38.428736 5088 patch_prober.go:29] interesting pod/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 20:05:38.437637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:38.428827 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" podUID="1f2b960b-8db8-4b84-b6d1-9ac12823e278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 20:05:38.901648 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:38.884709 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:38.953953 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:38.945785 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 20:05:39.025630 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:39.013483 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 20:05:39.025630 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:39.013654 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 30 20:05:39.025630 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:39.013729 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 20:05:39.028436 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:39.028404 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:05:39.093750 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:39.093694 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" podUID="98275e84-2e20-46db-8264-27ec2ac2e2f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:39.094053 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:39.035723 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 20:05:39.392975 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:39.392927 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 20:05:39.547168 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:39.547127 5088 patch_prober.go:29] interesting pod/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 20:05:39.547358 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:39.547190 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" podUID="1f2b960b-8db8-4b84-b6d1-9ac12823e278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 20:05:39.557756 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:39.553677 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 20:05:39.636666 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:39.631678 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" podUID="a0c9594e-8534-4538-a4bc-2d40d918e362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 20:05:39.636666 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:39.621930 5088 patch_prober.go:29] interesting pod/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:05:39.636666 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:39.631755 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" podUID="d44744f7-8f4c-4c79-978b-52649ce5a5ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:39.842639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:39.826290 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" Oct 30 20:05:39.869773 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:39.869745 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:05:39.870869 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:39.870833 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:39.943425 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:39.943394 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:05:40.224278 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:40.224186 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:05:40.334514 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:40.317559 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" Oct 30 20:05:40.341458 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:40.341434 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" Oct 30 20:05:40.342484 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:40.331854 5088 patch_prober.go:29] interesting pod/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 20:05:40.344080 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:40.344044 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" podUID="1f2b960b-8db8-4b84-b6d1-9ac12823e278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 20:05:40.397414 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:40.397384 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 20:05:40.583111 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:40.583078 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d7df91-4crlv" Oct 30 20:05:40.664002 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:40.663969 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb63c45f-e1d2-41b8-8723-55f67fa0f5a2-kube-api-access-lqc9k\" (UniqueName: \"kubernetes.io/projected/bb63c45f-e1d2-41b8-8723-55f67fa0f5a2-kube-api-access-lqc9k\") pod \"bb63c45f-e1d2-41b8-8723-55f67fa0f5a2\" (UID: \"bb63c45f-e1d2-41b8-8723-55f67fa0f5a2\") " Oct 30 20:05:40.664188 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:40.664044 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb63c45f-e1d2-41b8-8723-55f67fa0f5a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb63c45f-e1d2-41b8-8723-55f67fa0f5a2-maas-api-openshift-ca\") pod \"bb63c45f-e1d2-41b8-8723-55f67fa0f5a2\" (UID: \"bb63c45f-e1d2-41b8-8723-55f67fa0f5a2\") " Oct 30 20:05:40.664486 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:40.664462 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb63c45f-e1d2-41b8-8723-55f67fa0f5a2-maas-api-openshift-ca" pod "bb63c45f-e1d2-41b8-8723-55f67fa0f5a2" (UID: "bb63c45f-e1d2-41b8-8723-55f67fa0f5a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:05:40.665023 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:40.664995 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb63c45f-e1d2-41b8-8723-55f67fa0f5a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb63c45f-e1d2-41b8-8723-55f67fa0f5a2-maas-api-xks-ca\") pod \"bb63c45f-e1d2-41b8-8723-55f67fa0f5a2\" (UID: \"bb63c45f-e1d2-41b8-8723-55f67fa0f5a2\") " Oct 30 20:05:40.666273 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:40.666249 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb63c45f-e1d2-41b8-8723-55f67fa0f5a2-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:05:40.671876 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:40.671845 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb63c45f-e1d2-41b8-8723-55f67fa0f5a2-kube-api-access-lqc9k" pod "bb63c45f-e1d2-41b8-8723-55f67fa0f5a2" (UID: "bb63c45f-e1d2-41b8-8723-55f67fa0f5a2"). InnerVolumeSpecName "kube-api-access-lqc9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:05:40.675751 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:40.675723 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb63c45f-e1d2-41b8-8723-55f67fa0f5a2-maas-api-xks-ca" pod "bb63c45f-e1d2-41b8-8723-55f67fa0f5a2" (UID: "bb63c45f-e1d2-41b8-8723-55f67fa0f5a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:05:40.742443 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:40.742419 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d7df91-4crlv" Oct 30 20:05:40.748387 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:40.748350 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d7df91-4crlv" event={"ID":"bb63c45f-e1d2-41b8-8723-55f67fa0f5a2","Type":"ContainerDied","Data":"fa635a94ae6c6ae8f7efd9df011cc15133d98b9f0ef642d02d77799800c41b70"} Oct 30 20:05:40.748563 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:40.748408 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa635a94ae6c6ae8f7efd9df011cc15133d98b9f0ef642d02d77799800c41b70" Oct 30 20:05:40.768075 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:40.767592 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb63c45f-e1d2-41b8-8723-55f67fa0f5a2-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:05:40.768075 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:40.767657 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqc9k\" (UniqueName: \"kubernetes.io/projected/bb63c45f-e1d2-41b8-8723-55f67fa0f5a2-kube-api-access-lqc9k\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:05:40.884636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:40.884234 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" Oct 30 20:05:40.890635 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:40.886383 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" Oct 30 20:05:40.890635 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:40.887339 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" Oct 30 20:05:41.295527 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:41.286717 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" Oct 30 20:05:41.360475 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:41.360435 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" Oct 30 20:05:41.756261 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:41.756232 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" Oct 30 20:05:48.614950 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:48.614918 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cad5119-c2c5m"] Oct 30 20:05:48.615657 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:48.615640 5088 cpu_manager.go:414] "containerMap: removing container" podUID="bb63c45f-e1d2-41b8-8723-55f67fa0f5a2" containerName="revoke-keys" Oct 30 20:05:48.615657 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:48.615658 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb63c45f-e1d2-41b8-8723-55f67fa0f5a2" containerName="revoke-keys" Oct 30 20:05:48.615797 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:48.615784 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb63c45f-e1d2-41b8-8723-55f67fa0f5a2" containerName="revoke-keys" Oct 30 20:05:48.620919 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:48.620900 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cad5119-c2c5m" Oct 30 20:05:48.767212 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:48.763189 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:05:48.767212 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:48.763242 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:05:48.778470 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:48.778437 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cad5119-c2c5m"] Oct 30 20:05:48.865164 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:48.865088 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs4r8\" (UniqueName: \"kubernetes.io/projected/92c93d36-4f3f-4670-a8f4-c8d2fd372cff-kube-api-access-bs4r8\") pod \"maas-api-revoke-sub-9cad5119-c2c5m\" (UID: \"92c93d36-4f3f-4670-a8f4-c8d2fd372cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cad5119-c2c5m" Oct 30 20:05:48.865306 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:48.865268 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92c93d36-4f3f-4670-a8f4-c8d2fd372cff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9cad5119-c2c5m\" (UID: \"92c93d36-4f3f-4670-a8f4-c8d2fd372cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cad5119-c2c5m" Oct 30 20:05:48.865706 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:48.865676 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92c93d36-4f3f-4670-a8f4-c8d2fd372cff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9cad5119-c2c5m\" (UID: \"92c93d36-4f3f-4670-a8f4-c8d2fd372cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cad5119-c2c5m" Oct 30 20:05:48.910129 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:48.910097 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:05:48.967226 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:48.967199 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92c93d36-4f3f-4670-a8f4-c8d2fd372cff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9cad5119-c2c5m\" (UID: \"92c93d36-4f3f-4670-a8f4-c8d2fd372cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cad5119-c2c5m" Oct 30 20:05:48.967362 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:48.967284 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs4r8\" (UniqueName: \"kubernetes.io/projected/92c93d36-4f3f-4670-a8f4-c8d2fd372cff-kube-api-access-bs4r8\") pod \"maas-api-revoke-sub-9cad5119-c2c5m\" (UID: \"92c93d36-4f3f-4670-a8f4-c8d2fd372cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cad5119-c2c5m" Oct 30 20:05:48.967409 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:48.967396 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92c93d36-4f3f-4670-a8f4-c8d2fd372cff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9cad5119-c2c5m\" (UID: \"92c93d36-4f3f-4670-a8f4-c8d2fd372cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cad5119-c2c5m" Oct 30 20:05:48.967882 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:48.967861 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92c93d36-4f3f-4670-a8f4-c8d2fd372cff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9cad5119-c2c5m\" (UID: \"92c93d36-4f3f-4670-a8f4-c8d2fd372cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cad5119-c2c5m" Oct 30 20:05:48.969859 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:48.969840 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92c93d36-4f3f-4670-a8f4-c8d2fd372cff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9cad5119-c2c5m\" (UID: \"92c93d36-4f3f-4670-a8f4-c8d2fd372cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cad5119-c2c5m" Oct 30 20:05:48.995061 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:48.995034 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs4r8\" (UniqueName: \"kubernetes.io/projected/92c93d36-4f3f-4670-a8f4-c8d2fd372cff-kube-api-access-bs4r8\") pod \"maas-api-revoke-sub-9cad5119-c2c5m\" (UID: \"92c93d36-4f3f-4670-a8f4-c8d2fd372cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cad5119-c2c5m" Oct 30 20:05:49.084234 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:49.084208 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cad5119-c2c5m" Oct 30 20:05:49.407507 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:05:49.407470 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92c93d36_4f3f_4670_a8f4_c8d2fd372cff.slice/crio-cddd2d2d505c2c253e440360d00be83ab08a4fcac589cdaaf9f8fda8238978d5 WatchSource:0}: Error finding container cddd2d2d505c2c253e440360d00be83ab08a4fcac589cdaaf9f8fda8238978d5: Status 404 returned error can't find the container with id cddd2d2d505c2c253e440360d00be83ab08a4fcac589cdaaf9f8fda8238978d5 Oct 30 20:05:49.408651 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:49.408587 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cad5119-c2c5m"] Oct 30 20:05:49.893638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:49.888908 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cad5119-c2c5m" event={"ID":"92c93d36-4f3f-4670-a8f4-c8d2fd372cff","Type":"ContainerStarted","Data":"9780e31b2f0eb4868cc069ebc55fda95e3ce12500c33763138c5cf7e02729dc9"} Oct 30 20:05:49.893638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:49.888982 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cad5119-c2c5m" event={"ID":"92c93d36-4f3f-4670-a8f4-c8d2fd372cff","Type":"ContainerStarted","Data":"cddd2d2d505c2c253e440360d00be83ab08a4fcac589cdaaf9f8fda8238978d5"} Oct 30 20:05:50.954636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:50.954182 5088 generic.go:356] "Generic (PLEG): container finished" podID="92c93d36-4f3f-4670-a8f4-c8d2fd372cff" containerID="9780e31b2f0eb4868cc069ebc55fda95e3ce12500c33763138c5cf7e02729dc9" exitCode=0 Oct 30 20:05:50.954636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:50.954369 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cad5119-c2c5m" event={"ID":"92c93d36-4f3f-4670-a8f4-c8d2fd372cff","Type":"ContainerDied","Data":"9780e31b2f0eb4868cc069ebc55fda95e3ce12500c33763138c5cf7e02729dc9"} Oct 30 20:05:51.584644 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:51.570708 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:51.611765 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:51.611697 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:51.685646 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:51.684708 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:51.691642 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:51.685891 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 30 20:05:51.723647 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:51.716811 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:51.850723 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:51.849711 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:51.850723 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:51.850190 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 20:05:51.929639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:51.926725 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:52.600775 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:52.598734 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:52.600775 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:52.599507 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:05:52.873784 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:52.854838 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:52.902638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:52.887290 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 30 20:05:52.919226 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:52.919173 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 20:05:53.037344 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:53.037291 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:53.037912 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:52.924278 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:53.101715 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:53.073458 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 20:05:53.664180 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:53.484012 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:53.664180 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:53.617664 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:53.664180 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:53.651141 5088 patch_prober.go:29] interesting pod/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:05:53.664180 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:53.653168 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" podUID="d44744f7-8f4c-4c79-978b-52649ce5a5ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:53.664180 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:05:53.652877 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:05:53.702698 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:53.702658 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:05:53.702967 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:53.702938 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" podUID="98275e84-2e20-46db-8264-27ec2ac2e2f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:53.722059 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:53.722024 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" Oct 30 20:05:53.722295 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:53.722279 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" Oct 30 20:05:53.722458 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:53.722445 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:05:53.722601 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:53.722588 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 20:05:53.723439 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:53.723418 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" Oct 30 20:05:53.723855 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:53.723836 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:05:53.830850 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:53.830814 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cad5119-c2c5m" Oct 30 20:05:53.873248 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:53.873207 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 20:05:53.925690 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:53.900549 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92c93d36-4f3f-4670-a8f4-c8d2fd372cff-kube-api-access-bs4r8\" (UniqueName: \"kubernetes.io/projected/92c93d36-4f3f-4670-a8f4-c8d2fd372cff-kube-api-access-bs4r8\") pod \"92c93d36-4f3f-4670-a8f4-c8d2fd372cff\" (UID: \"92c93d36-4f3f-4670-a8f4-c8d2fd372cff\") " Oct 30 20:05:54.338420 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:54.323525 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92c93d36-4f3f-4670-a8f4-c8d2fd372cff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92c93d36-4f3f-4670-a8f4-c8d2fd372cff-maas-api-openshift-ca\") pod \"92c93d36-4f3f-4670-a8f4-c8d2fd372cff\" (UID: \"92c93d36-4f3f-4670-a8f4-c8d2fd372cff\") " Oct 30 20:05:54.338958 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:54.338921 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92c93d36-4f3f-4670-a8f4-c8d2fd372cff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92c93d36-4f3f-4670-a8f4-c8d2fd372cff-maas-api-xks-ca\") pod \"92c93d36-4f3f-4670-a8f4-c8d2fd372cff\" (UID: \"92c93d36-4f3f-4670-a8f4-c8d2fd372cff\") " Oct 30 20:05:54.366638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:54.314212 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:54.631686 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:54.370250 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": context deadline exceeded" Oct 30 20:05:54.631686 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:54.399683 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 20:05:54.631686 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:54.433246 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:54.631686 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:54.462399 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 30 20:05:54.631686 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:54.524192 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:54.671122 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:54.323443 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92c93d36-4f3f-4670-a8f4-c8d2fd372cff-kube-api-access-bs4r8" pod "92c93d36-4f3f-4670-a8f4-c8d2fd372cff" (UID: "92c93d36-4f3f-4670-a8f4-c8d2fd372cff"). InnerVolumeSpecName "kube-api-access-bs4r8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:05:54.678043 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:54.678002 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92c93d36-4f3f-4670-a8f4-c8d2fd372cff-maas-api-openshift-ca" pod "92c93d36-4f3f-4670-a8f4-c8d2fd372cff" (UID: "92c93d36-4f3f-4670-a8f4-c8d2fd372cff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:05:54.687637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:54.679323 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 20:05:54.895702 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:54.881869 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 20:05:55.206708 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:54.978944 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs4r8\" (UniqueName: \"kubernetes.io/projected/92c93d36-4f3f-4670-a8f4-c8d2fd372cff-kube-api-access-bs4r8\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:05:55.206708 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:55.187430 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92c93d36-4f3f-4670-a8f4-c8d2fd372cff-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:05:55.206708 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:55.126458 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:05:55.206708 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:55.187539 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" podUID="f4b44c6e-891a-4073-9192-035aeaad10a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:55.326664 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:55.323308 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:55.398431 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:55.398359 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92c93d36-4f3f-4670-a8f4-c8d2fd372cff-maas-api-xks-ca" pod "92c93d36-4f3f-4670-a8f4-c8d2fd372cff" (UID: "92c93d36-4f3f-4670-a8f4-c8d2fd372cff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:05:55.547665 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:55.536984 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92c93d36-4f3f-4670-a8f4-c8d2fd372cff-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:05:55.583088 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:55.583010 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:55.602138 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:55.602107 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cad5119-c2c5m" Oct 30 20:05:55.875696 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:55.791671 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:55.875696 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:55.820003 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 20:05:55.875696 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:55.820388 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 20:05:55.875696 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:55.820707 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 30 20:05:56.231690 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:05:56.210869 5088 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.232s" Oct 30 20:05:56.399668 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:56.397445 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cad5119-c2c5m" event={"ID":"92c93d36-4f3f-4670-a8f4-c8d2fd372cff","Type":"ContainerDied","Data":"cddd2d2d505c2c253e440360d00be83ab08a4fcac589cdaaf9f8fda8238978d5"} Oct 30 20:05:56.399668 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:56.397511 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cddd2d2d505c2c253e440360d00be83ab08a4fcac589cdaaf9f8fda8238978d5" Oct 30 20:05:56.399668 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:56.282148 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:56.399668 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:56.324486 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:56.613802 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:56.613761 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 30 20:05:56.617687 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:56.617648 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded" Oct 30 20:05:56.676678 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:56.675429 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:05:56.815859 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:56.815827 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" Oct 30 20:05:56.825237 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:56.824947 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 20:05:56.826559 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:56.826523 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:56.880291 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:56.880201 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:05:56.911347 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:56.905015 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" Oct 30 20:05:56.911726 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:56.911690 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:05:56.911928 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:56.911914 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 20:05:56.917794 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:05:56.917776 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" Oct 30 20:06:01.184379 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:01.184182 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acbac306-99gbd"] Oct 30 20:06:01.186387 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:01.186359 5088 cpu_manager.go:414] "containerMap: removing container" podUID="92c93d36-4f3f-4670-a8f4-c8d2fd372cff" containerName="revoke-keys" Oct 30 20:06:01.186581 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:01.186535 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92c93d36-4f3f-4670-a8f4-c8d2fd372cff" containerName="revoke-keys" Oct 30 20:06:01.187051 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:01.187033 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="92c93d36-4f3f-4670-a8f4-c8d2fd372cff" containerName="revoke-keys" Oct 30 20:06:01.192892 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:01.192872 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acbac306-99gbd" Oct 30 20:06:01.204812 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:01.204781 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acbac306-99gbd"] Oct 30 20:06:01.212270 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:01.212247 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:06:01.212425 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:01.212291 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:06:01.330349 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:01.330318 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5984058e-2af6-4a63-9c6d-00827d986a40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acbac306-99gbd\" (UID: \"5984058e-2af6-4a63-9c6d-00827d986a40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acbac306-99gbd" Oct 30 20:06:01.330512 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:01.330460 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtpvs\" (UniqueName: \"kubernetes.io/projected/5984058e-2af6-4a63-9c6d-00827d986a40-kube-api-access-mtpvs\") pod \"maas-api-revoke-sub-acbac306-99gbd\" (UID: \"5984058e-2af6-4a63-9c6d-00827d986a40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acbac306-99gbd" Oct 30 20:06:01.330660 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:01.330646 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5984058e-2af6-4a63-9c6d-00827d986a40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acbac306-99gbd\" (UID: \"5984058e-2af6-4a63-9c6d-00827d986a40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acbac306-99gbd" Oct 30 20:06:01.432154 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:01.432117 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtpvs\" (UniqueName: \"kubernetes.io/projected/5984058e-2af6-4a63-9c6d-00827d986a40-kube-api-access-mtpvs\") pod \"maas-api-revoke-sub-acbac306-99gbd\" (UID: \"5984058e-2af6-4a63-9c6d-00827d986a40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acbac306-99gbd" Oct 30 20:06:01.432323 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:01.432290 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5984058e-2af6-4a63-9c6d-00827d986a40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acbac306-99gbd\" (UID: \"5984058e-2af6-4a63-9c6d-00827d986a40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acbac306-99gbd" Oct 30 20:06:01.432414 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:01.432342 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5984058e-2af6-4a63-9c6d-00827d986a40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acbac306-99gbd\" (UID: \"5984058e-2af6-4a63-9c6d-00827d986a40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acbac306-99gbd" Oct 30 20:06:01.432967 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:01.432941 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5984058e-2af6-4a63-9c6d-00827d986a40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acbac306-99gbd\" (UID: \"5984058e-2af6-4a63-9c6d-00827d986a40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acbac306-99gbd" Oct 30 20:06:01.434919 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:01.434866 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5984058e-2af6-4a63-9c6d-00827d986a40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acbac306-99gbd\" (UID: \"5984058e-2af6-4a63-9c6d-00827d986a40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acbac306-99gbd" Oct 30 20:06:01.458310 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:01.458273 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtpvs\" (UniqueName: \"kubernetes.io/projected/5984058e-2af6-4a63-9c6d-00827d986a40-kube-api-access-mtpvs\") pod \"maas-api-revoke-sub-acbac306-99gbd\" (UID: \"5984058e-2af6-4a63-9c6d-00827d986a40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acbac306-99gbd" Oct 30 20:06:01.511896 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:01.511873 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acbac306-99gbd" Oct 30 20:06:01.863333 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:01.861600 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acbac306-99gbd"] Oct 30 20:06:01.876379 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:06:01.876171 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5984058e_2af6_4a63_9c6d_00827d986a40.slice/crio-f88632584f6c209f54b7f3da8c2b19722acdc9212aa00867d522941ad37f2719 WatchSource:0}: Error finding container f88632584f6c209f54b7f3da8c2b19722acdc9212aa00867d522941ad37f2719: Status 404 returned error can't find the container with id f88632584f6c209f54b7f3da8c2b19722acdc9212aa00867d522941ad37f2719 Oct 30 20:06:02.844236 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:02.844206 5088 generic.go:356] "Generic (PLEG): container finished" podID="5984058e-2af6-4a63-9c6d-00827d986a40" containerID="e177dca68f51f73c541f0e1faea273dc955ae463601e62ed7eacabe2e9d777e1" exitCode=0 Oct 30 20:06:02.844592 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:02.844281 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acbac306-99gbd" event={"ID":"5984058e-2af6-4a63-9c6d-00827d986a40","Type":"ContainerDied","Data":"e177dca68f51f73c541f0e1faea273dc955ae463601e62ed7eacabe2e9d777e1"} Oct 30 20:06:02.844592 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:02.844329 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acbac306-99gbd" event={"ID":"5984058e-2af6-4a63-9c6d-00827d986a40","Type":"ContainerStarted","Data":"f88632584f6c209f54b7f3da8c2b19722acdc9212aa00867d522941ad37f2719"} Oct 30 20:06:03.343934 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:03.343906 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd332a36-hxcw2"] Oct 30 20:06:03.348414 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:03.348396 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd332a36-hxcw2" Oct 30 20:06:03.393486 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:03.393459 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd332a36-hxcw2"] Oct 30 20:06:03.475352 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:03.475319 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f4eba44-f3cb-4dd7-8557-0310f51ddf26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd332a36-hxcw2\" (UID: \"5f4eba44-f3cb-4dd7-8557-0310f51ddf26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd332a36-hxcw2" Oct 30 20:06:03.475514 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:03.475368 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f4eba44-f3cb-4dd7-8557-0310f51ddf26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd332a36-hxcw2\" (UID: \"5f4eba44-f3cb-4dd7-8557-0310f51ddf26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd332a36-hxcw2" Oct 30 20:06:03.475653 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:03.475632 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcvg7\" (UniqueName: \"kubernetes.io/projected/5f4eba44-f3cb-4dd7-8557-0310f51ddf26-kube-api-access-lcvg7\") pod \"maas-api-revoke-sub-cd332a36-hxcw2\" (UID: \"5f4eba44-f3cb-4dd7-8557-0310f51ddf26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd332a36-hxcw2" Oct 30 20:06:03.576549 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:03.576518 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcvg7\" (UniqueName: \"kubernetes.io/projected/5f4eba44-f3cb-4dd7-8557-0310f51ddf26-kube-api-access-lcvg7\") pod \"maas-api-revoke-sub-cd332a36-hxcw2\" (UID: \"5f4eba44-f3cb-4dd7-8557-0310f51ddf26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd332a36-hxcw2" Oct 30 20:06:03.576773 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:03.576607 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f4eba44-f3cb-4dd7-8557-0310f51ddf26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd332a36-hxcw2\" (UID: \"5f4eba44-f3cb-4dd7-8557-0310f51ddf26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd332a36-hxcw2" Oct 30 20:06:03.576773 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:03.576662 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f4eba44-f3cb-4dd7-8557-0310f51ddf26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd332a36-hxcw2\" (UID: \"5f4eba44-f3cb-4dd7-8557-0310f51ddf26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd332a36-hxcw2" Oct 30 20:06:03.577331 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:03.577310 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f4eba44-f3cb-4dd7-8557-0310f51ddf26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd332a36-hxcw2\" (UID: \"5f4eba44-f3cb-4dd7-8557-0310f51ddf26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd332a36-hxcw2" Oct 30 20:06:03.578967 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:03.578944 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f4eba44-f3cb-4dd7-8557-0310f51ddf26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd332a36-hxcw2\" (UID: \"5f4eba44-f3cb-4dd7-8557-0310f51ddf26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd332a36-hxcw2" Oct 30 20:06:03.608206 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:03.608147 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcvg7\" (UniqueName: \"kubernetes.io/projected/5f4eba44-f3cb-4dd7-8557-0310f51ddf26-kube-api-access-lcvg7\") pod \"maas-api-revoke-sub-cd332a36-hxcw2\" (UID: \"5f4eba44-f3cb-4dd7-8557-0310f51ddf26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd332a36-hxcw2" Oct 30 20:06:03.678830 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:03.678791 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd332a36-hxcw2" Oct 30 20:06:04.046039 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:04.046012 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acbac306-99gbd" Oct 30 20:06:04.101146 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:06:04.101117 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f4eba44_f3cb_4dd7_8557_0310f51ddf26.slice/crio-317e2074ec95a47d0c0a7a45e795fbd09da17872c702f2b22887efa8ccc1c244 WatchSource:0}: Error finding container 317e2074ec95a47d0c0a7a45e795fbd09da17872c702f2b22887efa8ccc1c244: Status 404 returned error can't find the container with id 317e2074ec95a47d0c0a7a45e795fbd09da17872c702f2b22887efa8ccc1c244 Oct 30 20:06:04.102528 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:04.102505 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd332a36-hxcw2"] Oct 30 20:06:04.191488 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:04.191423 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5984058e-2af6-4a63-9c6d-00827d986a40-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5984058e-2af6-4a63-9c6d-00827d986a40-maas-api-xks-ca\") pod \"5984058e-2af6-4a63-9c6d-00827d986a40\" (UID: \"5984058e-2af6-4a63-9c6d-00827d986a40\") " Oct 30 20:06:04.191642 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:04.191599 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5984058e-2af6-4a63-9c6d-00827d986a40-kube-api-access-mtpvs\" (UniqueName: \"kubernetes.io/projected/5984058e-2af6-4a63-9c6d-00827d986a40-kube-api-access-mtpvs\") pod \"5984058e-2af6-4a63-9c6d-00827d986a40\" (UID: \"5984058e-2af6-4a63-9c6d-00827d986a40\") " Oct 30 20:06:04.191760 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:04.191745 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5984058e-2af6-4a63-9c6d-00827d986a40-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5984058e-2af6-4a63-9c6d-00827d986a40-maas-api-openshift-ca\") pod \"5984058e-2af6-4a63-9c6d-00827d986a40\" (UID: \"5984058e-2af6-4a63-9c6d-00827d986a40\") " Oct 30 20:06:04.192111 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:04.192088 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5984058e-2af6-4a63-9c6d-00827d986a40-maas-api-openshift-ca" pod "5984058e-2af6-4a63-9c6d-00827d986a40" (UID: "5984058e-2af6-4a63-9c6d-00827d986a40"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:06:04.192347 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:04.192332 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5984058e-2af6-4a63-9c6d-00827d986a40-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:06:04.193374 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:04.193357 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5984058e-2af6-4a63-9c6d-00827d986a40-maas-api-xks-ca" pod "5984058e-2af6-4a63-9c6d-00827d986a40" (UID: "5984058e-2af6-4a63-9c6d-00827d986a40"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:06:04.193639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:04.193600 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5984058e-2af6-4a63-9c6d-00827d986a40-kube-api-access-mtpvs" pod "5984058e-2af6-4a63-9c6d-00827d986a40" (UID: "5984058e-2af6-4a63-9c6d-00827d986a40"). InnerVolumeSpecName "kube-api-access-mtpvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:06:04.294215 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:04.294186 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5984058e-2af6-4a63-9c6d-00827d986a40-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:06:04.294215 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:04.294213 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtpvs\" (UniqueName: \"kubernetes.io/projected/5984058e-2af6-4a63-9c6d-00827d986a40-kube-api-access-mtpvs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:06:04.861011 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:04.860935 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acbac306-99gbd" Oct 30 20:06:04.861011 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:04.860966 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acbac306-99gbd" event={"ID":"5984058e-2af6-4a63-9c6d-00827d986a40","Type":"ContainerDied","Data":"f88632584f6c209f54b7f3da8c2b19722acdc9212aa00867d522941ad37f2719"} Oct 30 20:06:04.861238 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:04.861021 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f88632584f6c209f54b7f3da8c2b19722acdc9212aa00867d522941ad37f2719" Oct 30 20:06:04.862903 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:04.862876 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd332a36-hxcw2" event={"ID":"5f4eba44-f3cb-4dd7-8557-0310f51ddf26","Type":"ContainerStarted","Data":"34cec33ae3f1c9111edc95f0c34fbd8067eb728565379b352b63a49205dfa7b9"} Oct 30 20:06:04.863035 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:04.862925 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd332a36-hxcw2" event={"ID":"5f4eba44-f3cb-4dd7-8557-0310f51ddf26","Type":"ContainerStarted","Data":"317e2074ec95a47d0c0a7a45e795fbd09da17872c702f2b22887efa8ccc1c244"} Oct 30 20:06:04.913447 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:04.913324 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd332a36-hxcw2" podStartSLOduration=1.9133099759999999 podStartE2EDuration="1.913309976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:04.911895599 +0000 UTC m=+1126.324609379" watchObservedRunningTime="2026-10-30 20:06:04.913309976 +0000 UTC m=+1126.326023755" Oct 30 20:06:05.872231 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:05.872200 5088 generic.go:356] "Generic (PLEG): container finished" podID="5f4eba44-f3cb-4dd7-8557-0310f51ddf26" containerID="34cec33ae3f1c9111edc95f0c34fbd8067eb728565379b352b63a49205dfa7b9" exitCode=0 Oct 30 20:06:05.872666 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:05.872264 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd332a36-hxcw2" event={"ID":"5f4eba44-f3cb-4dd7-8557-0310f51ddf26","Type":"ContainerDied","Data":"34cec33ae3f1c9111edc95f0c34fbd8067eb728565379b352b63a49205dfa7b9"} Oct 30 20:06:07.072558 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:07.072532 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd332a36-hxcw2" Oct 30 20:06:07.132657 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:07.132595 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f4eba44-f3cb-4dd7-8557-0310f51ddf26-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f4eba44-f3cb-4dd7-8557-0310f51ddf26-maas-api-xks-ca\") pod \"5f4eba44-f3cb-4dd7-8557-0310f51ddf26\" (UID: \"5f4eba44-f3cb-4dd7-8557-0310f51ddf26\") " Oct 30 20:06:07.132814 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:07.132716 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f4eba44-f3cb-4dd7-8557-0310f51ddf26-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f4eba44-f3cb-4dd7-8557-0310f51ddf26-maas-api-openshift-ca\") pod \"5f4eba44-f3cb-4dd7-8557-0310f51ddf26\" (UID: \"5f4eba44-f3cb-4dd7-8557-0310f51ddf26\") " Oct 30 20:06:07.132909 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:07.132891 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f4eba44-f3cb-4dd7-8557-0310f51ddf26-kube-api-access-lcvg7\" (UniqueName: \"kubernetes.io/projected/5f4eba44-f3cb-4dd7-8557-0310f51ddf26-kube-api-access-lcvg7\") pod \"5f4eba44-f3cb-4dd7-8557-0310f51ddf26\" (UID: \"5f4eba44-f3cb-4dd7-8557-0310f51ddf26\") " Oct 30 20:06:07.133131 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:07.133104 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f4eba44-f3cb-4dd7-8557-0310f51ddf26-maas-api-openshift-ca" pod "5f4eba44-f3cb-4dd7-8557-0310f51ddf26" (UID: "5f4eba44-f3cb-4dd7-8557-0310f51ddf26"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:06:07.133596 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:07.133569 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f4eba44-f3cb-4dd7-8557-0310f51ddf26-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:06:07.134880 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:07.134839 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f4eba44-f3cb-4dd7-8557-0310f51ddf26-maas-api-xks-ca" pod "5f4eba44-f3cb-4dd7-8557-0310f51ddf26" (UID: "5f4eba44-f3cb-4dd7-8557-0310f51ddf26"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:06:07.135022 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:07.134991 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f4eba44-f3cb-4dd7-8557-0310f51ddf26-kube-api-access-lcvg7" pod "5f4eba44-f3cb-4dd7-8557-0310f51ddf26" (UID: "5f4eba44-f3cb-4dd7-8557-0310f51ddf26"). InnerVolumeSpecName "kube-api-access-lcvg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:06:07.235477 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:07.235443 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcvg7\" (UniqueName: \"kubernetes.io/projected/5f4eba44-f3cb-4dd7-8557-0310f51ddf26-kube-api-access-lcvg7\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:06:07.235477 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:07.235473 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f4eba44-f3cb-4dd7-8557-0310f51ddf26-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:06:07.901670 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:07.901642 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd332a36-hxcw2" Oct 30 20:06:07.901670 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:07.901671 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd332a36-hxcw2" event={"ID":"5f4eba44-f3cb-4dd7-8557-0310f51ddf26","Type":"ContainerDied","Data":"317e2074ec95a47d0c0a7a45e795fbd09da17872c702f2b22887efa8ccc1c244"} Oct 30 20:06:07.901903 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:07.901726 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="317e2074ec95a47d0c0a7a45e795fbd09da17872c702f2b22887efa8ccc1c244" Oct 30 20:06:10.248155 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.248121 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp"] Oct 30 20:06:10.248805 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.248789 5088 cpu_manager.go:414] "containerMap: removing container" podUID="5f4eba44-f3cb-4dd7-8557-0310f51ddf26" containerName="revoke-keys" Oct 30 20:06:10.248805 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.248805 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f4eba44-f3cb-4dd7-8557-0310f51ddf26" containerName="revoke-keys" Oct 30 20:06:10.248903 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.248817 5088 cpu_manager.go:414] "containerMap: removing container" podUID="5984058e-2af6-4a63-9c6d-00827d986a40" containerName="revoke-keys" Oct 30 20:06:10.248903 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.248823 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5984058e-2af6-4a63-9c6d-00827d986a40" containerName="revoke-keys" Oct 30 20:06:10.248966 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.248953 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f4eba44-f3cb-4dd7-8557-0310f51ddf26" containerName="revoke-keys" Oct 30 20:06:10.248966 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.248963 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="5984058e-2af6-4a63-9c6d-00827d986a40" containerName="revoke-keys" Oct 30 20:06:10.252774 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.252757 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" Oct 30 20:06:10.277058 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.277030 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8"] Oct 30 20:06:10.281842 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.281820 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" Oct 30 20:06:10.282442 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.282420 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qszrk\" (UniqueName: \"kubernetes.io/projected/14b2ad56-7181-420c-8ad1-bdc00f25e928-kube-api-access-qszrk\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp\" (UID: \"14b2ad56-7181-420c-8ad1-bdc00f25e928\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" Oct 30 20:06:10.282532 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.282491 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14b2ad56-7181-420c-8ad1-bdc00f25e928-istio-token\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp\" (UID: \"14b2ad56-7181-420c-8ad1-bdc00f25e928\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" Oct 30 20:06:10.282600 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.282585 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14b2ad56-7181-420c-8ad1-bdc00f25e928-istio-podinfo\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp\" (UID: \"14b2ad56-7181-420c-8ad1-bdc00f25e928\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" Oct 30 20:06:10.282691 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.282680 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14b2ad56-7181-420c-8ad1-bdc00f25e928-istiod-ca-cert\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp\" (UID: \"14b2ad56-7181-420c-8ad1-bdc00f25e928\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" Oct 30 20:06:10.282750 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.282734 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14b2ad56-7181-420c-8ad1-bdc00f25e928-istio-data\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp\" (UID: \"14b2ad56-7181-420c-8ad1-bdc00f25e928\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" Oct 30 20:06:10.282820 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.282779 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b2ad56-7181-420c-8ad1-bdc00f25e928-credential-socket\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp\" (UID: \"14b2ad56-7181-420c-8ad1-bdc00f25e928\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" Oct 30 20:06:10.282916 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.282900 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14b2ad56-7181-420c-8ad1-bdc00f25e928-workload-certs\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp\" (UID: \"14b2ad56-7181-420c-8ad1-bdc00f25e928\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" Oct 30 20:06:10.282974 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.282963 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b2ad56-7181-420c-8ad1-bdc00f25e928-workload-socket\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp\" (UID: \"14b2ad56-7181-420c-8ad1-bdc00f25e928\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" Oct 30 20:06:10.283074 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.283062 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14b2ad56-7181-420c-8ad1-bdc00f25e928-istio-envoy\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp\" (UID: \"14b2ad56-7181-420c-8ad1-bdc00f25e928\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" Oct 30 20:06:10.302568 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.302542 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp"] Oct 30 20:06:10.312049 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.312020 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8"] Oct 30 20:06:10.384981 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.384948 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14b2ad56-7181-420c-8ad1-bdc00f25e928-istiod-ca-cert\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp\" (UID: \"14b2ad56-7181-420c-8ad1-bdc00f25e928\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" Oct 30 20:06:10.385168 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.384995 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14b2ad56-7181-420c-8ad1-bdc00f25e928-istio-data\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp\" (UID: \"14b2ad56-7181-420c-8ad1-bdc00f25e928\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" Oct 30 20:06:10.385168 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.385043 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b2ad56-7181-420c-8ad1-bdc00f25e928-credential-socket\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp\" (UID: \"14b2ad56-7181-420c-8ad1-bdc00f25e928\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" Oct 30 20:06:10.385168 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.385143 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab112498-5d12-44e0-85b3-0c9f5dc003d6-istio-envoy\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8\" (UID: \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" Oct 30 20:06:10.385278 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.385177 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14b2ad56-7181-420c-8ad1-bdc00f25e928-workload-certs\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp\" (UID: \"14b2ad56-7181-420c-8ad1-bdc00f25e928\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" Oct 30 20:06:10.385318 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.385297 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b2ad56-7181-420c-8ad1-bdc00f25e928-workload-socket\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp\" (UID: \"14b2ad56-7181-420c-8ad1-bdc00f25e928\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" Oct 30 20:06:10.385398 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.385362 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab112498-5d12-44e0-85b3-0c9f5dc003d6-istio-data\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8\" (UID: \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" Oct 30 20:06:10.385474 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.385446 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14b2ad56-7181-420c-8ad1-bdc00f25e928-istio-envoy\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp\" (UID: \"14b2ad56-7181-420c-8ad1-bdc00f25e928\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" Oct 30 20:06:10.385705 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.385477 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab112498-5d12-44e0-85b3-0c9f5dc003d6-istio-token\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8\" (UID: \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" Oct 30 20:06:10.385705 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.385496 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14b2ad56-7181-420c-8ad1-bdc00f25e928-istio-data\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp\" (UID: \"14b2ad56-7181-420c-8ad1-bdc00f25e928\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" Oct 30 20:06:10.385705 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.385542 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b2ad56-7181-420c-8ad1-bdc00f25e928-credential-socket\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp\" (UID: \"14b2ad56-7181-420c-8ad1-bdc00f25e928\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" Oct 30 20:06:10.385705 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.385595 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14b2ad56-7181-420c-8ad1-bdc00f25e928-workload-certs\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp\" (UID: \"14b2ad56-7181-420c-8ad1-bdc00f25e928\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" Oct 30 20:06:10.385705 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.385632 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qszrk\" (UniqueName: \"kubernetes.io/projected/14b2ad56-7181-420c-8ad1-bdc00f25e928-kube-api-access-qszrk\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp\" (UID: \"14b2ad56-7181-420c-8ad1-bdc00f25e928\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" Oct 30 20:06:10.386001 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.385720 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b2ad56-7181-420c-8ad1-bdc00f25e928-workload-socket\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp\" (UID: \"14b2ad56-7181-420c-8ad1-bdc00f25e928\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" Oct 30 20:06:10.386001 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.385807 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14b2ad56-7181-420c-8ad1-bdc00f25e928-istiod-ca-cert\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp\" (UID: \"14b2ad56-7181-420c-8ad1-bdc00f25e928\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" Oct 30 20:06:10.386001 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.385847 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab112498-5d12-44e0-85b3-0c9f5dc003d6-istiod-ca-cert\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8\" (UID: \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" Oct 30 20:06:10.386001 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.385884 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab112498-5d12-44e0-85b3-0c9f5dc003d6-credential-socket\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8\" (UID: \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" Oct 30 20:06:10.386001 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.385984 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab112498-5d12-44e0-85b3-0c9f5dc003d6-workload-certs\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8\" (UID: \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" Oct 30 20:06:10.386263 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.386045 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab112498-5d12-44e0-85b3-0c9f5dc003d6-workload-socket\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8\" (UID: \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" Oct 30 20:06:10.386263 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.386085 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14b2ad56-7181-420c-8ad1-bdc00f25e928-istio-token\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp\" (UID: \"14b2ad56-7181-420c-8ad1-bdc00f25e928\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" Oct 30 20:06:10.386263 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.386167 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab112498-5d12-44e0-85b3-0c9f5dc003d6-istio-podinfo\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8\" (UID: \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" Oct 30 20:06:10.386263 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.386198 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8688v\" (UniqueName: \"kubernetes.io/projected/ab112498-5d12-44e0-85b3-0c9f5dc003d6-kube-api-access-8688v\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8\" (UID: \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" Oct 30 20:06:10.386263 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.386235 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14b2ad56-7181-420c-8ad1-bdc00f25e928-istio-podinfo\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp\" (UID: \"14b2ad56-7181-420c-8ad1-bdc00f25e928\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" Oct 30 20:06:10.387821 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.387796 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14b2ad56-7181-420c-8ad1-bdc00f25e928-istio-envoy\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp\" (UID: \"14b2ad56-7181-420c-8ad1-bdc00f25e928\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" Oct 30 20:06:10.388692 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.388667 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14b2ad56-7181-420c-8ad1-bdc00f25e928-istio-podinfo\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp\" (UID: \"14b2ad56-7181-420c-8ad1-bdc00f25e928\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" Oct 30 20:06:10.412113 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.412086 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qszrk\" (UniqueName: \"kubernetes.io/projected/14b2ad56-7181-420c-8ad1-bdc00f25e928-kube-api-access-qszrk\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp\" (UID: \"14b2ad56-7181-420c-8ad1-bdc00f25e928\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" Oct 30 20:06:10.418713 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.418690 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14b2ad56-7181-420c-8ad1-bdc00f25e928-istio-token\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp\" (UID: \"14b2ad56-7181-420c-8ad1-bdc00f25e928\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" Oct 30 20:06:10.438368 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.438346 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2"] Oct 30 20:06:10.443907 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.443889 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" Oct 30 20:06:10.488668 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.487538 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2"] Oct 30 20:06:10.488668 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.487983 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab112498-5d12-44e0-85b3-0c9f5dc003d6-istio-token\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8\" (UID: \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" Oct 30 20:06:10.488668 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.488066 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51889ad-a7b6-41f2-9857-679ac04b7bea-workload-socket\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2\" (UID: \"a51889ad-a7b6-41f2-9857-679ac04b7bea\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" Oct 30 20:06:10.488668 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.488091 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a51889ad-a7b6-41f2-9857-679ac04b7bea-istio-data\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2\" (UID: \"a51889ad-a7b6-41f2-9857-679ac04b7bea\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" Oct 30 20:06:10.488668 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.488120 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab112498-5d12-44e0-85b3-0c9f5dc003d6-istiod-ca-cert\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8\" (UID: \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" Oct 30 20:06:10.488668 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.488149 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab112498-5d12-44e0-85b3-0c9f5dc003d6-credential-socket\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8\" (UID: \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" Oct 30 20:06:10.488668 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.488177 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a51889ad-a7b6-41f2-9857-679ac04b7bea-workload-certs\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2\" (UID: \"a51889ad-a7b6-41f2-9857-679ac04b7bea\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" Oct 30 20:06:10.488668 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.488211 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab112498-5d12-44e0-85b3-0c9f5dc003d6-workload-certs\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8\" (UID: \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" Oct 30 20:06:10.488668 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.488663 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab112498-5d12-44e0-85b3-0c9f5dc003d6-workload-certs\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8\" (UID: \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" Oct 30 20:06:10.489334 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.489310 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab112498-5d12-44e0-85b3-0c9f5dc003d6-istiod-ca-cert\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8\" (UID: \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" Oct 30 20:06:10.489659 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.489632 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab112498-5d12-44e0-85b3-0c9f5dc003d6-credential-socket\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8\" (UID: \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" Oct 30 20:06:10.489865 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.489844 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a51889ad-a7b6-41f2-9857-679ac04b7bea-istio-podinfo\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2\" (UID: \"a51889ad-a7b6-41f2-9857-679ac04b7bea\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" Oct 30 20:06:10.489977 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.489890 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab112498-5d12-44e0-85b3-0c9f5dc003d6-workload-socket\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8\" (UID: \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" Oct 30 20:06:10.490059 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.489994 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab112498-5d12-44e0-85b3-0c9f5dc003d6-istio-podinfo\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8\" (UID: \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" Oct 30 20:06:10.490059 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.490037 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8688v\" (UniqueName: \"kubernetes.io/projected/ab112498-5d12-44e0-85b3-0c9f5dc003d6-kube-api-access-8688v\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8\" (UID: \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" Oct 30 20:06:10.490176 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.490086 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51889ad-a7b6-41f2-9857-679ac04b7bea-credential-socket\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2\" (UID: \"a51889ad-a7b6-41f2-9857-679ac04b7bea\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" Oct 30 20:06:10.490176 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.490127 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a51889ad-a7b6-41f2-9857-679ac04b7bea-istio-envoy\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2\" (UID: \"a51889ad-a7b6-41f2-9857-679ac04b7bea\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" Oct 30 20:06:10.490176 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.490154 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a51889ad-a7b6-41f2-9857-679ac04b7bea-istiod-ca-cert\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2\" (UID: \"a51889ad-a7b6-41f2-9857-679ac04b7bea\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" Oct 30 20:06:10.490333 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.490241 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab112498-5d12-44e0-85b3-0c9f5dc003d6-workload-socket\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8\" (UID: \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" Oct 30 20:06:10.490475 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.490443 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab112498-5d12-44e0-85b3-0c9f5dc003d6-istio-envoy\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8\" (UID: \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" Oct 30 20:06:10.490583 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.490529 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm29m\" (UniqueName: \"kubernetes.io/projected/a51889ad-a7b6-41f2-9857-679ac04b7bea-kube-api-access-rm29m\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2\" (UID: \"a51889ad-a7b6-41f2-9857-679ac04b7bea\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" Oct 30 20:06:10.490583 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.490555 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a51889ad-a7b6-41f2-9857-679ac04b7bea-istio-token\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2\" (UID: \"a51889ad-a7b6-41f2-9857-679ac04b7bea\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" Oct 30 20:06:10.490750 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.490582 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab112498-5d12-44e0-85b3-0c9f5dc003d6-istio-data\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8\" (UID: \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" Oct 30 20:06:10.490927 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.490903 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab112498-5d12-44e0-85b3-0c9f5dc003d6-istio-data\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8\" (UID: \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" Oct 30 20:06:10.492792 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.492767 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab112498-5d12-44e0-85b3-0c9f5dc003d6-istio-envoy\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8\" (UID: \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" Oct 30 20:06:10.493758 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.493730 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab112498-5d12-44e0-85b3-0c9f5dc003d6-istio-podinfo\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8\" (UID: \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" Oct 30 20:06:10.521025 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.520966 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab112498-5d12-44e0-85b3-0c9f5dc003d6-istio-token\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8\" (UID: \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" Oct 30 20:06:10.521159 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.521142 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8688v\" (UniqueName: \"kubernetes.io/projected/ab112498-5d12-44e0-85b3-0c9f5dc003d6-kube-api-access-8688v\") pod \"e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8\" (UID: \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\") " pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" Oct 30 20:06:10.566945 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.566920 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" Oct 30 20:06:10.592798 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.592768 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51889ad-a7b6-41f2-9857-679ac04b7bea-workload-socket\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2\" (UID: \"a51889ad-a7b6-41f2-9857-679ac04b7bea\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" Oct 30 20:06:10.592918 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.592811 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a51889ad-a7b6-41f2-9857-679ac04b7bea-istio-data\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2\" (UID: \"a51889ad-a7b6-41f2-9857-679ac04b7bea\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" Oct 30 20:06:10.592918 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.592849 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a51889ad-a7b6-41f2-9857-679ac04b7bea-workload-certs\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2\" (UID: \"a51889ad-a7b6-41f2-9857-679ac04b7bea\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" Oct 30 20:06:10.592918 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.592902 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a51889ad-a7b6-41f2-9857-679ac04b7bea-istio-podinfo\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2\" (UID: \"a51889ad-a7b6-41f2-9857-679ac04b7bea\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" Oct 30 20:06:10.593085 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.592994 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51889ad-a7b6-41f2-9857-679ac04b7bea-credential-socket\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2\" (UID: \"a51889ad-a7b6-41f2-9857-679ac04b7bea\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" Oct 30 20:06:10.593085 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.593029 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a51889ad-a7b6-41f2-9857-679ac04b7bea-istio-envoy\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2\" (UID: \"a51889ad-a7b6-41f2-9857-679ac04b7bea\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" Oct 30 20:06:10.593085 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.593060 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a51889ad-a7b6-41f2-9857-679ac04b7bea-istiod-ca-cert\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2\" (UID: \"a51889ad-a7b6-41f2-9857-679ac04b7bea\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" Oct 30 20:06:10.593234 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.593143 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a51889ad-a7b6-41f2-9857-679ac04b7bea-istio-data\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2\" (UID: \"a51889ad-a7b6-41f2-9857-679ac04b7bea\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" Oct 30 20:06:10.593288 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.593248 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51889ad-a7b6-41f2-9857-679ac04b7bea-workload-socket\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2\" (UID: \"a51889ad-a7b6-41f2-9857-679ac04b7bea\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" Oct 30 20:06:10.593288 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.593275 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm29m\" (UniqueName: \"kubernetes.io/projected/a51889ad-a7b6-41f2-9857-679ac04b7bea-kube-api-access-rm29m\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2\" (UID: \"a51889ad-a7b6-41f2-9857-679ac04b7bea\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" Oct 30 20:06:10.593392 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.593301 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a51889ad-a7b6-41f2-9857-679ac04b7bea-istio-token\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2\" (UID: \"a51889ad-a7b6-41f2-9857-679ac04b7bea\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" Oct 30 20:06:10.593880 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.593842 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a51889ad-a7b6-41f2-9857-679ac04b7bea-istiod-ca-cert\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2\" (UID: \"a51889ad-a7b6-41f2-9857-679ac04b7bea\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" Oct 30 20:06:10.593880 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.593868 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51889ad-a7b6-41f2-9857-679ac04b7bea-credential-socket\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2\" (UID: \"a51889ad-a7b6-41f2-9857-679ac04b7bea\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" Oct 30 20:06:10.594102 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.594085 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a51889ad-a7b6-41f2-9857-679ac04b7bea-workload-certs\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2\" (UID: \"a51889ad-a7b6-41f2-9857-679ac04b7bea\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" Oct 30 20:06:10.596006 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.595980 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a51889ad-a7b6-41f2-9857-679ac04b7bea-istio-podinfo\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2\" (UID: \"a51889ad-a7b6-41f2-9857-679ac04b7bea\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" Oct 30 20:06:10.596101 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.596060 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a51889ad-a7b6-41f2-9857-679ac04b7bea-istio-envoy\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2\" (UID: \"a51889ad-a7b6-41f2-9857-679ac04b7bea\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" Oct 30 20:06:10.600473 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.600449 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" Oct 30 20:06:10.624719 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.623790 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a51889ad-a7b6-41f2-9857-679ac04b7bea-istio-token\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2\" (UID: \"a51889ad-a7b6-41f2-9857-679ac04b7bea\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" Oct 30 20:06:10.627648 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.626038 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm29m\" (UniqueName: \"kubernetes.io/projected/a51889ad-a7b6-41f2-9857-679ac04b7bea-kube-api-access-rm29m\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2\" (UID: \"a51889ad-a7b6-41f2-9857-679ac04b7bea\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" Oct 30 20:06:10.761129 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.761098 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" Oct 30 20:06:10.918019 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.917991 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp"] Oct 30 20:06:10.919037 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:06:10.919009 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14b2ad56_7181_420c_8ad1_bdc00f25e928.slice/crio-1993c6ed548f07135ac27b4feba9506113207bfe9eeb62258dddd5da744eb82e WatchSource:0}: Error finding container 1993c6ed548f07135ac27b4feba9506113207bfe9eeb62258dddd5da744eb82e: Status 404 returned error can't find the container with id 1993c6ed548f07135ac27b4feba9506113207bfe9eeb62258dddd5da744eb82e Oct 30 20:06:10.923004 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.922971 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:06:10.923098 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.923055 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:06:10.923148 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.923096 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:06:10.930998 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.930972 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw"] Oct 30 20:06:10.936877 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.936855 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" Oct 30 20:06:10.972114 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.972063 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8"] Oct 30 20:06:10.974450 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:06:10.974422 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab112498_5d12_44e0_85b3_0c9f5dc003d6.slice/crio-4c1ce0dee5c3b0e7b2d4a265a2f55533135b4b148f6ae3808f6aa98729c11006 WatchSource:0}: Error finding container 4c1ce0dee5c3b0e7b2d4a265a2f55533135b4b148f6ae3808f6aa98729c11006: Status 404 returned error can't find the container with id 4c1ce0dee5c3b0e7b2d4a265a2f55533135b4b148f6ae3808f6aa98729c11006 Oct 30 20:06:10.977681 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.977646 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:06:10.977766 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.977731 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:06:10.977820 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.977773 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:06:10.980540 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:10.980516 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw"] Oct 30 20:06:11.004185 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:11.004151 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baca0e89-a102-4354-af54-c14ccc418923-workload-socket\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw\" (UID: \"baca0e89-a102-4354-af54-c14ccc418923\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" Oct 30 20:06:11.021796 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:11.021732 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baca0e89-a102-4354-af54-c14ccc418923-istiod-ca-cert\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw\" (UID: \"baca0e89-a102-4354-af54-c14ccc418923\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" Oct 30 20:06:11.022319 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:11.022290 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baca0e89-a102-4354-af54-c14ccc418923-istio-podinfo\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw\" (UID: \"baca0e89-a102-4354-af54-c14ccc418923\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" Oct 30 20:06:11.022402 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:11.022346 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrf7k\" (UniqueName: \"kubernetes.io/projected/baca0e89-a102-4354-af54-c14ccc418923-kube-api-access-nrf7k\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw\" (UID: \"baca0e89-a102-4354-af54-c14ccc418923\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" Oct 30 20:06:11.022462 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:11.022422 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baca0e89-a102-4354-af54-c14ccc418923-istio-envoy\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw\" (UID: \"baca0e89-a102-4354-af54-c14ccc418923\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" Oct 30 20:06:11.022572 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:11.022551 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baca0e89-a102-4354-af54-c14ccc418923-istio-token\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw\" (UID: \"baca0e89-a102-4354-af54-c14ccc418923\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" Oct 30 20:06:11.022688 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:11.022605 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baca0e89-a102-4354-af54-c14ccc418923-workload-certs\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw\" (UID: \"baca0e89-a102-4354-af54-c14ccc418923\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" Oct 30 20:06:11.022814 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:11.022798 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baca0e89-a102-4354-af54-c14ccc418923-istio-data\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw\" (UID: \"baca0e89-a102-4354-af54-c14ccc418923\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" Oct 30 20:06:11.022896 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:11.022882 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baca0e89-a102-4354-af54-c14ccc418923-credential-socket\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw\" (UID: \"baca0e89-a102-4354-af54-c14ccc418923\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" Oct 30 20:06:11.103553 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:11.103520 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2"] Oct 30 20:06:11.115099 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:11.114996 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:06:11.116151 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:11.116126 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:06:11.116344 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:11.116327 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:06:11.125009 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:11.124987 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baca0e89-a102-4354-af54-c14ccc418923-workload-certs\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw\" (UID: \"baca0e89-a102-4354-af54-c14ccc418923\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" Oct 30 20:06:11.125213 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:11.125199 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baca0e89-a102-4354-af54-c14ccc418923-istio-data\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw\" (UID: \"baca0e89-a102-4354-af54-c14ccc418923\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" Oct 30 20:06:11.125323 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:11.125312 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baca0e89-a102-4354-af54-c14ccc418923-credential-socket\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw\" (UID: \"baca0e89-a102-4354-af54-c14ccc418923\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" Oct 30 20:06:11.125447 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:11.125436 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baca0e89-a102-4354-af54-c14ccc418923-workload-socket\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw\" (UID: \"baca0e89-a102-4354-af54-c14ccc418923\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" Oct 30 20:06:11.125513 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:11.125505 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baca0e89-a102-4354-af54-c14ccc418923-istiod-ca-cert\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw\" (UID: \"baca0e89-a102-4354-af54-c14ccc418923\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" Oct 30 20:06:11.125696 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:11.125683 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baca0e89-a102-4354-af54-c14ccc418923-istio-podinfo\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw\" (UID: \"baca0e89-a102-4354-af54-c14ccc418923\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" Oct 30 20:06:11.125771 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:11.125762 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrf7k\" (UniqueName: \"kubernetes.io/projected/baca0e89-a102-4354-af54-c14ccc418923-kube-api-access-nrf7k\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw\" (UID: \"baca0e89-a102-4354-af54-c14ccc418923\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" Oct 30 20:06:11.125902 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:11.125887 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baca0e89-a102-4354-af54-c14ccc418923-istio-envoy\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw\" (UID: \"baca0e89-a102-4354-af54-c14ccc418923\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" Oct 30 20:06:11.125991 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:11.125983 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baca0e89-a102-4354-af54-c14ccc418923-istio-token\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw\" (UID: \"baca0e89-a102-4354-af54-c14ccc418923\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" Oct 30 20:06:11.127046 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:11.127024 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baca0e89-a102-4354-af54-c14ccc418923-workload-certs\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw\" (UID: \"baca0e89-a102-4354-af54-c14ccc418923\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" Oct 30 20:06:11.127688 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:11.127663 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baca0e89-a102-4354-af54-c14ccc418923-istiod-ca-cert\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw\" (UID: \"baca0e89-a102-4354-af54-c14ccc418923\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" Oct 30 20:06:11.129032 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:11.128074 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baca0e89-a102-4354-af54-c14ccc418923-credential-socket\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw\" (UID: \"baca0e89-a102-4354-af54-c14ccc418923\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" Oct 30 20:06:11.129220 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:11.128289 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baca0e89-a102-4354-af54-c14ccc418923-istio-data\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw\" (UID: \"baca0e89-a102-4354-af54-c14ccc418923\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" Oct 30 20:06:11.129320 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:11.128455 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baca0e89-a102-4354-af54-c14ccc418923-workload-socket\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw\" (UID: \"baca0e89-a102-4354-af54-c14ccc418923\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" Oct 30 20:06:11.131549 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:11.131507 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baca0e89-a102-4354-af54-c14ccc418923-istio-podinfo\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw\" (UID: \"baca0e89-a102-4354-af54-c14ccc418923\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" Oct 30 20:06:11.137942 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:11.137920 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baca0e89-a102-4354-af54-c14ccc418923-istio-envoy\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw\" (UID: \"baca0e89-a102-4354-af54-c14ccc418923\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" Oct 30 20:06:11.163924 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:11.163461 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baca0e89-a102-4354-af54-c14ccc418923-istio-token\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw\" (UID: \"baca0e89-a102-4354-af54-c14ccc418923\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" Oct 30 20:06:11.165674 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:11.165596 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrf7k\" (UniqueName: \"kubernetes.io/projected/baca0e89-a102-4354-af54-c14ccc418923-kube-api-access-nrf7k\") pod \"e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw\" (UID: \"baca0e89-a102-4354-af54-c14ccc418923\") " pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" Oct 30 20:06:11.272025 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:11.271981 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" Oct 30 20:06:11.714924 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:11.714900 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw"] Oct 30 20:06:11.715061 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:06:11.715042 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaca0e89_a102_4354_af54_c14ccc418923.slice/crio-f525517430be53ab0dd22245442935ae843a8330231eee8fabda7a6331ac36a7 WatchSource:0}: Error finding container f525517430be53ab0dd22245442935ae843a8330231eee8fabda7a6331ac36a7: Status 404 returned error can't find the container with id f525517430be53ab0dd22245442935ae843a8330231eee8fabda7a6331ac36a7 Oct 30 20:06:11.718259 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:11.718226 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:06:11.718340 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:11.718298 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:06:11.718340 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:11.718328 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:06:11.950431 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:11.944127 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" event={"ID":"baca0e89-a102-4354-af54-c14ccc418923","Type":"ContainerStarted","Data":"945cf965ae8ee1f5738db7747631301dc305f411de087b51d4c826da927c53a1"} Oct 30 20:06:11.951224 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:11.950717 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" event={"ID":"baca0e89-a102-4354-af54-c14ccc418923","Type":"ContainerStarted","Data":"f525517430be53ab0dd22245442935ae843a8330231eee8fabda7a6331ac36a7"} Oct 30 20:06:11.951512 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:11.951494 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" event={"ID":"14b2ad56-7181-420c-8ad1-bdc00f25e928","Type":"ContainerStarted","Data":"4358d77f290648c02af2f9a40e5a7ed64a0a45b4e236f0c73a7c725314a93bd1"} Oct 30 20:06:11.951726 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:11.951712 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" event={"ID":"14b2ad56-7181-420c-8ad1-bdc00f25e928","Type":"ContainerStarted","Data":"1993c6ed548f07135ac27b4feba9506113207bfe9eeb62258dddd5da744eb82e"} Oct 30 20:06:11.952217 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:11.952195 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" event={"ID":"a51889ad-a7b6-41f2-9857-679ac04b7bea","Type":"ContainerStarted","Data":"d9162f37e426872971fc72411af9413abb14758a4fd7f0a4977b158942bc274f"} Oct 30 20:06:11.952518 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:11.952503 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" event={"ID":"a51889ad-a7b6-41f2-9857-679ac04b7bea","Type":"ContainerStarted","Data":"87ea412953d2a90266ea9dccc100fff17edf29ac885beca0537e6f9347b70ec5"} Oct 30 20:06:11.955337 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:11.955315 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" event={"ID":"ab112498-5d12-44e0-85b3-0c9f5dc003d6","Type":"ContainerStarted","Data":"e68213262edc0b90dd827831d6b42294e4f6c2e96e19943a2dbf1a2894f0065b"} Oct 30 20:06:11.955677 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:11.955660 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" event={"ID":"ab112498-5d12-44e0-85b3-0c9f5dc003d6","Type":"ContainerStarted","Data":"4c1ce0dee5c3b0e7b2d4a265a2f55533135b4b148f6ae3808f6aa98729c11006"} Oct 30 20:06:12.022649 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:12.012350 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" podStartSLOduration=2.012328382 podStartE2EDuration="2.012328382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:12.003315225 +0000 UTC m=+1133.416029033" watchObservedRunningTime="2026-10-30 20:06:12.012328382 +0000 UTC m=+1133.425042166" Oct 30 20:06:12.151649 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:12.138687 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" podStartSLOduration=2.138666464 podStartE2EDuration="2.138666464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:12.079147196 +0000 UTC m=+1133.491860980" watchObservedRunningTime="2026-10-30 20:06:12.138666464 +0000 UTC m=+1133.551380244" Oct 30 20:06:12.179470 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:12.179406 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" podStartSLOduration=2.1793906769999998 podStartE2EDuration="2.179390677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:12.137906855 +0000 UTC m=+1133.550620661" watchObservedRunningTime="2026-10-30 20:06:12.179390677 +0000 UTC m=+1133.592104458" Oct 30 20:06:12.238638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:12.234160 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" podStartSLOduration=2.234139593 podStartE2EDuration="2.234139593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:12.19932292 +0000 UTC m=+1133.612036726" watchObservedRunningTime="2026-10-30 20:06:12.234139593 +0000 UTC m=+1133.646853376" Oct 30 20:06:12.274816 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:12.274722 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" Oct 30 20:06:12.275445 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:12.275419 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 20:06:12.275677 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:12.275586 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" podUID="baca0e89-a102-4354-af54-c14ccc418923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 20:06:12.567685 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:12.567569 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" Oct 30 20:06:12.568873 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:12.568849 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 20:06:12.568994 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:12.568892 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" podUID="14b2ad56-7181-420c-8ad1-bdc00f25e928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 20:06:12.611031 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:12.610997 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" Oct 30 20:06:12.614002 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:12.613970 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 20:06:12.614229 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:12.614204 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" podUID="ab112498-5d12-44e0-85b3-0c9f5dc003d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 20:06:12.775847 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:12.775814 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" Oct 30 20:06:12.776023 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:12.775912 5088 patch_prober.go:29] interesting pod/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 20:06:12.776023 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:12.775944 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" podUID="a51889ad-a7b6-41f2-9857-679ac04b7bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 20:06:13.274077 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:13.274036 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 20:06:13.274357 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:13.274327 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" podUID="baca0e89-a102-4354-af54-c14ccc418923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 20:06:13.573175 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:13.573082 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 20:06:13.573771 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:13.573740 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" podUID="14b2ad56-7181-420c-8ad1-bdc00f25e928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 20:06:13.616292 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:13.616257 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 20:06:13.616572 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:13.616543 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" podUID="ab112498-5d12-44e0-85b3-0c9f5dc003d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 20:06:13.763041 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:13.763003 5088 patch_prober.go:29] interesting pod/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 20:06:13.763359 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:13.763327 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" podUID="a51889ad-a7b6-41f2-9857-679ac04b7bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 20:06:13.886812 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:13.886709 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 20:06:13.945794 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:13.945736 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:14.126063 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:14.125760 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:14.275636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:14.275054 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 20:06:14.275636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:14.275131 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" podUID="baca0e89-a102-4354-af54-c14ccc418923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 20:06:14.568695 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:14.567969 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 20:06:14.568695 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:14.568032 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" podUID="14b2ad56-7181-420c-8ad1-bdc00f25e928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 20:06:14.595884 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:14.595842 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:14.626637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:14.621239 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 20:06:14.627355 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:14.627324 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 20:06:14.627527 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:14.627503 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" podUID="ab112498-5d12-44e0-85b3-0c9f5dc003d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 20:06:14.801897 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:14.801849 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 20:06:14.802545 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:14.802512 5088 patch_prober.go:29] interesting pod/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 20:06:14.802688 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:14.802574 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" podUID="a51889ad-a7b6-41f2-9857-679ac04b7bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 20:06:14.840257 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:14.840137 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:14.907638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:14.897835 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:14.928642 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:14.927715 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:14.942142 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:14.940875 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" Oct 30 20:06:15.590640 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:15.588098 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 20:06:15.590640 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:15.588158 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" podUID="baca0e89-a102-4354-af54-c14ccc418923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 20:06:15.590640 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:15.589933 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 20:06:15.590640 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:15.590068 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" podUID="14b2ad56-7181-420c-8ad1-bdc00f25e928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 20:06:15.600632 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:15.596684 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 30 20:06:15.600632 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:15.597019 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" Oct 30 20:06:15.602757 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:15.602733 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 20:06:15.602949 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:15.602926 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" podUID="ab112498-5d12-44e0-85b3-0c9f5dc003d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 20:06:15.642759 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:15.642714 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 20:06:15.643408 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:15.643384 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:06:15.818645 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:15.803472 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 20:06:15.819353 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:15.819319 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" Oct 30 20:06:15.849535 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:15.849447 5088 patch_prober.go:29] interesting pod/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 20:06:15.849784 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:15.849755 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" podUID="a51889ad-a7b6-41f2-9857-679ac04b7bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 20:06:15.948578 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:15.901780 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:15.948578 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:15.902135 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 30 20:06:15.948578 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:15.902427 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 20:06:15.964792 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:15.964752 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:15.974697 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:15.965366 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:06:16.328639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:16.316120 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 20:06:16.328639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:16.316179 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" podUID="baca0e89-a102-4354-af54-c14ccc418923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 20:06:16.630826 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:16.573268 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 20:06:16.630826 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:16.573333 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" podUID="14b2ad56-7181-420c-8ad1-bdc00f25e928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 20:06:16.639635 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:16.636379 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 30 20:06:16.651638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:16.641745 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 20:06:16.651638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:16.641795 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" podUID="ab112498-5d12-44e0-85b3-0c9f5dc003d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 20:06:16.651638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:16.642342 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded" Oct 30 20:06:16.682639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:16.652480 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 20:06:16.703320 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:16.703277 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:16.834381 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:16.833933 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:16.898700 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:16.895252 5088 patch_prober.go:29] interesting pod/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 20:06:16.898700 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:16.895317 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" podUID="a51889ad-a7b6-41f2-9857-679ac04b7bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 20:06:16.906457 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:16.905723 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:16.906646 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:16.906499 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 20:06:16.973648 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:16.967133 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:17.104647 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:17.090661 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-470dd631-q8tgv"] Oct 30 20:06:17.123165 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:17.123131 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-470dd631-q8tgv" Oct 30 20:06:17.147636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:17.145997 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:06:17.147636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:17.146692 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:06:17.175689 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:17.175514 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97f8f58f-a1ca-4489-9b23-d065cf67d9ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-470dd631-q8tgv\" (UID: \"97f8f58f-a1ca-4489-9b23-d065cf67d9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-470dd631-q8tgv" Oct 30 20:06:17.175856 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:17.175742 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnmpg\" (UniqueName: \"kubernetes.io/projected/97f8f58f-a1ca-4489-9b23-d065cf67d9ef-kube-api-access-pnmpg\") pod \"maas-api-revoke-sub-470dd631-q8tgv\" (UID: \"97f8f58f-a1ca-4489-9b23-d065cf67d9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-470dd631-q8tgv" Oct 30 20:06:17.176644 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:17.176110 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97f8f58f-a1ca-4489-9b23-d065cf67d9ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-470dd631-q8tgv\" (UID: \"97f8f58f-a1ca-4489-9b23-d065cf67d9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-470dd631-q8tgv" Oct 30 20:06:17.203191 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:17.201906 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-470dd631-q8tgv"] Oct 30 20:06:17.316634 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:17.277345 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnmpg\" (UniqueName: \"kubernetes.io/projected/97f8f58f-a1ca-4489-9b23-d065cf67d9ef-kube-api-access-pnmpg\") pod \"maas-api-revoke-sub-470dd631-q8tgv\" (UID: \"97f8f58f-a1ca-4489-9b23-d065cf67d9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-470dd631-q8tgv" Oct 30 20:06:17.316634 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:17.277574 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97f8f58f-a1ca-4489-9b23-d065cf67d9ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-470dd631-q8tgv\" (UID: \"97f8f58f-a1ca-4489-9b23-d065cf67d9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-470dd631-q8tgv" Oct 30 20:06:17.316634 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:17.277650 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97f8f58f-a1ca-4489-9b23-d065cf67d9ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-470dd631-q8tgv\" (UID: \"97f8f58f-a1ca-4489-9b23-d065cf67d9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-470dd631-q8tgv" Oct 30 20:06:17.316634 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:17.278584 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97f8f58f-a1ca-4489-9b23-d065cf67d9ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-470dd631-q8tgv\" (UID: \"97f8f58f-a1ca-4489-9b23-d065cf67d9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-470dd631-q8tgv" Oct 30 20:06:17.336637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:17.317570 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 20:06:17.336943 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:17.336905 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" podUID="baca0e89-a102-4354-af54-c14ccc418923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 20:06:17.592649 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:17.585324 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnmpg\" (UniqueName: \"kubernetes.io/projected/97f8f58f-a1ca-4489-9b23-d065cf67d9ef-kube-api-access-pnmpg\") pod \"maas-api-revoke-sub-470dd631-q8tgv\" (UID: \"97f8f58f-a1ca-4489-9b23-d065cf67d9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-470dd631-q8tgv" Oct 30 20:06:17.593015 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:17.592986 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 20:06:17.593208 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:17.593185 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" podUID="14b2ad56-7181-420c-8ad1-bdc00f25e928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 20:06:17.664636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:17.632955 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 20:06:17.664636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:17.633018 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" podUID="ab112498-5d12-44e0-85b3-0c9f5dc003d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 20:06:17.664636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:17.640692 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:17.702133 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:17.700089 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 20:06:17.729638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:17.706932 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:17.729638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:17.707330 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:06:17.774655 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:17.761259 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97f8f58f-a1ca-4489-9b23-d065cf67d9ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-470dd631-q8tgv\" (UID: \"97f8f58f-a1ca-4489-9b23-d065cf67d9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-470dd631-q8tgv" Oct 30 20:06:17.796952 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:17.796918 5088 patch_prober.go:29] interesting pod/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 20:06:17.952730 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:17.951029 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" podUID="a51889ad-a7b6-41f2-9857-679ac04b7bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 20:06:17.952730 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:17.835415 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-470dd631-q8tgv" Oct 30 20:06:17.954163 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:17.928919 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 30 20:06:17.954163 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:17.928962 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 20:06:17.954163 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:17.835464 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:18.086088 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:18.048605 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:18.469818 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:18.415016 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 20:06:18.469818 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:18.415069 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" podUID="baca0e89-a102-4354-af54-c14ccc418923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 20:06:18.701443 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:18.666745 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:18.715960 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:18.701403 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:18.725696 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:18.716258 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:18.815961 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:18.676948 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 20:06:18.815961 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:18.812833 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" podUID="ab112498-5d12-44e0-85b3-0c9f5dc003d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 20:06:18.815961 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:18.698730 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 20:06:18.815961 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:18.812904 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" podUID="14b2ad56-7181-420c-8ad1-bdc00f25e928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 20:06:18.815961 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:18.787973 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:19.040701 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:18.961433 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 20:06:19.040701 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:18.963909 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:19.086479 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:19.086133 5088 patch_prober.go:29] interesting pod/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 20:06:19.086479 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:19.086196 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" podUID="a51889ad-a7b6-41f2-9857-679ac04b7bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 20:06:19.127891 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:19.127840 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:19.731040 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:19.730840 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:19.890650 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:19.888413 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 20:06:19.922682 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:19.895212 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 20:06:19.922682 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:19.895271 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" podUID="baca0e89-a102-4354-af54-c14ccc418923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 20:06:19.922682 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:19.895380 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 20:06:19.922682 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:19.895404 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" podUID="14b2ad56-7181-420c-8ad1-bdc00f25e928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 20:06:19.922682 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:19.897849 5088 patch_prober.go:29] interesting pod/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:06:19.922682 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:19.897897 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" podUID="ff410c2e-9048-4e5c-965b-1261336561bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:19.922682 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:19.898020 5088 patch_prober.go:29] interesting pod/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 20:06:19.922682 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:19.898064 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" podUID="a51889ad-a7b6-41f2-9857-679ac04b7bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 20:06:19.936425 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:19.934467 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 20:06:20.030879 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:20.015201 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:20.054861 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:20.054824 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 20:06:20.062638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:20.058741 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" podUID="ab112498-5d12-44e0-85b3-0c9f5dc003d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 20:06:20.234347 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:20.057904 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wtlm7" podUID="419ffdb0-c223-4426-872a-50e3073812e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:20.234347 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:20.075307 5088 patch_prober.go:29] interesting pod/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:06:20.234347 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:20.149535 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" podUID="d209c20c-000f-4cc4-9792-5a71eeb0a773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:20.234347 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:20.095843 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9" podUID="1fa4e381-a575-4bf9-a65d-33bdc3093587" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:20.234347 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:20.181948 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:20.234347 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:20.227810 5088 patch_prober.go:29] interesting pod/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:06:20.234347 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:20.227892 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" podUID="fc3ed9fa-b7f0-418f-8ac9-422f9e765027" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:20.347696 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:20.278564 5088 trace.go:236] Trace[1082790546]: "Calculate volume metrics of ca-trust-extracted for pod openshift-image-registry/image-registry-857cdcc958-b2kxp" (30-Oct-2026 20:06:19.089) (total time: 1189ms): Oct 30 20:06:20.347696 ip-10-0-132-173 kubenswrapper[5088]: Trace[1082790546]: [1.189493974s] [1.189493974s] END Oct 30 20:06:20.603693 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:20.602833 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:20.631642 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:20.631587 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 20:06:20.658639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:20.424684 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f4snn" podUID="5ec64166-d2fc-47f6-96d1-48ea0ee1b70c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:20.659142 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:20.288114 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 20:06:20.659416 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:20.417825 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:06:20.659561 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:20.659534 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" podUID="b55ec6e2-9392-4d8d-8340-ef12f674970c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:20.728569 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:20.728512 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 20:06:20.747660 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:20.728842 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" podUID="ab112498-5d12-44e0-85b3-0c9f5dc003d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 20:06:20.772724 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:20.745760 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:20.843577 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:20.843530 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:20.909692 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:20.904446 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 30 20:06:21.042642 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:21.017347 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 20:06:21.110636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:21.045840 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:21.191031 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:21.187417 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:21.225639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:21.193065 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 20:06:21.225639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:21.210545 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" podUID="14b2ad56-7181-420c-8ad1-bdc00f25e928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 20:06:21.225639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:20.659353 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" podUID="baca0e89-a102-4354-af54-c14ccc418923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 20:06:21.352207 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:21.244445 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:06:21.352207 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:21.244507 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" podUID="b56fca64-1150-4198-945e-7e830300c11d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:21.373896 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:21.373861 5088 patch_prober.go:29] interesting pod/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 20:06:21.374116 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:21.374086 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" podUID="a51889ad-a7b6-41f2-9857-679ac04b7bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 20:06:21.453691 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:21.425920 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 20:06:21.453691 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:21.425997 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" podUID="baca0e89-a102-4354-af54-c14ccc418923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 20:06:21.628103 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:21.628066 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 20:06:21.630312 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:21.630275 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" podUID="14b2ad56-7181-420c-8ad1-bdc00f25e928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 20:06:21.631136 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:21.631114 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 20:06:21.631283 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:21.631264 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" podUID="ab112498-5d12-44e0-85b3-0c9f5dc003d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 20:06:21.652673 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:21.631597 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:21.713820 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:06:21.690245 5088 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.948s" Oct 30 20:06:21.774168 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:21.773034 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 30 20:06:21.774168 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:21.774066 5088 patch_prober.go:29] interesting pod/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 20:06:21.774168 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:21.774111 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" podUID="a51889ad-a7b6-41f2-9857-679ac04b7bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 20:06:21.961637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:21.958139 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:22.008661 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:22.003689 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:22.079711 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:22.057664 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:22.128466 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:22.093564 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-j6v5l-pmxpq"] Oct 30 20:06:22.200903 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:22.200856 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:22.202413 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:22.202376 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:22.203275 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:22.203248 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 20:06:22.286694 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:22.284177 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 20:06:22.286694 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:22.284255 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" podUID="baca0e89-a102-4354-af54-c14ccc418923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 20:06:22.288436 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:22.288402 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-j6v5l-pmxpq"] Oct 30 20:06:22.288657 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:22.288639 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-470dd631-q8tgv"] Oct 30 20:06:22.288919 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:22.288896 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-j6v5l-pmxpq" Oct 30 20:06:22.310317 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:22.310290 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-9xs5t\"" Oct 30 20:06:22.428637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:22.421067 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4af0759-e61a-48f6-821a-e1ad34261f11-trusted-ca-bundle\") pod \"periodic-gathering-j6v5l-pmxpq\" (UID: \"a4af0759-e61a-48f6-821a-e1ad34261f11\") " pod="openshift-insights/periodic-gathering-j6v5l-pmxpq" Oct 30 20:06:22.428637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:22.421154 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a4af0759-e61a-48f6-821a-e1ad34261f11-archives-path\") pod \"periodic-gathering-j6v5l-pmxpq\" (UID: \"a4af0759-e61a-48f6-821a-e1ad34261f11\") " pod="openshift-insights/periodic-gathering-j6v5l-pmxpq" Oct 30 20:06:22.428637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:22.421215 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4af0759-e61a-48f6-821a-e1ad34261f11-service-ca-bundle\") pod \"periodic-gathering-j6v5l-pmxpq\" (UID: \"a4af0759-e61a-48f6-821a-e1ad34261f11\") " pod="openshift-insights/periodic-gathering-j6v5l-pmxpq" Oct 30 20:06:22.428637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:22.421366 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvw4q\" (UniqueName: \"kubernetes.io/projected/a4af0759-e61a-48f6-821a-e1ad34261f11-kube-api-access-gvw4q\") pod \"periodic-gathering-j6v5l-pmxpq\" (UID: \"a4af0759-e61a-48f6-821a-e1ad34261f11\") " pod="openshift-insights/periodic-gathering-j6v5l-pmxpq" Oct 30 20:06:22.625402 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:22.625313 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 20:06:22.625729 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:22.625693 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" podUID="14b2ad56-7181-420c-8ad1-bdc00f25e928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 20:06:22.677636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:22.634951 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvw4q\" (UniqueName: \"kubernetes.io/projected/a4af0759-e61a-48f6-821a-e1ad34261f11-kube-api-access-gvw4q\") pod \"periodic-gathering-j6v5l-pmxpq\" (UID: \"a4af0759-e61a-48f6-821a-e1ad34261f11\") " pod="openshift-insights/periodic-gathering-j6v5l-pmxpq" Oct 30 20:06:22.677636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:22.635196 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4af0759-e61a-48f6-821a-e1ad34261f11-trusted-ca-bundle\") pod \"periodic-gathering-j6v5l-pmxpq\" (UID: \"a4af0759-e61a-48f6-821a-e1ad34261f11\") " pod="openshift-insights/periodic-gathering-j6v5l-pmxpq" Oct 30 20:06:22.677636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:22.635263 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a4af0759-e61a-48f6-821a-e1ad34261f11-archives-path\") pod \"periodic-gathering-j6v5l-pmxpq\" (UID: \"a4af0759-e61a-48f6-821a-e1ad34261f11\") " pod="openshift-insights/periodic-gathering-j6v5l-pmxpq" Oct 30 20:06:22.677636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:22.635325 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4af0759-e61a-48f6-821a-e1ad34261f11-service-ca-bundle\") pod \"periodic-gathering-j6v5l-pmxpq\" (UID: \"a4af0759-e61a-48f6-821a-e1ad34261f11\") " pod="openshift-insights/periodic-gathering-j6v5l-pmxpq" Oct 30 20:06:22.677636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:22.636193 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4af0759-e61a-48f6-821a-e1ad34261f11-service-ca-bundle\") pod \"periodic-gathering-j6v5l-pmxpq\" (UID: \"a4af0759-e61a-48f6-821a-e1ad34261f11\") " pod="openshift-insights/periodic-gathering-j6v5l-pmxpq" Oct 30 20:06:22.677636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:22.637243 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4af0759-e61a-48f6-821a-e1ad34261f11-trusted-ca-bundle\") pod \"periodic-gathering-j6v5l-pmxpq\" (UID: \"a4af0759-e61a-48f6-821a-e1ad34261f11\") " pod="openshift-insights/periodic-gathering-j6v5l-pmxpq" Oct 30 20:06:22.677636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:22.637528 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a4af0759-e61a-48f6-821a-e1ad34261f11-archives-path\") pod \"periodic-gathering-j6v5l-pmxpq\" (UID: \"a4af0759-e61a-48f6-821a-e1ad34261f11\") " pod="openshift-insights/periodic-gathering-j6v5l-pmxpq" Oct 30 20:06:22.677636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:22.662506 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 20:06:22.702648 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:22.696717 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 20:06:22.702648 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:22.696777 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" podUID="ab112498-5d12-44e0-85b3-0c9f5dc003d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 20:06:22.778652 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:22.753112 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:06:22.778652 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:22.753178 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" podUID="a255c4a5-c090-49d2-84ba-536dfb9b8a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:22.781926 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:22.779986 5088 patch_prober.go:29] interesting pod/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 20:06:22.781926 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:22.780042 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" podUID="a51889ad-a7b6-41f2-9857-679ac04b7bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 20:06:22.781926 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:22.780118 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:22.960446 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:22.960291 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:23.056087 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:23.056050 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvw4q\" (UniqueName: \"kubernetes.io/projected/a4af0759-e61a-48f6-821a-e1ad34261f11-kube-api-access-gvw4q\") pod \"periodic-gathering-j6v5l-pmxpq\" (UID: \"a4af0759-e61a-48f6-821a-e1ad34261f11\") " pod="openshift-insights/periodic-gathering-j6v5l-pmxpq" Oct 30 20:06:23.191499 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:23.181275 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 20:06:23.534637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:23.525368 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:23.534637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:23.229785 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 20:06:23.534637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:23.229391 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:23.534637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:23.254657 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-j6v5l-pmxpq" Oct 30 20:06:23.543761 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:23.229815 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:23.674642 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:23.651560 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 20:06:23.832296 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:23.832253 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" podUID="ab112498-5d12-44e0-85b3-0c9f5dc003d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 20:06:23.838744 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:23.724806 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 20:06:23.838980 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:23.838947 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" podUID="baca0e89-a102-4354-af54-c14ccc418923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 20:06:23.839102 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:23.724875 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 20:06:23.839198 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:23.839182 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" podUID="14b2ad56-7181-420c-8ad1-bdc00f25e928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 20:06:23.882641 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:23.781833 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:23.994670 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:23.850002 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:24.134638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:24.087641 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded" Oct 30 20:06:24.134638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:24.087747 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:24.262636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:24.258768 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:24.558689 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:24.549721 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:24.558689 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:06:24.550451 5088 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.785s" Oct 30 20:06:24.558689 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:24.550484 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea50d2fa-jxdxf"] Oct 30 20:06:24.577501 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:24.573731 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 20:06:24.577501 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:24.573800 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" podUID="baca0e89-a102-4354-af54-c14ccc418923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 20:06:24.624248 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:24.624207 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 20:06:24.641683 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:24.637554 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea50d2fa-jxdxf" Oct 30 20:06:24.683123 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:24.683017 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 20:06:24.731762 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:24.731695 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 20:06:24.769418 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:24.769376 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 20:06:24.769765 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:24.769732 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" podUID="14b2ad56-7181-420c-8ad1-bdc00f25e928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 20:06:24.980706 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:24.975889 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:06:24.980706 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:24.975981 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" podUID="f4b44c6e-891a-4073-9192-035aeaad10a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:25.073582 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:25.073545 5088 patch_prober.go:29] interesting pod/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 20:06:25.073911 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:25.073831 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" podUID="a51889ad-a7b6-41f2-9857-679ac04b7bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 20:06:25.076646 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:25.073926 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:06:25.114776 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:25.114720 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" podUID="98275e84-2e20-46db-8264-27ec2ac2e2f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:25.114985 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:25.074422 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 20:06:25.115157 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:25.115132 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" podUID="ab112498-5d12-44e0-85b3-0c9f5dc003d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 20:06:25.135639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:25.134599 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 30 20:06:25.198119 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:25.197484 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 20:06:25.223970 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:25.223932 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtrjg\" (UniqueName: \"kubernetes.io/projected/67c834be-8e1e-4f8a-a0d4-565fb0f806a6-kube-api-access-vtrjg\") pod \"maas-api-revoke-sub-ea50d2fa-jxdxf\" (UID: \"67c834be-8e1e-4f8a-a0d4-565fb0f806a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea50d2fa-jxdxf" Oct 30 20:06:25.224441 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:25.224401 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67c834be-8e1e-4f8a-a0d4-565fb0f806a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea50d2fa-jxdxf\" (UID: \"67c834be-8e1e-4f8a-a0d4-565fb0f806a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea50d2fa-jxdxf" Oct 30 20:06:25.246744 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:25.246692 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:25.322683 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:25.322635 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67c834be-8e1e-4f8a-a0d4-565fb0f806a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea50d2fa-jxdxf\" (UID: \"67c834be-8e1e-4f8a-a0d4-565fb0f806a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea50d2fa-jxdxf" Oct 30 20:06:25.491945 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:25.491905 5088 patch_prober.go:29] interesting pod/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 20:06:25.492199 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:25.492168 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" podUID="a51889ad-a7b6-41f2-9857-679ac04b7bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 20:06:25.518691 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:25.510690 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67c834be-8e1e-4f8a-a0d4-565fb0f806a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea50d2fa-jxdxf\" (UID: \"67c834be-8e1e-4f8a-a0d4-565fb0f806a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea50d2fa-jxdxf" Oct 30 20:06:25.544403 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:25.544034 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 20:06:25.544403 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:25.544096 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" podUID="baca0e89-a102-4354-af54-c14ccc418923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 20:06:25.553296 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:25.552846 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:25.624591 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:25.624559 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67c834be-8e1e-4f8a-a0d4-565fb0f806a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea50d2fa-jxdxf\" (UID: \"67c834be-8e1e-4f8a-a0d4-565fb0f806a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea50d2fa-jxdxf" Oct 30 20:06:25.667636 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:06:25.635846 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:06:25.745292 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:25.745188 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:25.746126 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:25.746088 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 20:06:25.747008 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:25.746980 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" podUID="14b2ad56-7181-420c-8ad1-bdc00f25e928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 20:06:25.747117 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:25.746289 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 20:06:25.793490 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:25.793408 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 20:06:25.793736 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:25.793709 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" podUID="ab112498-5d12-44e0-85b3-0c9f5dc003d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 20:06:25.960241 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:25.903883 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67c834be-8e1e-4f8a-a0d4-565fb0f806a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea50d2fa-jxdxf\" (UID: \"67c834be-8e1e-4f8a-a0d4-565fb0f806a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea50d2fa-jxdxf" Oct 30 20:06:25.960241 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:25.904108 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtrjg\" (UniqueName: \"kubernetes.io/projected/67c834be-8e1e-4f8a-a0d4-565fb0f806a6-kube-api-access-vtrjg\") pod \"maas-api-revoke-sub-ea50d2fa-jxdxf\" (UID: \"67c834be-8e1e-4f8a-a0d4-565fb0f806a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea50d2fa-jxdxf" Oct 30 20:06:25.960241 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:25.905462 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67c834be-8e1e-4f8a-a0d4-565fb0f806a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea50d2fa-jxdxf\" (UID: \"67c834be-8e1e-4f8a-a0d4-565fb0f806a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea50d2fa-jxdxf" Oct 30 20:06:25.960241 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:25.936005 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-470dd631-q8tgv" event={"ID":"97f8f58f-a1ca-4489-9b23-d065cf67d9ef","Type":"ContainerStarted","Data":"35b1f5468e14436b3ebedf83adc7d90983576d62716b8ae432e2fc760d246924"} Oct 30 20:06:25.960241 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:25.936121 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" Oct 30 20:06:25.960241 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:25.936154 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea50d2fa-jxdxf"] Oct 30 20:06:25.960241 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:25.936187 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" Oct 30 20:06:26.136446 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:26.136369 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" Oct 30 20:06:26.273588 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:26.273559 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" Oct 30 20:06:26.276309 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:26.276274 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 20:06:26.281018 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:26.280993 5088 patch_prober.go:29] interesting pod/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 20:06:26.281224 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:26.281181 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" podUID="a51889ad-a7b6-41f2-9857-679ac04b7bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 20:06:26.282975 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:26.282946 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:26.355354 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:26.300677 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 30 20:06:26.355354 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:26.329521 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtrjg\" (UniqueName: \"kubernetes.io/projected/67c834be-8e1e-4f8a-a0d4-565fb0f806a6-kube-api-access-vtrjg\") pod \"maas-api-revoke-sub-ea50d2fa-jxdxf\" (UID: \"67c834be-8e1e-4f8a-a0d4-565fb0f806a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea50d2fa-jxdxf" Oct 30 20:06:26.355354 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:26.354698 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 20:06:26.355354 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:26.354746 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" podUID="baca0e89-a102-4354-af54-c14ccc418923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 20:06:26.355354 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:26.354826 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:26.371558 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:26.371527 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:06:26.385631 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:26.372509 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" Oct 30 20:06:26.385631 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:26.377584 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-j6v5l-pmxpq"] Oct 30 20:06:26.543306 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:26.510884 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea50d2fa-jxdxf" Oct 30 20:06:26.554285 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:26.553953 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:26.584520 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:26.574796 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 20:06:26.584520 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:26.574859 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" podUID="14b2ad56-7181-420c-8ad1-bdc00f25e928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 20:06:26.584520 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:26.578924 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:06:26.600665 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:26.597041 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" Oct 30 20:06:26.602599 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:26.602498 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 20:06:26.602599 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:26.602548 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" podUID="ab112498-5d12-44e0-85b3-0c9f5dc003d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 20:06:26.635413 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:26.635385 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" Oct 30 20:06:26.758327 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:26.751982 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:26.767786 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:26.767732 5088 patch_prober.go:29] interesting pod/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 20:06:26.769981 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:26.768010 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" podUID="a51889ad-a7b6-41f2-9857-679ac04b7bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 20:06:26.848901 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:26.843051 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:06:26.878309 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:26.878277 5088 generic.go:356] "Generic (PLEG): container finished" podID="97f8f58f-a1ca-4489-9b23-d065cf67d9ef" containerID="0f24a00bc89f1b48287b00ea3bbd8a19162402791b3e7d1763abefe344dd0d6c" exitCode=0 Oct 30 20:06:26.878796 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:26.878770 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-470dd631-q8tgv" event={"ID":"97f8f58f-a1ca-4489-9b23-d065cf67d9ef","Type":"ContainerDied","Data":"0f24a00bc89f1b48287b00ea3bbd8a19162402791b3e7d1763abefe344dd0d6c"} Oct 30 20:06:26.927769 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:26.927676 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-j6v5l-pmxpq" event={"ID":"a4af0759-e61a-48f6-821a-e1ad34261f11","Type":"ContainerStarted","Data":"f8f5e6e5c59f943f241c9fd72105d3fdd1426ac1369a14c29d5653f53fc545af"} Oct 30 20:06:27.295364 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:27.295173 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 20:06:27.295813 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:27.295779 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" podUID="baca0e89-a102-4354-af54-c14ccc418923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 20:06:27.572852 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:27.572759 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 20:06:27.573123 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:27.573090 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" podUID="14b2ad56-7181-420c-8ad1-bdc00f25e928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 20:06:28.082687 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:06:28.072939 5088 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67c834be_8e1e_4f8a_a0d4_565fb0f806a6.slice/crio-13de5b2ee11ffe0c20c1900314db3afbb6db43ca4b19b52f8126903cd024cd75\": RecentStats: unable to find data in memory cache]" Oct 30 20:06:28.156271 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:06:28.156228 5088 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67c834be_8e1e_4f8a_a0d4_565fb0f806a6.slice/crio-13de5b2ee11ffe0c20c1900314db3afbb6db43ca4b19b52f8126903cd024cd75\": RecentStats: unable to find data in memory cache]" Oct 30 20:06:28.199198 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:28.199163 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" Oct 30 20:06:28.376345 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:28.376262 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea50d2fa-jxdxf"] Oct 30 20:06:28.389329 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:28.387397 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 20:06:28.389329 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:28.387463 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" podUID="baca0e89-a102-4354-af54-c14ccc418923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 20:06:28.415070 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:28.415046 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" Oct 30 20:06:28.447667 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:28.447526 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea50d2fa-jxdxf" event={"ID":"67c834be-8e1e-4f8a-a0d4-565fb0f806a6","Type":"ContainerStarted","Data":"13de5b2ee11ffe0c20c1900314db3afbb6db43ca4b19b52f8126903cd024cd75"} Oct 30 20:06:28.547465 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:28.547374 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-j6v5l-pmxpq" event={"ID":"a4af0759-e61a-48f6-821a-e1ad34261f11","Type":"ContainerStarted","Data":"d6efd7473c24c5e49dc2f91caa389a0281d1692701d86ef43538945ce2d50786"} Oct 30 20:06:28.596941 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:28.596882 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-j6v5l-pmxpq" podStartSLOduration=10.596863984 podStartE2EDuration="10.596863984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:28.59279061 +0000 UTC m=+1150.005504403" watchObservedRunningTime="2026-10-30 20:06:28.596863984 +0000 UTC m=+1150.009577764" Oct 30 20:06:28.688197 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:28.650874 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:06:28.688197 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:28.650983 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" podUID="ab112498-5d12-44e0-85b3-0c9f5dc003d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:29.013239 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:29.009596 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" Oct 30 20:06:29.016643 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:29.014285 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" Oct 30 20:06:29.069401 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:29.069308 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-470dd631-q8tgv" Oct 30 20:06:29.282882 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:29.281822 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97f8f58f-a1ca-4489-9b23-d065cf67d9ef-kube-api-access-pnmpg\" (UniqueName: \"kubernetes.io/projected/97f8f58f-a1ca-4489-9b23-d065cf67d9ef-kube-api-access-pnmpg\") pod \"97f8f58f-a1ca-4489-9b23-d065cf67d9ef\" (UID: \"97f8f58f-a1ca-4489-9b23-d065cf67d9ef\") " Oct 30 20:06:29.282882 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:29.281940 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97f8f58f-a1ca-4489-9b23-d065cf67d9ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97f8f58f-a1ca-4489-9b23-d065cf67d9ef-maas-api-openshift-ca\") pod \"97f8f58f-a1ca-4489-9b23-d065cf67d9ef\" (UID: \"97f8f58f-a1ca-4489-9b23-d065cf67d9ef\") " Oct 30 20:06:29.282882 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:29.282065 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97f8f58f-a1ca-4489-9b23-d065cf67d9ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97f8f58f-a1ca-4489-9b23-d065cf67d9ef-maas-api-xks-ca\") pod \"97f8f58f-a1ca-4489-9b23-d065cf67d9ef\" (UID: \"97f8f58f-a1ca-4489-9b23-d065cf67d9ef\") " Oct 30 20:06:29.300766 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:29.287137 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97f8f58f-a1ca-4489-9b23-d065cf67d9ef-maas-api-openshift-ca" pod "97f8f58f-a1ca-4489-9b23-d065cf67d9ef" (UID: "97f8f58f-a1ca-4489-9b23-d065cf67d9ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:06:29.316120 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:29.316091 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97f8f58f-a1ca-4489-9b23-d065cf67d9ef-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:06:29.322074 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:29.322031 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f8f58f-a1ca-4489-9b23-d065cf67d9ef-kube-api-access-pnmpg" pod "97f8f58f-a1ca-4489-9b23-d065cf67d9ef" (UID: "97f8f58f-a1ca-4489-9b23-d065cf67d9ef"). InnerVolumeSpecName "kube-api-access-pnmpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:06:29.323914 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:29.323891 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97f8f58f-a1ca-4489-9b23-d065cf67d9ef-maas-api-xks-ca" pod "97f8f58f-a1ca-4489-9b23-d065cf67d9ef" (UID: "97f8f58f-a1ca-4489-9b23-d065cf67d9ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:06:29.418535 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:29.418496 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnmpg\" (UniqueName: \"kubernetes.io/projected/97f8f58f-a1ca-4489-9b23-d065cf67d9ef-kube-api-access-pnmpg\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:06:29.418535 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:29.418537 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97f8f58f-a1ca-4489-9b23-d065cf67d9ef-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:06:29.560930 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:29.560839 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-470dd631-q8tgv" event={"ID":"97f8f58f-a1ca-4489-9b23-d065cf67d9ef","Type":"ContainerDied","Data":"35b1f5468e14436b3ebedf83adc7d90983576d62716b8ae432e2fc760d246924"} Oct 30 20:06:29.560930 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:29.560906 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35b1f5468e14436b3ebedf83adc7d90983576d62716b8ae432e2fc760d246924" Oct 30 20:06:29.561177 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:29.560949 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-470dd631-q8tgv" Oct 30 20:06:29.564763 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:29.564735 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea50d2fa-jxdxf" event={"ID":"67c834be-8e1e-4f8a-a0d4-565fb0f806a6","Type":"ContainerStarted","Data":"eadcbf3ad4695c28b102286c1fd0ef0502be02b365d937b01045691fb090b939"} Oct 30 20:06:29.617202 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:29.617133 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea50d2fa-jxdxf" podStartSLOduration=6.617113189 podStartE2EDuration="6.617113189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:29.614085423 +0000 UTC m=+1151.026799203" watchObservedRunningTime="2026-10-30 20:06:29.617113189 +0000 UTC m=+1151.029826969" Oct 30 20:06:29.694838 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:29.694780 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:06:29.695018 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:29.694865 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" podUID="ab112498-5d12-44e0-85b3-0c9f5dc003d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:30.289637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:30.286932 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:06:30.289637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:30.287027 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" podUID="baca0e89-a102-4354-af54-c14ccc418923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:30.613437 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:30.613343 5088 generic.go:356] "Generic (PLEG): container finished" podID="67c834be-8e1e-4f8a-a0d4-565fb0f806a6" containerID="eadcbf3ad4695c28b102286c1fd0ef0502be02b365d937b01045691fb090b939" exitCode=0 Oct 30 20:06:30.625212 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:30.625168 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea50d2fa-jxdxf" event={"ID":"67c834be-8e1e-4f8a-a0d4-565fb0f806a6","Type":"ContainerDied","Data":"eadcbf3ad4695c28b102286c1fd0ef0502be02b365d937b01045691fb090b939"} Oct 30 20:06:30.703902 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:30.703865 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:06:30.704226 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:30.704194 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" podUID="ab112498-5d12-44e0-85b3-0c9f5dc003d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:30.834741 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:30.834701 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" Oct 30 20:06:30.848643 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:30.843536 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" Oct 30 20:06:31.328811 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:31.328769 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:06:31.328992 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:31.328844 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" podUID="baca0e89-a102-4354-af54-c14ccc418923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:31.400777 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:31.400743 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" Oct 30 20:06:31.402279 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:31.402240 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" Oct 30 20:06:31.829228 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:31.829199 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea50d2fa-jxdxf" Oct 30 20:06:31.903745 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:31.903668 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67c834be-8e1e-4f8a-a0d4-565fb0f806a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67c834be-8e1e-4f8a-a0d4-565fb0f806a6-maas-api-xks-ca\") pod \"67c834be-8e1e-4f8a-a0d4-565fb0f806a6\" (UID: \"67c834be-8e1e-4f8a-a0d4-565fb0f806a6\") " Oct 30 20:06:31.903922 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:31.903903 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67c834be-8e1e-4f8a-a0d4-565fb0f806a6-kube-api-access-vtrjg\" (UniqueName: \"kubernetes.io/projected/67c834be-8e1e-4f8a-a0d4-565fb0f806a6-kube-api-access-vtrjg\") pod \"67c834be-8e1e-4f8a-a0d4-565fb0f806a6\" (UID: \"67c834be-8e1e-4f8a-a0d4-565fb0f806a6\") " Oct 30 20:06:31.904064 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:31.904033 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67c834be-8e1e-4f8a-a0d4-565fb0f806a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67c834be-8e1e-4f8a-a0d4-565fb0f806a6-maas-api-openshift-ca\") pod \"67c834be-8e1e-4f8a-a0d4-565fb0f806a6\" (UID: \"67c834be-8e1e-4f8a-a0d4-565fb0f806a6\") " Oct 30 20:06:31.904581 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:31.904546 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67c834be-8e1e-4f8a-a0d4-565fb0f806a6-maas-api-openshift-ca" pod "67c834be-8e1e-4f8a-a0d4-565fb0f806a6" (UID: "67c834be-8e1e-4f8a-a0d4-565fb0f806a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:06:31.905102 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:31.905077 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67c834be-8e1e-4f8a-a0d4-565fb0f806a6-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:06:31.905938 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:31.905910 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67c834be-8e1e-4f8a-a0d4-565fb0f806a6-maas-api-xks-ca" pod "67c834be-8e1e-4f8a-a0d4-565fb0f806a6" (UID: "67c834be-8e1e-4f8a-a0d4-565fb0f806a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:06:31.906123 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:31.906103 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67c834be-8e1e-4f8a-a0d4-565fb0f806a6-kube-api-access-vtrjg" pod "67c834be-8e1e-4f8a-a0d4-565fb0f806a6" (UID: "67c834be-8e1e-4f8a-a0d4-565fb0f806a6"). InnerVolumeSpecName "kube-api-access-vtrjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:06:32.006363 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:32.006336 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67c834be-8e1e-4f8a-a0d4-565fb0f806a6-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:06:32.006363 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:32.006363 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtrjg\" (UniqueName: \"kubernetes.io/projected/67c834be-8e1e-4f8a-a0d4-565fb0f806a6-kube-api-access-vtrjg\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:06:32.655830 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:32.655802 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea50d2fa-jxdxf" Oct 30 20:06:32.656010 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:32.655826 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea50d2fa-jxdxf" event={"ID":"67c834be-8e1e-4f8a-a0d4-565fb0f806a6","Type":"ContainerDied","Data":"13de5b2ee11ffe0c20c1900314db3afbb6db43ca4b19b52f8126903cd024cd75"} Oct 30 20:06:32.656010 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:32.655894 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13de5b2ee11ffe0c20c1900314db3afbb6db43ca4b19b52f8126903cd024cd75" Oct 30 20:06:37.160691 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:37.160657 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-308588d2-nl64j"] Oct 30 20:06:37.161366 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:37.161350 5088 cpu_manager.go:414] "containerMap: removing container" podUID="67c834be-8e1e-4f8a-a0d4-565fb0f806a6" containerName="revoke-keys" Oct 30 20:06:37.161366 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:37.161366 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67c834be-8e1e-4f8a-a0d4-565fb0f806a6" containerName="revoke-keys" Oct 30 20:06:37.161482 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:37.161387 5088 cpu_manager.go:414] "containerMap: removing container" podUID="97f8f58f-a1ca-4489-9b23-d065cf67d9ef" containerName="revoke-keys" Oct 30 20:06:37.161482 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:37.161392 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97f8f58f-a1ca-4489-9b23-d065cf67d9ef" containerName="revoke-keys" Oct 30 20:06:37.161544 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:37.161515 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="67c834be-8e1e-4f8a-a0d4-565fb0f806a6" containerName="revoke-keys" Oct 30 20:06:37.161544 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:37.161527 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="97f8f58f-a1ca-4489-9b23-d065cf67d9ef" containerName="revoke-keys" Oct 30 20:06:37.165738 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:37.165715 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-308588d2-nl64j" Oct 30 20:06:37.190589 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:37.190564 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-308588d2-nl64j"] Oct 30 20:06:37.196844 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:37.196822 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:06:37.196967 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:37.196900 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:06:37.305124 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:37.305088 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9d900df-2b7f-4e2c-9de7-76d587992dbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-308588d2-nl64j\" (UID: \"f9d900df-2b7f-4e2c-9de7-76d587992dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-308588d2-nl64j" Oct 30 20:06:37.305124 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:37.305122 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwkdn\" (UniqueName: \"kubernetes.io/projected/f9d900df-2b7f-4e2c-9de7-76d587992dbc-kube-api-access-gwkdn\") pod \"maas-api-revoke-sub-308588d2-nl64j\" (UID: \"f9d900df-2b7f-4e2c-9de7-76d587992dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-308588d2-nl64j" Oct 30 20:06:37.305316 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:37.305280 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9d900df-2b7f-4e2c-9de7-76d587992dbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-308588d2-nl64j\" (UID: \"f9d900df-2b7f-4e2c-9de7-76d587992dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-308588d2-nl64j" Oct 30 20:06:37.407064 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:37.407030 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9d900df-2b7f-4e2c-9de7-76d587992dbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-308588d2-nl64j\" (UID: \"f9d900df-2b7f-4e2c-9de7-76d587992dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-308588d2-nl64j" Oct 30 20:06:37.407234 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:37.407068 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwkdn\" (UniqueName: \"kubernetes.io/projected/f9d900df-2b7f-4e2c-9de7-76d587992dbc-kube-api-access-gwkdn\") pod \"maas-api-revoke-sub-308588d2-nl64j\" (UID: \"f9d900df-2b7f-4e2c-9de7-76d587992dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-308588d2-nl64j" Oct 30 20:06:37.407234 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:37.407181 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9d900df-2b7f-4e2c-9de7-76d587992dbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-308588d2-nl64j\" (UID: \"f9d900df-2b7f-4e2c-9de7-76d587992dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-308588d2-nl64j" Oct 30 20:06:37.407818 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:37.407788 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9d900df-2b7f-4e2c-9de7-76d587992dbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-308588d2-nl64j\" (UID: \"f9d900df-2b7f-4e2c-9de7-76d587992dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-308588d2-nl64j" Oct 30 20:06:37.409796 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:37.409765 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9d900df-2b7f-4e2c-9de7-76d587992dbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-308588d2-nl64j\" (UID: \"f9d900df-2b7f-4e2c-9de7-76d587992dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-308588d2-nl64j" Oct 30 20:06:37.433456 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:37.433396 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwkdn\" (UniqueName: \"kubernetes.io/projected/f9d900df-2b7f-4e2c-9de7-76d587992dbc-kube-api-access-gwkdn\") pod \"maas-api-revoke-sub-308588d2-nl64j\" (UID: \"f9d900df-2b7f-4e2c-9de7-76d587992dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-308588d2-nl64j" Oct 30 20:06:37.480474 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:37.480451 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-308588d2-nl64j" Oct 30 20:06:37.789887 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:06:37.789853 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9d900df_2b7f_4e2c_9de7_76d587992dbc.slice/crio-10bc31cf18f8ac5f6276347675e404025a170f5da2b18048ae16179950bc7e5b WatchSource:0}: Error finding container 10bc31cf18f8ac5f6276347675e404025a170f5da2b18048ae16179950bc7e5b: Status 404 returned error can't find the container with id 10bc31cf18f8ac5f6276347675e404025a170f5da2b18048ae16179950bc7e5b Oct 30 20:06:37.791176 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:37.791151 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-308588d2-nl64j"] Oct 30 20:06:38.705923 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:38.705880 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-308588d2-nl64j" event={"ID":"f9d900df-2b7f-4e2c-9de7-76d587992dbc","Type":"ContainerStarted","Data":"223cc417b99fed109810a38316ebd90205fca5c70e153c5074619d86494663be"} Oct 30 20:06:38.706309 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:38.705967 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-308588d2-nl64j" event={"ID":"f9d900df-2b7f-4e2c-9de7-76d587992dbc","Type":"ContainerStarted","Data":"10bc31cf18f8ac5f6276347675e404025a170f5da2b18048ae16179950bc7e5b"} Oct 30 20:06:38.757667 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:38.757602 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-308588d2-nl64j" podStartSLOduration=1.757588717 podStartE2EDuration="1.757588717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:38.755407131 +0000 UTC m=+1160.168120910" watchObservedRunningTime="2026-10-30 20:06:38.757588717 +0000 UTC m=+1160.170302497" Oct 30 20:06:39.074326 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:39.074278 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:06:39.716284 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:39.716249 5088 generic.go:356] "Generic (PLEG): container finished" podID="f9d900df-2b7f-4e2c-9de7-76d587992dbc" containerID="223cc417b99fed109810a38316ebd90205fca5c70e153c5074619d86494663be" exitCode=0 Oct 30 20:06:39.716700 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:39.716327 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-308588d2-nl64j" event={"ID":"f9d900df-2b7f-4e2c-9de7-76d587992dbc","Type":"ContainerDied","Data":"223cc417b99fed109810a38316ebd90205fca5c70e153c5074619d86494663be"} Oct 30 20:06:41.624636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:41.624173 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:41.660635 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:41.654808 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:41.788248 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:41.787688 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 20:06:41.839455 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:41.801330 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs"] Oct 30 20:06:41.839874 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:41.839766 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:42.063783 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:42.063718 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:42.176347 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:42.095721 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 30 20:06:42.177411 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:42.095767 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 20:06:42.178943 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:42.175144 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:42.545530 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:42.545491 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" Oct 30 20:06:42.857686 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:42.854017 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-efbac3\"/\"distinct-2-w2-efbac3-kserve-self-signed-certs\"" Oct 30 20:06:42.857686 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:42.854196 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-efbac3\"/\"kube-root-ca.crt\"" Oct 30 20:06:42.857686 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:42.854346 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-efbac3\"/\"openshift-service-ca.crt\"" Oct 30 20:06:42.905240 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:42.878001 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 30 20:06:42.933082 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:42.914290 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:42.942351 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:42.942295 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 20:06:43.112708 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:43.039193 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fec493c7-3f5a-459e-99e0-a4c816d5a907-tls-certs\") pod \"distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs\" (UID: \"fec493c7-3f5a-459e-99e0-a4c816d5a907\") " pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" Oct 30 20:06:43.112708 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:43.106808 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fec493c7-3f5a-459e-99e0-a4c816d5a907-dshm\") pod \"distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs\" (UID: \"fec493c7-3f5a-459e-99e0-a4c816d5a907\") " pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" Oct 30 20:06:43.112708 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:43.107174 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg882\" (UniqueName: \"kubernetes.io/projected/fec493c7-3f5a-459e-99e0-a4c816d5a907-kube-api-access-vg882\") pod \"distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs\" (UID: \"fec493c7-3f5a-459e-99e0-a4c816d5a907\") " pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" Oct 30 20:06:43.112708 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:43.107242 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fec493c7-3f5a-459e-99e0-a4c816d5a907-model-cache\") pod \"distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs\" (UID: \"fec493c7-3f5a-459e-99e0-a4c816d5a907\") " pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" Oct 30 20:06:43.112708 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:43.107295 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fec493c7-3f5a-459e-99e0-a4c816d5a907-tmp-dir\") pod \"distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs\" (UID: \"fec493c7-3f5a-459e-99e0-a4c816d5a907\") " pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" Oct 30 20:06:43.112708 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:43.107366 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fec493c7-3f5a-459e-99e0-a4c816d5a907-home\") pod \"distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs\" (UID: \"fec493c7-3f5a-459e-99e0-a4c816d5a907\") " pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" Oct 30 20:06:43.180658 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:42.954300 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-efbac3\"/\"default-dockercfg-j897c\"" Oct 30 20:06:43.253391 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:43.251780 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:43.284651 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:43.254463 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 30 20:06:43.313841 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:43.313780 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 20:06:43.565638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:43.561680 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:43.953638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:43.949714 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": context deadline exceeded" Oct 30 20:06:43.982760 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:43.955569 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 30 20:06:44.011478 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:06:44.011432 5088 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.166s" Oct 30 20:06:44.011750 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:44.011730 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs"] Oct 30 20:06:44.011853 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:43.955663 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:44.012163 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:43.955602 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 20:06:44.071639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:44.030099 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 20:06:44.147732 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:44.135445 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" Oct 30 20:06:44.147732 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:44.135639 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:06:44.147732 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:44.135682 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:06:44.147732 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:44.135719 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" Oct 30 20:06:44.147732 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:44.135755 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 20:06:44.147732 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:44.135791 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" Oct 30 20:06:44.147732 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:44.135825 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:06:44.268950 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:44.268913 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-308588d2-nl64j" Oct 30 20:06:44.418711 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:44.372430 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:44.528373 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:44.528330 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 20:06:44.528971 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:44.528938 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:44.534194 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:44.534158 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": context deadline exceeded" Oct 30 20:06:44.583722 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:44.559816 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fec493c7-3f5a-459e-99e0-a4c816d5a907-tls-certs\") pod \"distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs\" (UID: \"fec493c7-3f5a-459e-99e0-a4c816d5a907\") " pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" Oct 30 20:06:44.584070 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:44.584045 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fec493c7-3f5a-459e-99e0-a4c816d5a907-dshm\") pod \"distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs\" (UID: \"fec493c7-3f5a-459e-99e0-a4c816d5a907\") " pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" Oct 30 20:06:44.584532 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:44.584504 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg882\" (UniqueName: \"kubernetes.io/projected/fec493c7-3f5a-459e-99e0-a4c816d5a907-kube-api-access-vg882\") pod \"distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs\" (UID: \"fec493c7-3f5a-459e-99e0-a4c816d5a907\") " pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" Oct 30 20:06:44.592785 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:44.592750 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fec493c7-3f5a-459e-99e0-a4c816d5a907-model-cache\") pod \"distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs\" (UID: \"fec493c7-3f5a-459e-99e0-a4c816d5a907\") " pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" Oct 30 20:06:44.671977 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:44.671865 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fec493c7-3f5a-459e-99e0-a4c816d5a907-tmp-dir\") pod \"distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs\" (UID: \"fec493c7-3f5a-459e-99e0-a4c816d5a907\") " pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" Oct 30 20:06:44.672355 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:44.672315 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fec493c7-3f5a-459e-99e0-a4c816d5a907-home\") pod \"distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs\" (UID: \"fec493c7-3f5a-459e-99e0-a4c816d5a907\") " pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" Oct 30 20:06:44.688835 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:44.672968 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fec493c7-3f5a-459e-99e0-a4c816d5a907-home\") pod \"distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs\" (UID: \"fec493c7-3f5a-459e-99e0-a4c816d5a907\") " pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" Oct 30 20:06:44.689168 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:44.674982 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fec493c7-3f5a-459e-99e0-a4c816d5a907-model-cache\") pod \"distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs\" (UID: \"fec493c7-3f5a-459e-99e0-a4c816d5a907\") " pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" Oct 30 20:06:44.696638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:44.681990 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:44.798931 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:44.697200 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fec493c7-3f5a-459e-99e0-a4c816d5a907-tmp-dir\") pod \"distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs\" (UID: \"fec493c7-3f5a-459e-99e0-a4c816d5a907\") " pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" Oct 30 20:06:44.845635 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:44.817386 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fec493c7-3f5a-459e-99e0-a4c816d5a907-tls-certs\") pod \"distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs\" (UID: \"fec493c7-3f5a-459e-99e0-a4c816d5a907\") " pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" Oct 30 20:06:44.938297 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:44.847943 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9d900df-2b7f-4e2c-9de7-76d587992dbc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9d900df-2b7f-4e2c-9de7-76d587992dbc-maas-api-xks-ca\") pod \"f9d900df-2b7f-4e2c-9de7-76d587992dbc\" (UID: \"f9d900df-2b7f-4e2c-9de7-76d587992dbc\") " Oct 30 20:06:44.986645 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:44.891601 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9d900df-2b7f-4e2c-9de7-76d587992dbc-maas-api-xks-ca" pod "f9d900df-2b7f-4e2c-9de7-76d587992dbc" (UID: "f9d900df-2b7f-4e2c-9de7-76d587992dbc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:06:44.986645 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:44.892079 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fec493c7-3f5a-459e-99e0-a4c816d5a907-dshm\") pod \"distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs\" (UID: \"fec493c7-3f5a-459e-99e0-a4c816d5a907\") " pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" Oct 30 20:06:44.986645 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:44.945702 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg882\" (UniqueName: \"kubernetes.io/projected/fec493c7-3f5a-459e-99e0-a4c816d5a907-kube-api-access-vg882\") pod \"distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs\" (UID: \"fec493c7-3f5a-459e-99e0-a4c816d5a907\") " pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" Oct 30 20:06:44.986645 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:44.954842 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9d900df-2b7f-4e2c-9de7-76d587992dbc-kube-api-access-gwkdn\" (UniqueName: \"kubernetes.io/projected/f9d900df-2b7f-4e2c-9de7-76d587992dbc-kube-api-access-gwkdn\") pod \"f9d900df-2b7f-4e2c-9de7-76d587992dbc\" (UID: \"f9d900df-2b7f-4e2c-9de7-76d587992dbc\") " Oct 30 20:06:44.986645 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:44.980818 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9d900df-2b7f-4e2c-9de7-76d587992dbc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9d900df-2b7f-4e2c-9de7-76d587992dbc-maas-api-openshift-ca\") pod \"f9d900df-2b7f-4e2c-9de7-76d587992dbc\" (UID: \"f9d900df-2b7f-4e2c-9de7-76d587992dbc\") " Oct 30 20:06:45.045642 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:45.016976 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9d900df-2b7f-4e2c-9de7-76d587992dbc-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:06:45.045642 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:45.036323 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:45.045987 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:45.045855 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9d900df-2b7f-4e2c-9de7-76d587992dbc-kube-api-access-gwkdn" pod "f9d900df-2b7f-4e2c-9de7-76d587992dbc" (UID: "f9d900df-2b7f-4e2c-9de7-76d587992dbc"). InnerVolumeSpecName "kube-api-access-gwkdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:06:45.205185 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:45.201886 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 20:06:45.205185 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:45.202289 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 20:06:45.206450 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:45.206419 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9d900df-2b7f-4e2c-9de7-76d587992dbc-maas-api-openshift-ca" pod "f9d900df-2b7f-4e2c-9de7-76d587992dbc" (UID: "f9d900df-2b7f-4e2c-9de7-76d587992dbc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:06:45.221537 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:45.217178 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" Oct 30 20:06:45.492704 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:45.437446 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwkdn\" (UniqueName: \"kubernetes.io/projected/f9d900df-2b7f-4e2c-9de7-76d587992dbc-kube-api-access-gwkdn\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:06:45.492704 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:45.437502 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9d900df-2b7f-4e2c-9de7-76d587992dbc-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:06:45.512861 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:45.512792 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:45.547006 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:45.546940 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:45.589779 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:45.589735 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:45.806827 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:45.806719 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:45.985282 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:45.985230 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-308588d2-nl64j" event={"ID":"f9d900df-2b7f-4e2c-9de7-76d587992dbc","Type":"ContainerDied","Data":"10bc31cf18f8ac5f6276347675e404025a170f5da2b18048ae16179950bc7e5b"} Oct 30 20:06:46.039856 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:45.986461 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-308588d2-nl64j" Oct 30 20:06:46.051726 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:46.051677 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10bc31cf18f8ac5f6276347675e404025a170f5da2b18048ae16179950bc7e5b" Oct 30 20:06:46.055738 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:46.055674 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 30 20:06:46.282335 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:46.255942 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:46.282335 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:46.279928 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 20:06:46.591699 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:46.586713 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:46.591699 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:46.586987 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:46.591699 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:46.589713 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:46.626639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:46.625716 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 20:06:47.079317 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:47.079259 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:47.088068 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:47.088015 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:47.160386 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:47.136564 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:06:47.160386 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:47.136643 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" podUID="ab112498-5d12-44e0-85b3-0c9f5dc003d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:47.325704 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:47.320847 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 20:06:47.325704 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:47.321148 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 20:06:47.492795 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:47.492752 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:06:47.496639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:47.493030 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" podUID="baca0e89-a102-4354-af54-c14ccc418923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:47.607473 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:47.607428 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:47.624786 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:47.607866 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 20:06:47.646341 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:47.646301 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 20:06:47.791703 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:47.771227 5088 patch_prober.go:29] interesting pod/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:06:48.074848 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:48.074737 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" podUID="f2757424-fb74-48ff-82c3-ccb79a27fd2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:48.105893 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:48.097846 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:48.109480 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:48.109431 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 20:06:48.214396 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:48.213761 5088 patch_prober.go:29] interesting pod/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:06:48.214396 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:48.213851 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" podUID="678518a9-8731-4f6f-a7a1-4b0e64070e92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:48.556637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:48.554038 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 20:06:48.556637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:48.554427 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 20:06:48.682821 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:48.682772 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:48.690704 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:48.683500 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 20:06:48.855412 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:48.825081 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:49.124734 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:48.984436 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:49.155645 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:49.154733 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 20:06:49.729584 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:49.728166 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 20:06:49.742885 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:49.742835 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 20:06:49.890105 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:49.743269 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": context deadline exceeded" Oct 30 20:06:49.890437 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:49.783858 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 30 20:06:49.912791 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:49.856093 5088 patch_prober.go:29] interesting pod/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:06:50.173873 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:50.140269 5088 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-qdj4h container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.11:8443/healthz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:06:50.174625 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:50.174583 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-7cd56b8dbc-qdj4h" podUID="da850825-59ab-480f-9047-afd7d2bdc19f" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.11:8443/healthz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:50.359643 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:50.108831 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:50.399037 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:50.140042 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9" podUID="1fa4e381-a575-4bf9-a65d-33bdc3093587" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded" Oct 30 20:06:50.399710 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:49.886131 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:50.399983 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:50.020114 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:50.400187 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:50.140305 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:06:50.400305 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:50.140930 5088 patch_prober.go:29] interesting pod/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:06:50.400448 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:50.400415 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" podUID="d209c20c-000f-4cc4-9792-5a71eeb0a773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:50.400594 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:50.241144 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" podUID="4789c6d9-e328-4890-a2b3-a2fdbfdb04b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:50.400757 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:50.174546 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:50.400946 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:50.247777 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:06:50.416770 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:50.416712 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" podUID="419aecf7-a250-42ad-995b-e7ebd9630a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:50.417112 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:50.298115 5088 patch_prober.go:29] interesting pod/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:06:50.417266 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:50.417242 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" podUID="cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:50.417394 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:50.304053 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:06:50.417505 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:50.417487 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" podUID="b55ec6e2-9392-4d8d-8340-ef12f674970c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:50.496052 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:06:50.495997 5088 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.551s" Oct 30 20:06:50.496346 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:50.496325 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0a61055-88j9x"] Oct 30 20:06:50.497823 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:50.497801 5088 cpu_manager.go:414] "containerMap: removing container" podUID="f9d900df-2b7f-4e2c-9de7-76d587992dbc" containerName="revoke-keys" Oct 30 20:06:50.497985 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:50.497970 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9d900df-2b7f-4e2c-9de7-76d587992dbc" containerName="revoke-keys" Oct 30 20:06:50.498450 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:50.498434 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9d900df-2b7f-4e2c-9de7-76d587992dbc" containerName="revoke-keys" Oct 30 20:06:50.551939 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:50.551880 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" podUID="05a46855-a143-4c6d-8c87-404dc271bdb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:50.617053 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:50.617001 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0a61055-88j9x"] Oct 30 20:06:50.617325 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:50.617295 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs"] Oct 30 20:06:50.620198 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:50.617923 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a61055-88j9x" Oct 30 20:06:50.778031 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:50.763745 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:06:50.778031 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:50.763819 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" podUID="b56fca64-1150-4198-945e-7e830300c11d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:50.801811 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:50.801750 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 20:06:50.810645 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:50.805435 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec80a8b1-509a-4c56-a03c-f5a2a5e3f720-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0a61055-88j9x\" (UID: \"ec80a8b1-509a-4c56-a03c-f5a2a5e3f720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a61055-88j9x" Oct 30 20:06:50.810645 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:50.805745 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg5b9\" (UniqueName: \"kubernetes.io/projected/ec80a8b1-509a-4c56-a03c-f5a2a5e3f720-kube-api-access-mg5b9\") pod \"maas-api-revoke-sub-c0a61055-88j9x\" (UID: \"ec80a8b1-509a-4c56-a03c-f5a2a5e3f720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a61055-88j9x" Oct 30 20:06:50.810645 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:50.806124 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec80a8b1-509a-4c56-a03c-f5a2a5e3f720-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0a61055-88j9x\" (UID: \"ec80a8b1-509a-4c56-a03c-f5a2a5e3f720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a61055-88j9x" Oct 30 20:06:50.915592 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:50.915536 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 20:06:50.916338 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:50.916311 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:06:50.916540 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:50.916517 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" podUID="0b36276d-bc5b-4415-8cf9-461aee572f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:50.918052 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:50.918025 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg5b9\" (UniqueName: \"kubernetes.io/projected/ec80a8b1-509a-4c56-a03c-f5a2a5e3f720-kube-api-access-mg5b9\") pod \"maas-api-revoke-sub-c0a61055-88j9x\" (UID: \"ec80a8b1-509a-4c56-a03c-f5a2a5e3f720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a61055-88j9x" Oct 30 20:06:50.918459 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:50.918441 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec80a8b1-509a-4c56-a03c-f5a2a5e3f720-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0a61055-88j9x\" (UID: \"ec80a8b1-509a-4c56-a03c-f5a2a5e3f720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a61055-88j9x" Oct 30 20:06:50.918711 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:50.918679 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec80a8b1-509a-4c56-a03c-f5a2a5e3f720-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0a61055-88j9x\" (UID: \"ec80a8b1-509a-4c56-a03c-f5a2a5e3f720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a61055-88j9x" Oct 30 20:06:50.920387 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:50.920074 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec80a8b1-509a-4c56-a03c-f5a2a5e3f720-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0a61055-88j9x\" (UID: \"ec80a8b1-509a-4c56-a03c-f5a2a5e3f720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a61055-88j9x" Oct 30 20:06:51.152832 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:51.141916 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:06:51.152832 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:51.142034 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:06:51.180761 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:51.180704 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:51.464703 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:51.441011 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 20:06:51.705691 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:51.443685 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:51.740923 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:51.732905 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec80a8b1-509a-4c56-a03c-f5a2a5e3f720-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0a61055-88j9x\" (UID: \"ec80a8b1-509a-4c56-a03c-f5a2a5e3f720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a61055-88j9x" Oct 30 20:06:51.740923 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:51.444520 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:51.740923 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:51.659484 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded" Oct 30 20:06:51.740923 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:51.444484 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:51.812981 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:51.812921 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 20:06:52.082700 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:52.081508 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:52.198653 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:52.194579 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 30 20:06:52.314092 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:52.312809 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 20:06:52.319838 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:52.319802 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" event={"ID":"fec493c7-3f5a-459e-99e0-a4c816d5a907","Type":"ContainerStarted","Data":"06afed01b949ca08c765a19d5e4ae7e6f7ac6f831b6b61480d34d26e42444a8b"} Oct 30 20:06:52.328215 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:52.328181 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:06:52.358068 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:52.357974 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:06:52.537655 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:52.531671 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg5b9\" (UniqueName: \"kubernetes.io/projected/ec80a8b1-509a-4c56-a03c-f5a2a5e3f720-kube-api-access-mg5b9\") pod \"maas-api-revoke-sub-c0a61055-88j9x\" (UID: \"ec80a8b1-509a-4c56-a03c-f5a2a5e3f720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a61055-88j9x" Oct 30 20:06:52.585919 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:06:52.560327 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:06:52.615729 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:06:52.608480 5088 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9d900df_2b7f_4e2c_9de7_76d587992dbc.slice\": RecentStats: unable to find data in memory cache]" Oct 30 20:06:52.619891 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:52.619857 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a61055-88j9x" Oct 30 20:06:52.620503 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:52.620485 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fb6fdb9-6wq2f"] Oct 30 20:06:52.706654 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:52.695980 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 20:06:52.709482 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:52.709436 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fb6fdb9-6wq2f"] Oct 30 20:06:52.709682 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:52.709561 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" Oct 30 20:06:52.710905 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:52.710844 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb6fdb9-6wq2f" Oct 30 20:06:52.720915 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:52.712363 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" Oct 30 20:06:52.736641 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:52.729635 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:06:52.761764 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:52.740689 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:52.803600 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:52.803560 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6mdf\" (UniqueName: \"kubernetes.io/projected/8f63941f-399d-48c0-8203-0bba97e49e8a-kube-api-access-b6mdf\") pod \"maas-api-revoke-sub-8fb6fdb9-6wq2f\" (UID: \"8f63941f-399d-48c0-8203-0bba97e49e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb6fdb9-6wq2f" Oct 30 20:06:52.804087 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:52.804061 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f63941f-399d-48c0-8203-0bba97e49e8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fb6fdb9-6wq2f\" (UID: \"8f63941f-399d-48c0-8203-0bba97e49e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb6fdb9-6wq2f" Oct 30 20:06:52.804336 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:52.804320 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f63941f-399d-48c0-8203-0bba97e49e8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fb6fdb9-6wq2f\" (UID: \"8f63941f-399d-48c0-8203-0bba97e49e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb6fdb9-6wq2f" Oct 30 20:06:52.807572 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:52.807540 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" Oct 30 20:06:52.906036 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:52.905939 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6mdf\" (UniqueName: \"kubernetes.io/projected/8f63941f-399d-48c0-8203-0bba97e49e8a-kube-api-access-b6mdf\") pod \"maas-api-revoke-sub-8fb6fdb9-6wq2f\" (UID: \"8f63941f-399d-48c0-8203-0bba97e49e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb6fdb9-6wq2f" Oct 30 20:06:52.906226 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:52.906059 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f63941f-399d-48c0-8203-0bba97e49e8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fb6fdb9-6wq2f\" (UID: \"8f63941f-399d-48c0-8203-0bba97e49e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb6fdb9-6wq2f" Oct 30 20:06:52.906329 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:52.906296 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f63941f-399d-48c0-8203-0bba97e49e8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fb6fdb9-6wq2f\" (UID: \"8f63941f-399d-48c0-8203-0bba97e49e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb6fdb9-6wq2f" Oct 30 20:06:52.906810 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:52.906787 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f63941f-399d-48c0-8203-0bba97e49e8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fb6fdb9-6wq2f\" (UID: \"8f63941f-399d-48c0-8203-0bba97e49e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb6fdb9-6wq2f" Oct 30 20:06:52.908906 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:52.908879 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f63941f-399d-48c0-8203-0bba97e49e8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fb6fdb9-6wq2f\" (UID: \"8f63941f-399d-48c0-8203-0bba97e49e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb6fdb9-6wq2f" Oct 30 20:06:52.938589 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:52.938541 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6mdf\" (UniqueName: \"kubernetes.io/projected/8f63941f-399d-48c0-8203-0bba97e49e8a-kube-api-access-b6mdf\") pod \"maas-api-revoke-sub-8fb6fdb9-6wq2f\" (UID: \"8f63941f-399d-48c0-8203-0bba97e49e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb6fdb9-6wq2f" Oct 30 20:06:53.112312 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:53.112274 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb6fdb9-6wq2f" Oct 30 20:06:53.112940 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:53.112911 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0a61055-88j9x"] Oct 30 20:06:53.356749 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:53.355061 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" event={"ID":"fec493c7-3f5a-459e-99e0-a4c816d5a907","Type":"ContainerStarted","Data":"ac82bf60d85161dcf77d16cf1dd8fdc013432c2b30ea62b68562d08e33b9ebfe"} Oct 30 20:06:53.361164 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:53.361079 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a61055-88j9x" event={"ID":"ec80a8b1-509a-4c56-a03c-f5a2a5e3f720","Type":"ContainerStarted","Data":"3286f53199bc75ec1f264feede3bb6eb10a299d298d1de5caeb82ad3116d7206"} Oct 30 20:06:53.427807 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:53.427688 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" podStartSLOduration=12.427670372 podStartE2EDuration="12.427670372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:53.421399249 +0000 UTC m=+1174.834113028" watchObservedRunningTime="2026-10-30 20:06:53.427670372 +0000 UTC m=+1174.840384207" Oct 30 20:06:53.478569 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:53.478542 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fb6fdb9-6wq2f"] Oct 30 20:06:53.480553 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:06:53.480525 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f63941f_399d_48c0_8203_0bba97e49e8a.slice/crio-07ed612525aa1063784c2fbe3eda246dd2c7d6b5e8cbde7287e4408428823ef9 WatchSource:0}: Error finding container 07ed612525aa1063784c2fbe3eda246dd2c7d6b5e8cbde7287e4408428823ef9: Status 404 returned error can't find the container with id 07ed612525aa1063784c2fbe3eda246dd2c7d6b5e8cbde7287e4408428823ef9 Oct 30 20:06:54.222066 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:54.222034 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" Oct 30 20:06:54.373252 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:54.373216 5088 generic.go:356] "Generic (PLEG): container finished" podID="ec80a8b1-509a-4c56-a03c-f5a2a5e3f720" containerID="c89fbb0ffd35658bc753eb97cbe3d5cba0665a9f3a0b67a978964abfccab9945" exitCode=0 Oct 30 20:06:54.373690 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:54.373262 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a61055-88j9x" event={"ID":"ec80a8b1-509a-4c56-a03c-f5a2a5e3f720","Type":"ContainerDied","Data":"c89fbb0ffd35658bc753eb97cbe3d5cba0665a9f3a0b67a978964abfccab9945"} Oct 30 20:06:54.375523 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:54.375496 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb6fdb9-6wq2f" event={"ID":"8f63941f-399d-48c0-8203-0bba97e49e8a","Type":"ContainerStarted","Data":"e79a786e6a101b0a665f50bd8978730341ad793ba9a856a525178087cdc5b306"} Oct 30 20:06:54.375696 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:54.375549 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb6fdb9-6wq2f" event={"ID":"8f63941f-399d-48c0-8203-0bba97e49e8a","Type":"ContainerStarted","Data":"07ed612525aa1063784c2fbe3eda246dd2c7d6b5e8cbde7287e4408428823ef9"} Oct 30 20:06:54.468251 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:54.468197 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb6fdb9-6wq2f" podStartSLOduration=4.468180499 podStartE2EDuration="4.468180499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:54.466131173 +0000 UTC m=+1175.878844955" watchObservedRunningTime="2026-10-30 20:06:54.468180499 +0000 UTC m=+1175.880894307" Oct 30 20:06:55.222407 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:55.222368 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" Oct 30 20:06:55.348433 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:55.346752 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" Oct 30 20:06:55.390961 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:55.389883 5088 generic.go:356] "Generic (PLEG): container finished" podID="8f63941f-399d-48c0-8203-0bba97e49e8a" containerID="e79a786e6a101b0a665f50bd8978730341ad793ba9a856a525178087cdc5b306" exitCode=0 Oct 30 20:06:55.390961 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:55.390725 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb6fdb9-6wq2f" event={"ID":"8f63941f-399d-48c0-8203-0bba97e49e8a","Type":"ContainerDied","Data":"e79a786e6a101b0a665f50bd8978730341ad793ba9a856a525178087cdc5b306"} Oct 30 20:06:55.437957 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:55.437924 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" Oct 30 20:06:55.698543 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:55.698514 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a61055-88j9x" Oct 30 20:06:55.792776 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:55.792680 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec80a8b1-509a-4c56-a03c-f5a2a5e3f720-kube-api-access-mg5b9\" (UniqueName: \"kubernetes.io/projected/ec80a8b1-509a-4c56-a03c-f5a2a5e3f720-kube-api-access-mg5b9\") pod \"ec80a8b1-509a-4c56-a03c-f5a2a5e3f720\" (UID: \"ec80a8b1-509a-4c56-a03c-f5a2a5e3f720\") " Oct 30 20:06:55.792955 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:55.792862 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec80a8b1-509a-4c56-a03c-f5a2a5e3f720-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec80a8b1-509a-4c56-a03c-f5a2a5e3f720-maas-api-openshift-ca\") pod \"ec80a8b1-509a-4c56-a03c-f5a2a5e3f720\" (UID: \"ec80a8b1-509a-4c56-a03c-f5a2a5e3f720\") " Oct 30 20:06:55.793033 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:55.792994 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec80a8b1-509a-4c56-a03c-f5a2a5e3f720-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec80a8b1-509a-4c56-a03c-f5a2a5e3f720-maas-api-xks-ca\") pod \"ec80a8b1-509a-4c56-a03c-f5a2a5e3f720\" (UID: \"ec80a8b1-509a-4c56-a03c-f5a2a5e3f720\") " Oct 30 20:06:55.793427 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:55.793399 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec80a8b1-509a-4c56-a03c-f5a2a5e3f720-maas-api-openshift-ca" pod "ec80a8b1-509a-4c56-a03c-f5a2a5e3f720" (UID: "ec80a8b1-509a-4c56-a03c-f5a2a5e3f720"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:06:55.794164 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:55.794132 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec80a8b1-509a-4c56-a03c-f5a2a5e3f720-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:06:55.795331 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:55.795281 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec80a8b1-509a-4c56-a03c-f5a2a5e3f720-kube-api-access-mg5b9" pod "ec80a8b1-509a-4c56-a03c-f5a2a5e3f720" (UID: "ec80a8b1-509a-4c56-a03c-f5a2a5e3f720"). InnerVolumeSpecName "kube-api-access-mg5b9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:06:55.795710 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:55.795681 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec80a8b1-509a-4c56-a03c-f5a2a5e3f720-maas-api-xks-ca" pod "ec80a8b1-509a-4c56-a03c-f5a2a5e3f720" (UID: "ec80a8b1-509a-4c56-a03c-f5a2a5e3f720"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:06:55.895999 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:55.895966 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg5b9\" (UniqueName: \"kubernetes.io/projected/ec80a8b1-509a-4c56-a03c-f5a2a5e3f720-kube-api-access-mg5b9\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:06:55.895999 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:55.895996 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec80a8b1-509a-4c56-a03c-f5a2a5e3f720-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:06:56.400219 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:56.400186 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a61055-88j9x" Oct 30 20:06:56.400219 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:56.400206 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a61055-88j9x" event={"ID":"ec80a8b1-509a-4c56-a03c-f5a2a5e3f720","Type":"ContainerDied","Data":"3286f53199bc75ec1f264feede3bb6eb10a299d298d1de5caeb82ad3116d7206"} Oct 30 20:06:56.400747 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:56.400281 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3286f53199bc75ec1f264feede3bb6eb10a299d298d1de5caeb82ad3116d7206" Oct 30 20:06:56.624730 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:56.624428 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb6fdb9-6wq2f" Oct 30 20:06:56.752694 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:56.752664 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f63941f-399d-48c0-8203-0bba97e49e8a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f63941f-399d-48c0-8203-0bba97e49e8a-maas-api-openshift-ca\") pod \"8f63941f-399d-48c0-8203-0bba97e49e8a\" (UID: \"8f63941f-399d-48c0-8203-0bba97e49e8a\") " Oct 30 20:06:56.752894 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:56.752734 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f63941f-399d-48c0-8203-0bba97e49e8a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f63941f-399d-48c0-8203-0bba97e49e8a-maas-api-xks-ca\") pod \"8f63941f-399d-48c0-8203-0bba97e49e8a\" (UID: \"8f63941f-399d-48c0-8203-0bba97e49e8a\") " Oct 30 20:06:56.752894 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:56.752797 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f63941f-399d-48c0-8203-0bba97e49e8a-kube-api-access-b6mdf\" (UniqueName: \"kubernetes.io/projected/8f63941f-399d-48c0-8203-0bba97e49e8a-kube-api-access-b6mdf\") pod \"8f63941f-399d-48c0-8203-0bba97e49e8a\" (UID: \"8f63941f-399d-48c0-8203-0bba97e49e8a\") " Oct 30 20:06:56.753065 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:56.753034 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f63941f-399d-48c0-8203-0bba97e49e8a-maas-api-openshift-ca" pod "8f63941f-399d-48c0-8203-0bba97e49e8a" (UID: "8f63941f-399d-48c0-8203-0bba97e49e8a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:06:56.753829 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:56.753800 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f63941f-399d-48c0-8203-0bba97e49e8a-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:06:56.755679 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:56.755647 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f63941f-399d-48c0-8203-0bba97e49e8a-kube-api-access-b6mdf" pod "8f63941f-399d-48c0-8203-0bba97e49e8a" (UID: "8f63941f-399d-48c0-8203-0bba97e49e8a"). InnerVolumeSpecName "kube-api-access-b6mdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:06:56.759940 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:56.759904 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f63941f-399d-48c0-8203-0bba97e49e8a-maas-api-xks-ca" pod "8f63941f-399d-48c0-8203-0bba97e49e8a" (UID: "8f63941f-399d-48c0-8203-0bba97e49e8a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:06:56.856177 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:56.856142 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f63941f-399d-48c0-8203-0bba97e49e8a-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:06:56.856177 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:56.856172 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6mdf\" (UniqueName: \"kubernetes.io/projected/8f63941f-399d-48c0-8203-0bba97e49e8a-kube-api-access-b6mdf\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:06:57.498632 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:57.491504 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb6fdb9-6wq2f" Oct 30 20:06:57.505636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:57.504818 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb6fdb9-6wq2f" event={"ID":"8f63941f-399d-48c0-8203-0bba97e49e8a","Type":"ContainerDied","Data":"07ed612525aa1063784c2fbe3eda246dd2c7d6b5e8cbde7287e4408428823ef9"} Oct 30 20:06:57.505636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:57.504898 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07ed612525aa1063784c2fbe3eda246dd2c7d6b5e8cbde7287e4408428823ef9" Oct 30 20:06:59.878117 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:59.878073 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77119bdd-lg99n"] Oct 30 20:06:59.878826 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:59.878810 5088 cpu_manager.go:414] "containerMap: removing container" podUID="ec80a8b1-509a-4c56-a03c-f5a2a5e3f720" containerName="revoke-keys" Oct 30 20:06:59.878876 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:59.878828 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec80a8b1-509a-4c56-a03c-f5a2a5e3f720" containerName="revoke-keys" Oct 30 20:06:59.878917 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:59.878880 5088 cpu_manager.go:414] "containerMap: removing container" podUID="8f63941f-399d-48c0-8203-0bba97e49e8a" containerName="revoke-keys" Oct 30 20:06:59.878917 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:59.878886 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f63941f-399d-48c0-8203-0bba97e49e8a" containerName="revoke-keys" Oct 30 20:06:59.879008 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:59.878999 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f63941f-399d-48c0-8203-0bba97e49e8a" containerName="revoke-keys" Oct 30 20:06:59.879043 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:59.879013 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec80a8b1-509a-4c56-a03c-f5a2a5e3f720" containerName="revoke-keys" Oct 30 20:06:59.884477 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:59.884453 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77119bdd-lg99n" Oct 30 20:06:59.903097 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:59.903064 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77119bdd-lg99n"] Oct 30 20:06:59.910031 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:59.909981 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:06:59.910211 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:59.909985 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:06:59.955731 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:59.955686 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cec7595d-0333-45e9-9145-1a5ce41648b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77119bdd-lg99n\" (UID: \"cec7595d-0333-45e9-9145-1a5ce41648b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77119bdd-lg99n" Oct 30 20:06:59.956093 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:59.956062 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttvvm\" (UniqueName: \"kubernetes.io/projected/cec7595d-0333-45e9-9145-1a5ce41648b8-kube-api-access-ttvvm\") pod \"maas-api-revoke-sub-77119bdd-lg99n\" (UID: \"cec7595d-0333-45e9-9145-1a5ce41648b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77119bdd-lg99n" Oct 30 20:06:59.956269 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:06:59.956237 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cec7595d-0333-45e9-9145-1a5ce41648b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77119bdd-lg99n\" (UID: \"cec7595d-0333-45e9-9145-1a5ce41648b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77119bdd-lg99n" Oct 30 20:07:00.058270 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:00.058229 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cec7595d-0333-45e9-9145-1a5ce41648b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77119bdd-lg99n\" (UID: \"cec7595d-0333-45e9-9145-1a5ce41648b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77119bdd-lg99n" Oct 30 20:07:00.058455 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:00.058439 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttvvm\" (UniqueName: \"kubernetes.io/projected/cec7595d-0333-45e9-9145-1a5ce41648b8-kube-api-access-ttvvm\") pod \"maas-api-revoke-sub-77119bdd-lg99n\" (UID: \"cec7595d-0333-45e9-9145-1a5ce41648b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77119bdd-lg99n" Oct 30 20:07:00.058531 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:00.058516 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cec7595d-0333-45e9-9145-1a5ce41648b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77119bdd-lg99n\" (UID: \"cec7595d-0333-45e9-9145-1a5ce41648b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77119bdd-lg99n" Oct 30 20:07:00.059199 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:00.059176 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cec7595d-0333-45e9-9145-1a5ce41648b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77119bdd-lg99n\" (UID: \"cec7595d-0333-45e9-9145-1a5ce41648b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77119bdd-lg99n" Oct 30 20:07:00.060669 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:00.060647 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cec7595d-0333-45e9-9145-1a5ce41648b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77119bdd-lg99n\" (UID: \"cec7595d-0333-45e9-9145-1a5ce41648b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77119bdd-lg99n" Oct 30 20:07:00.085115 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:00.085081 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttvvm\" (UniqueName: \"kubernetes.io/projected/cec7595d-0333-45e9-9145-1a5ce41648b8-kube-api-access-ttvvm\") pod \"maas-api-revoke-sub-77119bdd-lg99n\" (UID: \"cec7595d-0333-45e9-9145-1a5ce41648b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77119bdd-lg99n" Oct 30 20:07:00.199340 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:00.199248 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77119bdd-lg99n" Oct 30 20:07:00.517263 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:07:00.517233 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcec7595d_0333_45e9_9145_1a5ce41648b8.slice/crio-6a77d6160fef565c3c46ad3fde7a296978b018f3ce1af6f2706abb431e63f279 WatchSource:0}: Error finding container 6a77d6160fef565c3c46ad3fde7a296978b018f3ce1af6f2706abb431e63f279: Status 404 returned error can't find the container with id 6a77d6160fef565c3c46ad3fde7a296978b018f3ce1af6f2706abb431e63f279 Oct 30 20:07:00.517586 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:00.517559 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77119bdd-lg99n"] Oct 30 20:07:00.531524 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:00.531499 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77119bdd-lg99n" event={"ID":"cec7595d-0333-45e9-9145-1a5ce41648b8","Type":"ContainerStarted","Data":"6a77d6160fef565c3c46ad3fde7a296978b018f3ce1af6f2706abb431e63f279"} Oct 30 20:07:01.600380 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:01.600286 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77119bdd-lg99n" event={"ID":"cec7595d-0333-45e9-9145-1a5ce41648b8","Type":"ContainerStarted","Data":"0b23069896ab0c14ad059cb94401b5fb1dd48892d85bc6369d2142399d570068"} Oct 30 20:07:01.701190 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:01.701108 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77119bdd-lg99n" podStartSLOduration=2.701082066 podStartE2EDuration="2.701082066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:01.687868729 +0000 UTC m=+1183.100582509" watchObservedRunningTime="2026-10-30 20:07:01.701082066 +0000 UTC m=+1183.113795846" Oct 30 20:07:02.698640 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:02.695098 5088 generic.go:356] "Generic (PLEG): container finished" podID="cec7595d-0333-45e9-9145-1a5ce41648b8" containerID="0b23069896ab0c14ad059cb94401b5fb1dd48892d85bc6369d2142399d570068" exitCode=0 Oct 30 20:07:02.698640 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:02.695458 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77119bdd-lg99n" event={"ID":"cec7595d-0333-45e9-9145-1a5ce41648b8","Type":"ContainerDied","Data":"0b23069896ab0c14ad059cb94401b5fb1dd48892d85bc6369d2142399d570068"} Oct 30 20:07:03.396638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:03.395721 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" podUID="fec493c7-3f5a-459e-99e0-a4c816d5a907" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:03.534638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:03.510541 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:03.534638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:03.510972 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": context deadline exceeded" Oct 30 20:07:03.565639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:03.564709 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:03.655700 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:03.639954 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:03.821184 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:03.821132 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 20:07:03.944995 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:03.944887 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded" Oct 30 20:07:03.999579 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:03.999541 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:07:03.999900 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:03.999866 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" podUID="05a46855-a143-4c6d-8c87-404dc271bdb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:04.055762 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:04.055708 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:04.110638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:04.107067 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:04.214693 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:04.213713 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:04.214869 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:04.214828 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:04.419638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:04.400703 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" podUID="fec493c7-3f5a-459e-99e0-a4c816d5a907" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:04.419638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:04.401576 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" Oct 30 20:07:04.663145 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:04.660699 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:04.663145 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:04.661364 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 20:07:04.663145 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:04.661633 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" Oct 30 20:07:04.663145 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:04.661693 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" Oct 30 20:07:04.663145 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:04.662363 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 30 20:07:04.663145 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:04.662745 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 20:07:05.149639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:05.121807 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:05.252704 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:05.226124 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:05.298705 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:05.291688 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:07:05.299025 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:05.298988 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" podUID="0b36276d-bc5b-4415-8cf9-461aee572f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:05.420509 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:05.394481 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 20:07:05.496336 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:05.421013 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:07:05.496667 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:05.496634 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" podUID="b56fca64-1150-4198-945e-7e830300c11d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:05.496828 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:05.421365 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:07:05.496982 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:05.496960 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" podUID="b55ec6e2-9392-4d8d-8340-ef12f674970c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:05.497187 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:05.436985 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" podUID="fec493c7-3f5a-459e-99e0-a4c816d5a907" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:05.559603 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:05.559551 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:07:05.559603 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:05.614814 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" podUID="419aecf7-a250-42ad-995b-e7ebd9630a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:05.559603 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:05.684265 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 20:07:05.559603 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:05.703001 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 30 20:07:05.980650 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:05.761113 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:05.980650 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:05.786719 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 20:07:06.132067 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:06.129823 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:06.262805 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:06.262752 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:06.332639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:06.272936 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" podUID="fec493c7-3f5a-459e-99e0-a4c816d5a907" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:06.486195 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:06.332916 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wtlm7" podUID="419ffdb0-c223-4426-872a-50e3073812e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded" Oct 30 20:07:06.504640 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:07:06.379148 5088 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.63s" Oct 30 20:07:06.545701 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:06.539957 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" Oct 30 20:07:06.545701 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:06.540044 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:07:06.545701 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:06.540236 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:07:06.545701 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:06.540294 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:07:06.545701 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:06.478094 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 20:07:06.545701 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:06.502886 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" podUID="fec493c7-3f5a-459e-99e0-a4c816d5a907" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:06.550269 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:06.535746 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:06.686883 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:06.686835 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 20:07:06.898727 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:06.766702 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 20:07:06.898727 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:06.766785 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:07.117639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:07.080668 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:07.173733 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:07.133274 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:07.332679 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:07.328732 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:07.542213 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:07.542158 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:07.546286 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:07.546246 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" podUID="fec493c7-3f5a-459e-99e0-a4c816d5a907" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": context deadline exceeded" Oct 30 20:07:07.572234 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:07.571850 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77119bdd-lg99n" Oct 30 20:07:07.836698 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:07.811567 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 20:07:07.839506 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:07.814713 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:07.839506 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:07.815073 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:07.871968 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:07.871916 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cec7595d-0333-45e9-9145-1a5ce41648b8-kube-api-access-ttvvm\" (UniqueName: \"kubernetes.io/projected/cec7595d-0333-45e9-9145-1a5ce41648b8-kube-api-access-ttvvm\") pod \"cec7595d-0333-45e9-9145-1a5ce41648b8\" (UID: \"cec7595d-0333-45e9-9145-1a5ce41648b8\") " Oct 30 20:07:07.872284 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:07.872247 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cec7595d-0333-45e9-9145-1a5ce41648b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cec7595d-0333-45e9-9145-1a5ce41648b8-maas-api-xks-ca\") pod \"cec7595d-0333-45e9-9145-1a5ce41648b8\" (UID: \"cec7595d-0333-45e9-9145-1a5ce41648b8\") " Oct 30 20:07:07.908988 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:07.908939 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cec7595d-0333-45e9-9145-1a5ce41648b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cec7595d-0333-45e9-9145-1a5ce41648b8-maas-api-openshift-ca\") pod \"cec7595d-0333-45e9-9145-1a5ce41648b8\" (UID: \"cec7595d-0333-45e9-9145-1a5ce41648b8\") " Oct 30 20:07:07.917142 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:07.917094 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cec7595d-0333-45e9-9145-1a5ce41648b8-maas-api-openshift-ca" pod "cec7595d-0333-45e9-9145-1a5ce41648b8" (UID: "cec7595d-0333-45e9-9145-1a5ce41648b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:07:07.923880 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:07.923838 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cec7595d-0333-45e9-9145-1a5ce41648b8-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:07:07.941284 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:07.941242 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:07:07.941587 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:07.941468 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" podUID="a255c4a5-c090-49d2-84ba-536dfb9b8a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:07.971735 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:07.969121 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cec7595d-0333-45e9-9145-1a5ce41648b8-maas-api-xks-ca" pod "cec7595d-0333-45e9-9145-1a5ce41648b8" (UID: "cec7595d-0333-45e9-9145-1a5ce41648b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:07:08.064058 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:08.027752 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77119bdd-lg99n" event={"ID":"cec7595d-0333-45e9-9145-1a5ce41648b8","Type":"ContainerDied","Data":"6a77d6160fef565c3c46ad3fde7a296978b018f3ce1af6f2706abb431e63f279"} Oct 30 20:07:08.064058 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:08.027831 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a77d6160fef565c3c46ad3fde7a296978b018f3ce1af6f2706abb431e63f279" Oct 30 20:07:08.064058 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:08.027912 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77119bdd-lg99n" Oct 30 20:07:08.133005 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:08.081941 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:08.133005 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:08.086282 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cec7595d-0333-45e9-9145-1a5ce41648b8-kube-api-access-ttvvm" pod "cec7595d-0333-45e9-9145-1a5ce41648b8" (UID: "cec7595d-0333-45e9-9145-1a5ce41648b8"). InnerVolumeSpecName "kube-api-access-ttvvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:07:08.133005 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:08.112799 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttvvm\" (UniqueName: \"kubernetes.io/projected/cec7595d-0333-45e9-9145-1a5ce41648b8-kube-api-access-ttvvm\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:07:08.133005 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:08.112835 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cec7595d-0333-45e9-9145-1a5ce41648b8-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:07:08.226700 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:08.138660 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:08.362455 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:08.361246 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 20:07:08.557224 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:08.556218 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" podUID="fec493c7-3f5a-459e-99e0-a4c816d5a907" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": context deadline exceeded" Oct 30 20:07:08.558796 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:08.558310 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 20:07:08.570092 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:08.569742 5088 patch_prober.go:29] interesting pod/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:07:08.570092 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:08.569807 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" podUID="d44744f7-8f4c-4c79-978b-52649ce5a5ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:08.727318 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:08.725723 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:07:08.727318 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:08.725791 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" podUID="98275e84-2e20-46db-8264-27ec2ac2e2f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:08.838953 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:08.838851 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:08.844681 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:08.839476 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:08.845236 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:08.839071 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 20:07:08.895397 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:08.852190 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:09.093182 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:09.092854 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:09.275651 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:09.243561 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:09.370701 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:09.366407 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 20:07:09.453124 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:09.452833 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": context deadline exceeded" Oct 30 20:07:09.590858 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:09.573060 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:09.639966 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:09.600347 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" podUID="fec493c7-3f5a-459e-99e0-a4c816d5a907" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:09.639966 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:09.623899 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:07:09.639966 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:09.623959 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" podUID="f4b44c6e-891a-4073-9192-035aeaad10a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:09.870957 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:09.870910 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:09.871561 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:09.871204 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 20:07:09.872090 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:09.871227 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 30 20:07:10.352666 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:10.342291 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:10.383196 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:10.355112 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:10.401632 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:10.385317 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:10.573461 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:10.573351 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:10.611440 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:10.611397 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" podUID="fec493c7-3f5a-459e-99e0-a4c816d5a907" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": context deadline exceeded" Oct 30 20:07:10.872168 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:10.872035 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:10.872930 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:10.872475 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 20:07:10.999331 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:10.999288 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r"] Oct 30 20:07:11.216070 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:11.215962 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:07:11.277553 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:11.277507 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:11.384408 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:11.384363 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" Oct 30 20:07:11.443879 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:11.443844 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" Oct 30 20:07:11.455447 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:11.443006 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:11.524447 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:11.524416 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 20:07:11.578341 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:11.578304 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 20:07:11.603302 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:11.603260 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:07:11.608002 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:11.607971 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" Oct 30 20:07:11.637265 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:11.637237 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:07:11.643520 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:11.643496 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" Oct 30 20:07:11.729002 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:11.728973 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1de1a9bb-lqpq5"] Oct 30 20:07:11.729847 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:11.729826 5088 cpu_manager.go:414] "containerMap: removing container" podUID="cec7595d-0333-45e9-9145-1a5ce41648b8" containerName="revoke-keys" Oct 30 20:07:11.729847 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:11.729845 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cec7595d-0333-45e9-9145-1a5ce41648b8" containerName="revoke-keys" Oct 30 20:07:11.730025 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:11.729973 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="cec7595d-0333-45e9-9145-1a5ce41648b8" containerName="revoke-keys" Oct 30 20:07:11.735282 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:11.735263 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de1a9bb-lqpq5" Oct 30 20:07:11.757392 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:11.757369 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:07:11.757565 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:11.757548 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:07:11.802234 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:11.802147 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6404ea32-98aa-4ca9-bf40-842e68f949cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1de1a9bb-lqpq5\" (UID: \"6404ea32-98aa-4ca9-bf40-842e68f949cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de1a9bb-lqpq5" Oct 30 20:07:11.802510 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:11.802484 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59w7c\" (UniqueName: \"kubernetes.io/projected/6404ea32-98aa-4ca9-bf40-842e68f949cb-kube-api-access-59w7c\") pod \"maas-api-revoke-sub-1de1a9bb-lqpq5\" (UID: \"6404ea32-98aa-4ca9-bf40-842e68f949cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de1a9bb-lqpq5" Oct 30 20:07:11.802681 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:11.802565 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6404ea32-98aa-4ca9-bf40-842e68f949cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1de1a9bb-lqpq5\" (UID: \"6404ea32-98aa-4ca9-bf40-842e68f949cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de1a9bb-lqpq5" Oct 30 20:07:11.810708 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:11.810607 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1de1a9bb-lqpq5"] Oct 30 20:07:11.816116 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:11.816080 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-626870-7fcn9"] Oct 30 20:07:11.822122 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:11.822096 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-626870-7fcn9" Oct 30 20:07:11.853784 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:11.853752 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-626870-7fcn9"] Oct 30 20:07:11.904722 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:11.904688 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74c24a9b-9591-4478-a762-58940c1438f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-626870-7fcn9\" (UID: \"74c24a9b-9591-4478-a762-58940c1438f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-626870-7fcn9" Oct 30 20:07:11.904925 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:11.904749 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74c24a9b-9591-4478-a762-58940c1438f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-626870-7fcn9\" (UID: \"74c24a9b-9591-4478-a762-58940c1438f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-626870-7fcn9" Oct 30 20:07:11.905010 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:11.904948 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6404ea32-98aa-4ca9-bf40-842e68f949cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1de1a9bb-lqpq5\" (UID: \"6404ea32-98aa-4ca9-bf40-842e68f949cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de1a9bb-lqpq5" Oct 30 20:07:11.905010 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:11.904997 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59w7c\" (UniqueName: \"kubernetes.io/projected/6404ea32-98aa-4ca9-bf40-842e68f949cb-kube-api-access-59w7c\") pod \"maas-api-revoke-sub-1de1a9bb-lqpq5\" (UID: \"6404ea32-98aa-4ca9-bf40-842e68f949cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de1a9bb-lqpq5" Oct 30 20:07:11.905176 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:11.905045 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6404ea32-98aa-4ca9-bf40-842e68f949cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1de1a9bb-lqpq5\" (UID: \"6404ea32-98aa-4ca9-bf40-842e68f949cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de1a9bb-lqpq5" Oct 30 20:07:11.905176 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:11.905122 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4lhk\" (UniqueName: \"kubernetes.io/projected/74c24a9b-9591-4478-a762-58940c1438f3-kube-api-access-h4lhk\") pod \"maas-api-revoke-keys-e2e-worker-w5-626870-7fcn9\" (UID: \"74c24a9b-9591-4478-a762-58940c1438f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-626870-7fcn9" Oct 30 20:07:11.905880 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:11.905839 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6404ea32-98aa-4ca9-bf40-842e68f949cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1de1a9bb-lqpq5\" (UID: \"6404ea32-98aa-4ca9-bf40-842e68f949cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de1a9bb-lqpq5" Oct 30 20:07:11.908236 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:11.908210 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6404ea32-98aa-4ca9-bf40-842e68f949cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1de1a9bb-lqpq5\" (UID: \"6404ea32-98aa-4ca9-bf40-842e68f949cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de1a9bb-lqpq5" Oct 30 20:07:11.950703 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:11.950660 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59w7c\" (UniqueName: \"kubernetes.io/projected/6404ea32-98aa-4ca9-bf40-842e68f949cb-kube-api-access-59w7c\") pod \"maas-api-revoke-sub-1de1a9bb-lqpq5\" (UID: \"6404ea32-98aa-4ca9-bf40-842e68f949cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de1a9bb-lqpq5" Oct 30 20:07:12.010012 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:12.009976 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4lhk\" (UniqueName: \"kubernetes.io/projected/74c24a9b-9591-4478-a762-58940c1438f3-kube-api-access-h4lhk\") pod \"maas-api-revoke-keys-e2e-worker-w5-626870-7fcn9\" (UID: \"74c24a9b-9591-4478-a762-58940c1438f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-626870-7fcn9" Oct 30 20:07:12.010228 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:12.010187 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74c24a9b-9591-4478-a762-58940c1438f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-626870-7fcn9\" (UID: \"74c24a9b-9591-4478-a762-58940c1438f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-626870-7fcn9" Oct 30 20:07:12.010228 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:12.010219 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74c24a9b-9591-4478-a762-58940c1438f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-626870-7fcn9\" (UID: \"74c24a9b-9591-4478-a762-58940c1438f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-626870-7fcn9" Oct 30 20:07:12.011066 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:12.011038 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74c24a9b-9591-4478-a762-58940c1438f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-626870-7fcn9\" (UID: \"74c24a9b-9591-4478-a762-58940c1438f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-626870-7fcn9" Oct 30 20:07:12.013210 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:12.013186 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74c24a9b-9591-4478-a762-58940c1438f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-626870-7fcn9\" (UID: \"74c24a9b-9591-4478-a762-58940c1438f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-626870-7fcn9" Oct 30 20:07:12.040481 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:12.040451 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4lhk\" (UniqueName: \"kubernetes.io/projected/74c24a9b-9591-4478-a762-58940c1438f3-kube-api-access-h4lhk\") pod \"maas-api-revoke-keys-e2e-worker-w5-626870-7fcn9\" (UID: \"74c24a9b-9591-4478-a762-58940c1438f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-626870-7fcn9" Oct 30 20:07:12.050315 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:12.050282 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de1a9bb-lqpq5" Oct 30 20:07:12.140697 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:12.140067 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-626870-7fcn9" Oct 30 20:07:12.408040 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:07:12.408006 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6404ea32_98aa_4ca9_bf40_842e68f949cb.slice/crio-870f5cede02af2973b854708b68a17d24b8b3e9bfd996ff1ea4b387b250d7a4e WatchSource:0}: Error finding container 870f5cede02af2973b854708b68a17d24b8b3e9bfd996ff1ea4b387b250d7a4e: Status 404 returned error can't find the container with id 870f5cede02af2973b854708b68a17d24b8b3e9bfd996ff1ea4b387b250d7a4e Oct 30 20:07:12.409952 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:12.409925 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1de1a9bb-lqpq5"] Oct 30 20:07:12.485739 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:12.485707 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-626870-7fcn9"] Oct 30 20:07:12.492825 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:07:12.492783 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74c24a9b_9591_4478_a762_58940c1438f3.slice/crio-9fee02e706e1c6d77800da538735305c0b756c25c9c40ae1de8d0f3608322d21 WatchSource:0}: Error finding container 9fee02e706e1c6d77800da538735305c0b756c25c9c40ae1de8d0f3608322d21: Status 404 returned error can't find the container with id 9fee02e706e1c6d77800da538735305c0b756c25c9c40ae1de8d0f3608322d21 Oct 30 20:07:12.647467 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:12.647426 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de1a9bb-lqpq5" event={"ID":"6404ea32-98aa-4ca9-bf40-842e68f949cb","Type":"ContainerStarted","Data":"870f5cede02af2973b854708b68a17d24b8b3e9bfd996ff1ea4b387b250d7a4e"} Oct 30 20:07:12.649260 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:12.649231 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-626870-7fcn9" event={"ID":"74c24a9b-9591-4478-a762-58940c1438f3","Type":"ContainerStarted","Data":"9fee02e706e1c6d77800da538735305c0b756c25c9c40ae1de8d0f3608322d21"} Oct 30 20:07:13.659603 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:13.659563 5088 generic.go:356] "Generic (PLEG): container finished" podID="74c24a9b-9591-4478-a762-58940c1438f3" containerID="6cae5d08d673b861727974c76117397e49a5786137f44879ea6594942e5ba40d" exitCode=0 Oct 30 20:07:13.660505 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:13.660435 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-626870-7fcn9" event={"ID":"74c24a9b-9591-4478-a762-58940c1438f3","Type":"ContainerDied","Data":"6cae5d08d673b861727974c76117397e49a5786137f44879ea6594942e5ba40d"} Oct 30 20:07:13.661652 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:13.661595 5088 generic.go:356] "Generic (PLEG): container finished" podID="6404ea32-98aa-4ca9-bf40-842e68f949cb" containerID="519936d87a96adac40aefbf4d5be3677a34523091745ae557b3e019d8b318ed4" exitCode=0 Oct 30 20:07:13.661757 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:13.661646 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de1a9bb-lqpq5" event={"ID":"6404ea32-98aa-4ca9-bf40-842e68f949cb","Type":"ContainerDied","Data":"519936d87a96adac40aefbf4d5be3677a34523091745ae557b3e019d8b318ed4"} Oct 30 20:07:14.905023 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:14.904992 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-626870-7fcn9" Oct 30 20:07:14.908756 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:14.908735 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de1a9bb-lqpq5" Oct 30 20:07:15.091720 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:15.091537 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6404ea32-98aa-4ca9-bf40-842e68f949cb-kube-api-access-59w7c\" (UniqueName: \"kubernetes.io/projected/6404ea32-98aa-4ca9-bf40-842e68f949cb-kube-api-access-59w7c\") pod \"6404ea32-98aa-4ca9-bf40-842e68f949cb\" (UID: \"6404ea32-98aa-4ca9-bf40-842e68f949cb\") " Oct 30 20:07:15.091720 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:15.091629 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74c24a9b-9591-4478-a762-58940c1438f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74c24a9b-9591-4478-a762-58940c1438f3-maas-api-xks-ca\") pod \"74c24a9b-9591-4478-a762-58940c1438f3\" (UID: \"74c24a9b-9591-4478-a762-58940c1438f3\") " Oct 30 20:07:15.091990 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:15.091748 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6404ea32-98aa-4ca9-bf40-842e68f949cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6404ea32-98aa-4ca9-bf40-842e68f949cb-maas-api-xks-ca\") pod \"6404ea32-98aa-4ca9-bf40-842e68f949cb\" (UID: \"6404ea32-98aa-4ca9-bf40-842e68f949cb\") " Oct 30 20:07:15.091990 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:15.091914 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74c24a9b-9591-4478-a762-58940c1438f3-kube-api-access-h4lhk\" (UniqueName: \"kubernetes.io/projected/74c24a9b-9591-4478-a762-58940c1438f3-kube-api-access-h4lhk\") pod \"74c24a9b-9591-4478-a762-58940c1438f3\" (UID: \"74c24a9b-9591-4478-a762-58940c1438f3\") " Oct 30 20:07:15.092290 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:15.092256 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74c24a9b-9591-4478-a762-58940c1438f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74c24a9b-9591-4478-a762-58940c1438f3-maas-api-openshift-ca\") pod \"74c24a9b-9591-4478-a762-58940c1438f3\" (UID: \"74c24a9b-9591-4478-a762-58940c1438f3\") " Oct 30 20:07:15.092402 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:15.092321 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6404ea32-98aa-4ca9-bf40-842e68f949cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6404ea32-98aa-4ca9-bf40-842e68f949cb-maas-api-openshift-ca\") pod \"6404ea32-98aa-4ca9-bf40-842e68f949cb\" (UID: \"6404ea32-98aa-4ca9-bf40-842e68f949cb\") " Oct 30 20:07:15.094487 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:15.093761 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6404ea32-98aa-4ca9-bf40-842e68f949cb-maas-api-openshift-ca" pod "6404ea32-98aa-4ca9-bf40-842e68f949cb" (UID: "6404ea32-98aa-4ca9-bf40-842e68f949cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:07:15.094487 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:15.094110 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74c24a9b-9591-4478-a762-58940c1438f3-maas-api-openshift-ca" pod "74c24a9b-9591-4478-a762-58940c1438f3" (UID: "74c24a9b-9591-4478-a762-58940c1438f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:07:15.122899 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:15.122781 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74c24a9b-9591-4478-a762-58940c1438f3-maas-api-xks-ca" pod "74c24a9b-9591-4478-a762-58940c1438f3" (UID: "74c24a9b-9591-4478-a762-58940c1438f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:07:15.135811 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:15.135756 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74c24a9b-9591-4478-a762-58940c1438f3-kube-api-access-h4lhk" pod "74c24a9b-9591-4478-a762-58940c1438f3" (UID: "74c24a9b-9591-4478-a762-58940c1438f3"). InnerVolumeSpecName "kube-api-access-h4lhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:07:15.160522 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:15.160470 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6404ea32-98aa-4ca9-bf40-842e68f949cb-kube-api-access-59w7c" pod "6404ea32-98aa-4ca9-bf40-842e68f949cb" (UID: "6404ea32-98aa-4ca9-bf40-842e68f949cb"). InnerVolumeSpecName "kube-api-access-59w7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:07:15.164192 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:15.164134 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6404ea32-98aa-4ca9-bf40-842e68f949cb-maas-api-xks-ca" pod "6404ea32-98aa-4ca9-bf40-842e68f949cb" (UID: "6404ea32-98aa-4ca9-bf40-842e68f949cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:07:15.194773 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:15.194740 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4lhk\" (UniqueName: \"kubernetes.io/projected/74c24a9b-9591-4478-a762-58940c1438f3-kube-api-access-h4lhk\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:07:15.194773 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:15.194772 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74c24a9b-9591-4478-a762-58940c1438f3-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:07:15.194959 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:15.194785 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6404ea32-98aa-4ca9-bf40-842e68f949cb-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:07:15.194959 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:15.194797 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59w7c\" (UniqueName: \"kubernetes.io/projected/6404ea32-98aa-4ca9-bf40-842e68f949cb-kube-api-access-59w7c\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:07:15.194959 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:15.194811 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74c24a9b-9591-4478-a762-58940c1438f3-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:07:15.194959 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:15.194823 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6404ea32-98aa-4ca9-bf40-842e68f949cb-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:07:15.682569 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:15.682539 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-626870-7fcn9" Oct 30 20:07:15.682761 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:15.682568 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-626870-7fcn9" event={"ID":"74c24a9b-9591-4478-a762-58940c1438f3","Type":"ContainerDied","Data":"9fee02e706e1c6d77800da538735305c0b756c25c9c40ae1de8d0f3608322d21"} Oct 30 20:07:15.682761 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:15.682653 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fee02e706e1c6d77800da538735305c0b756c25c9c40ae1de8d0f3608322d21" Oct 30 20:07:15.684282 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:15.684255 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de1a9bb-lqpq5" Oct 30 20:07:15.684424 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:15.684291 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de1a9bb-lqpq5" event={"ID":"6404ea32-98aa-4ca9-bf40-842e68f949cb","Type":"ContainerDied","Data":"870f5cede02af2973b854708b68a17d24b8b3e9bfd996ff1ea4b387b250d7a4e"} Oct 30 20:07:15.684424 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:15.684333 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="870f5cede02af2973b854708b68a17d24b8b3e9bfd996ff1ea4b387b250d7a4e" Oct 30 20:07:16.574641 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:16.574593 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj"] Oct 30 20:07:18.881523 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:18.881492 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qdj4h_da850825-59ab-480f-9047-afd7d2bdc19f/console-operator/1.log" Oct 30 20:07:18.882722 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:18.882699 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qdj4h_da850825-59ab-480f-9047-afd7d2bdc19f/console-operator/1.log" Oct 30 20:07:18.886354 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:18.886331 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xpjpn_688d079a-932d-4f63-97da-5cd6222a03e1/ovn-acl-logging/0.log" Oct 30 20:07:18.887349 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:18.887317 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xpjpn_688d079a-932d-4f63-97da-5cd6222a03e1/ovn-acl-logging/0.log" Oct 30 20:07:19.062571 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:19.062533 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e92c58f0-jzwmm"] Oct 30 20:07:19.075549 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:19.075515 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e92c58f0-jzwmm"] Oct 30 20:07:20.743342 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:20.743302 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="908e797e-bda5-4f61-ac6a-4c8c63638034" path="/var/lib/kubelet/pods/908e797e-bda5-4f61-ac6a-4c8c63638034/volumes" Oct 30 20:07:22.226655 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:22.225679 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9aac2627-nz2jn"] Oct 30 20:07:22.227244 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:22.226899 5088 cpu_manager.go:414] "containerMap: removing container" podUID="74c24a9b-9591-4478-a762-58940c1438f3" containerName="revoke-keys" Oct 30 20:07:22.227244 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:22.226923 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74c24a9b-9591-4478-a762-58940c1438f3" containerName="revoke-keys" Oct 30 20:07:22.227244 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:22.226952 5088 cpu_manager.go:414] "containerMap: removing container" podUID="6404ea32-98aa-4ca9-bf40-842e68f949cb" containerName="revoke-keys" Oct 30 20:07:22.227244 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:22.226962 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6404ea32-98aa-4ca9-bf40-842e68f949cb" containerName="revoke-keys" Oct 30 20:07:22.227244 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:22.227208 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="74c24a9b-9591-4478-a762-58940c1438f3" containerName="revoke-keys" Oct 30 20:07:22.227244 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:22.227228 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="6404ea32-98aa-4ca9-bf40-842e68f949cb" containerName="revoke-keys" Oct 30 20:07:22.256706 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:22.249556 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aac2627-nz2jn" Oct 30 20:07:22.278637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:22.272491 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:07:22.300638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:22.294884 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:07:22.398493 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:22.393972 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7754459-8e35-4d81-a0e8-325f39cd7c8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9aac2627-nz2jn\" (UID: \"f7754459-8e35-4d81-a0e8-325f39cd7c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aac2627-nz2jn" Oct 30 20:07:22.398493 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:22.394156 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7754459-8e35-4d81-a0e8-325f39cd7c8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9aac2627-nz2jn\" (UID: \"f7754459-8e35-4d81-a0e8-325f39cd7c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aac2627-nz2jn" Oct 30 20:07:22.398493 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:22.394202 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp5pm\" (UniqueName: \"kubernetes.io/projected/f7754459-8e35-4d81-a0e8-325f39cd7c8b-kube-api-access-zp5pm\") pod \"maas-api-revoke-sub-9aac2627-nz2jn\" (UID: \"f7754459-8e35-4d81-a0e8-325f39cd7c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aac2627-nz2jn" Oct 30 20:07:22.432216 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:22.423584 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9aac2627-nz2jn"] Oct 30 20:07:22.519845 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:22.498780 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7754459-8e35-4d81-a0e8-325f39cd7c8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9aac2627-nz2jn\" (UID: \"f7754459-8e35-4d81-a0e8-325f39cd7c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aac2627-nz2jn" Oct 30 20:07:22.519845 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:22.498844 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp5pm\" (UniqueName: \"kubernetes.io/projected/f7754459-8e35-4d81-a0e8-325f39cd7c8b-kube-api-access-zp5pm\") pod \"maas-api-revoke-sub-9aac2627-nz2jn\" (UID: \"f7754459-8e35-4d81-a0e8-325f39cd7c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aac2627-nz2jn" Oct 30 20:07:22.519845 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:22.499145 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7754459-8e35-4d81-a0e8-325f39cd7c8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9aac2627-nz2jn\" (UID: \"f7754459-8e35-4d81-a0e8-325f39cd7c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aac2627-nz2jn" Oct 30 20:07:22.519845 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:22.500059 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7754459-8e35-4d81-a0e8-325f39cd7c8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9aac2627-nz2jn\" (UID: \"f7754459-8e35-4d81-a0e8-325f39cd7c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aac2627-nz2jn" Oct 30 20:07:22.526148 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:22.525940 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7754459-8e35-4d81-a0e8-325f39cd7c8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9aac2627-nz2jn\" (UID: \"f7754459-8e35-4d81-a0e8-325f39cd7c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aac2627-nz2jn" Oct 30 20:07:22.548011 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:22.547805 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp5pm\" (UniqueName: \"kubernetes.io/projected/f7754459-8e35-4d81-a0e8-325f39cd7c8b-kube-api-access-zp5pm\") pod \"maas-api-revoke-sub-9aac2627-nz2jn\" (UID: \"f7754459-8e35-4d81-a0e8-325f39cd7c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aac2627-nz2jn" Oct 30 20:07:22.562830 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:07:22.562785 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:07:22.789313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:22.789223 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aac2627-nz2jn" Oct 30 20:07:22.964670 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:22.964630 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2891b522-8gd2t"] Oct 30 20:07:22.972315 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:22.972278 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2891b522-8gd2t" Oct 30 20:07:23.304634 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:23.289577 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b45a2b75-a359-49fd-8f33-3086717b7c39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2891b522-8gd2t\" (UID: \"b45a2b75-a359-49fd-8f33-3086717b7c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2891b522-8gd2t" Oct 30 20:07:23.304634 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:23.289788 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhgk8\" (UniqueName: \"kubernetes.io/projected/b45a2b75-a359-49fd-8f33-3086717b7c39-kube-api-access-dhgk8\") pod \"maas-api-revoke-sub-2891b522-8gd2t\" (UID: \"b45a2b75-a359-49fd-8f33-3086717b7c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2891b522-8gd2t" Oct 30 20:07:23.401097 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:23.401050 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b45a2b75-a359-49fd-8f33-3086717b7c39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2891b522-8gd2t\" (UID: \"b45a2b75-a359-49fd-8f33-3086717b7c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2891b522-8gd2t" Oct 30 20:07:23.415785 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:23.415729 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2891b522-8gd2t"] Oct 30 20:07:23.439198 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:23.439159 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe47f907-snhsh"] Oct 30 20:07:23.448353 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:23.447639 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe47f907-snhsh"] Oct 30 20:07:23.448353 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:23.447737 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe47f907-snhsh" Oct 30 20:07:23.455974 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:23.455934 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9aac2627-nz2jn"] Oct 30 20:07:23.504343 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:23.504303 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b45a2b75-a359-49fd-8f33-3086717b7c39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2891b522-8gd2t\" (UID: \"b45a2b75-a359-49fd-8f33-3086717b7c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2891b522-8gd2t" Oct 30 20:07:23.504526 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:23.504456 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhgk8\" (UniqueName: \"kubernetes.io/projected/b45a2b75-a359-49fd-8f33-3086717b7c39-kube-api-access-dhgk8\") pod \"maas-api-revoke-sub-2891b522-8gd2t\" (UID: \"b45a2b75-a359-49fd-8f33-3086717b7c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2891b522-8gd2t" Oct 30 20:07:23.504602 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:23.504554 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r99h\" (UniqueName: \"kubernetes.io/projected/ea2533f2-6a01-44f1-a528-7b5e3eb8dde4-kube-api-access-9r99h\") pod \"maas-api-revoke-sub-fe47f907-snhsh\" (UID: \"ea2533f2-6a01-44f1-a528-7b5e3eb8dde4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe47f907-snhsh" Oct 30 20:07:23.504777 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:23.504756 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea2533f2-6a01-44f1-a528-7b5e3eb8dde4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe47f907-snhsh\" (UID: \"ea2533f2-6a01-44f1-a528-7b5e3eb8dde4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe47f907-snhsh" Oct 30 20:07:23.504850 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:23.504836 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b45a2b75-a359-49fd-8f33-3086717b7c39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2891b522-8gd2t\" (UID: \"b45a2b75-a359-49fd-8f33-3086717b7c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2891b522-8gd2t" Oct 30 20:07:23.504943 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:23.504923 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea2533f2-6a01-44f1-a528-7b5e3eb8dde4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe47f907-snhsh\" (UID: \"ea2533f2-6a01-44f1-a528-7b5e3eb8dde4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe47f907-snhsh" Oct 30 20:07:23.505451 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:23.505429 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b45a2b75-a359-49fd-8f33-3086717b7c39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2891b522-8gd2t\" (UID: \"b45a2b75-a359-49fd-8f33-3086717b7c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2891b522-8gd2t" Oct 30 20:07:23.506699 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:23.506682 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b45a2b75-a359-49fd-8f33-3086717b7c39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2891b522-8gd2t\" (UID: \"b45a2b75-a359-49fd-8f33-3086717b7c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2891b522-8gd2t" Oct 30 20:07:23.530871 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:23.530836 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhgk8\" (UniqueName: \"kubernetes.io/projected/b45a2b75-a359-49fd-8f33-3086717b7c39-kube-api-access-dhgk8\") pod \"maas-api-revoke-sub-2891b522-8gd2t\" (UID: \"b45a2b75-a359-49fd-8f33-3086717b7c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2891b522-8gd2t" Oct 30 20:07:23.606435 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:23.606349 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea2533f2-6a01-44f1-a528-7b5e3eb8dde4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe47f907-snhsh\" (UID: \"ea2533f2-6a01-44f1-a528-7b5e3eb8dde4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe47f907-snhsh" Oct 30 20:07:23.606606 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:23.606510 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r99h\" (UniqueName: \"kubernetes.io/projected/ea2533f2-6a01-44f1-a528-7b5e3eb8dde4-kube-api-access-9r99h\") pod \"maas-api-revoke-sub-fe47f907-snhsh\" (UID: \"ea2533f2-6a01-44f1-a528-7b5e3eb8dde4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe47f907-snhsh" Oct 30 20:07:23.606606 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:23.606640 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea2533f2-6a01-44f1-a528-7b5e3eb8dde4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe47f907-snhsh\" (UID: \"ea2533f2-6a01-44f1-a528-7b5e3eb8dde4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe47f907-snhsh" Oct 30 20:07:23.607138 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:23.607111 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea2533f2-6a01-44f1-a528-7b5e3eb8dde4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe47f907-snhsh\" (UID: \"ea2533f2-6a01-44f1-a528-7b5e3eb8dde4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe47f907-snhsh" Oct 30 20:07:23.609080 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:23.609055 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea2533f2-6a01-44f1-a528-7b5e3eb8dde4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe47f907-snhsh\" (UID: \"ea2533f2-6a01-44f1-a528-7b5e3eb8dde4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe47f907-snhsh" Oct 30 20:07:23.634501 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:23.634470 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r99h\" (UniqueName: \"kubernetes.io/projected/ea2533f2-6a01-44f1-a528-7b5e3eb8dde4-kube-api-access-9r99h\") pod \"maas-api-revoke-sub-fe47f907-snhsh\" (UID: \"ea2533f2-6a01-44f1-a528-7b5e3eb8dde4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe47f907-snhsh" Oct 30 20:07:23.640345 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:23.640322 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2891b522-8gd2t" Oct 30 20:07:23.781344 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:23.780041 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe47f907-snhsh" Oct 30 20:07:23.835636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:23.831269 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aac2627-nz2jn" event={"ID":"f7754459-8e35-4d81-a0e8-325f39cd7c8b","Type":"ContainerStarted","Data":"1b15a5153f404f94dbcc529e88af09998f69506f690eccd875aff60f4c6e1e15"} Oct 30 20:07:24.339667 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:24.336015 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2891b522-8gd2t"] Oct 30 20:07:24.596333 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:24.596309 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe47f907-snhsh"] Oct 30 20:07:24.841912 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:24.841874 5088 generic.go:356] "Generic (PLEG): container finished" podID="f7754459-8e35-4d81-a0e8-325f39cd7c8b" containerID="a46e16e144f3398c61b6b3e965e037d426be69a1363383e46b2bd2c012054c9d" exitCode=0 Oct 30 20:07:24.842115 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:24.841957 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aac2627-nz2jn" event={"ID":"f7754459-8e35-4d81-a0e8-325f39cd7c8b","Type":"ContainerDied","Data":"a46e16e144f3398c61b6b3e965e037d426be69a1363383e46b2bd2c012054c9d"} Oct 30 20:07:24.843816 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:24.843793 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2891b522-8gd2t" event={"ID":"b45a2b75-a359-49fd-8f33-3086717b7c39","Type":"ContainerStarted","Data":"c767e8356d00fca7d9b63b67a3a4b969852fd55132a8603436807b992e3d36a7"} Oct 30 20:07:24.843935 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:24.843835 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2891b522-8gd2t" event={"ID":"b45a2b75-a359-49fd-8f33-3086717b7c39","Type":"ContainerStarted","Data":"15dc028c33dc914b776b4cb3781d8589192420e5a5b89cf73c1537002accf857"} Oct 30 20:07:24.845333 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:24.845312 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe47f907-snhsh" event={"ID":"ea2533f2-6a01-44f1-a528-7b5e3eb8dde4","Type":"ContainerStarted","Data":"f5967f3894b76b16546433ec7b29ec772a03d110f60d405e6a9815bc809f1242"} Oct 30 20:07:24.940882 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:24.940822 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2891b522-8gd2t" podStartSLOduration=2.940804378 podStartE2EDuration="2.940804378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:24.933196573 +0000 UTC m=+1206.345910361" watchObservedRunningTime="2026-10-30 20:07:24.940804378 +0000 UTC m=+1206.353518159" Oct 30 20:07:25.768783 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:25.768748 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf"] Oct 30 20:07:25.784457 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:25.783952 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" Oct 30 20:07:25.807470 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:25.807338 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf"] Oct 30 20:07:25.837306 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:25.837275 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/811db7c8-11d5-4e9d-8c8b-254c17906e84-istio-data\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf\" (UID: \"811db7c8-11d5-4e9d-8c8b-254c17906e84\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" Oct 30 20:07:25.837436 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:25.837402 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/811db7c8-11d5-4e9d-8c8b-254c17906e84-istio-envoy\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf\" (UID: \"811db7c8-11d5-4e9d-8c8b-254c17906e84\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" Oct 30 20:07:25.837480 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:25.837469 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/811db7c8-11d5-4e9d-8c8b-254c17906e84-workload-certs\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf\" (UID: \"811db7c8-11d5-4e9d-8c8b-254c17906e84\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" Oct 30 20:07:25.837576 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:25.837564 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/811db7c8-11d5-4e9d-8c8b-254c17906e84-istio-podinfo\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf\" (UID: \"811db7c8-11d5-4e9d-8c8b-254c17906e84\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" Oct 30 20:07:25.837657 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:25.837639 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98t24\" (UniqueName: \"kubernetes.io/projected/811db7c8-11d5-4e9d-8c8b-254c17906e84-kube-api-access-98t24\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf\" (UID: \"811db7c8-11d5-4e9d-8c8b-254c17906e84\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" Oct 30 20:07:25.837706 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:25.837692 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/811db7c8-11d5-4e9d-8c8b-254c17906e84-istio-token\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf\" (UID: \"811db7c8-11d5-4e9d-8c8b-254c17906e84\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" Oct 30 20:07:25.837818 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:25.837807 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/811db7c8-11d5-4e9d-8c8b-254c17906e84-credential-socket\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf\" (UID: \"811db7c8-11d5-4e9d-8c8b-254c17906e84\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" Oct 30 20:07:25.837849 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:25.837827 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/811db7c8-11d5-4e9d-8c8b-254c17906e84-workload-socket\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf\" (UID: \"811db7c8-11d5-4e9d-8c8b-254c17906e84\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" Oct 30 20:07:25.837883 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:25.837852 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/811db7c8-11d5-4e9d-8c8b-254c17906e84-istiod-ca-cert\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf\" (UID: \"811db7c8-11d5-4e9d-8c8b-254c17906e84\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" Oct 30 20:07:25.855565 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:25.855527 5088 generic.go:356] "Generic (PLEG): container finished" podID="b45a2b75-a359-49fd-8f33-3086717b7c39" containerID="c767e8356d00fca7d9b63b67a3a4b969852fd55132a8603436807b992e3d36a7" exitCode=0 Oct 30 20:07:25.855757 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:25.855654 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2891b522-8gd2t" event={"ID":"b45a2b75-a359-49fd-8f33-3086717b7c39","Type":"ContainerDied","Data":"c767e8356d00fca7d9b63b67a3a4b969852fd55132a8603436807b992e3d36a7"} Oct 30 20:07:25.857435 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:25.857412 5088 generic.go:356] "Generic (PLEG): container finished" podID="ea2533f2-6a01-44f1-a528-7b5e3eb8dde4" containerID="853c52cf37bf5e2f25d8c97df798ca299b49823df9a42d575e23789e1eb7fba9" exitCode=0 Oct 30 20:07:25.857540 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:25.857487 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe47f907-snhsh" event={"ID":"ea2533f2-6a01-44f1-a528-7b5e3eb8dde4","Type":"ContainerDied","Data":"853c52cf37bf5e2f25d8c97df798ca299b49823df9a42d575e23789e1eb7fba9"} Oct 30 20:07:25.940297 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:25.940248 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/811db7c8-11d5-4e9d-8c8b-254c17906e84-credential-socket\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf\" (UID: \"811db7c8-11d5-4e9d-8c8b-254c17906e84\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" Oct 30 20:07:25.940463 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:25.940305 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/811db7c8-11d5-4e9d-8c8b-254c17906e84-workload-socket\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf\" (UID: \"811db7c8-11d5-4e9d-8c8b-254c17906e84\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" Oct 30 20:07:25.940463 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:25.940348 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/811db7c8-11d5-4e9d-8c8b-254c17906e84-istiod-ca-cert\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf\" (UID: \"811db7c8-11d5-4e9d-8c8b-254c17906e84\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" Oct 30 20:07:25.940463 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:25.940434 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/811db7c8-11d5-4e9d-8c8b-254c17906e84-istio-data\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf\" (UID: \"811db7c8-11d5-4e9d-8c8b-254c17906e84\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" Oct 30 20:07:25.940727 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:25.940533 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/811db7c8-11d5-4e9d-8c8b-254c17906e84-istio-envoy\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf\" (UID: \"811db7c8-11d5-4e9d-8c8b-254c17906e84\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" Oct 30 20:07:25.940727 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:25.940660 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/811db7c8-11d5-4e9d-8c8b-254c17906e84-workload-certs\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf\" (UID: \"811db7c8-11d5-4e9d-8c8b-254c17906e84\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" Oct 30 20:07:25.940832 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:25.940771 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/811db7c8-11d5-4e9d-8c8b-254c17906e84-istio-podinfo\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf\" (UID: \"811db7c8-11d5-4e9d-8c8b-254c17906e84\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" Oct 30 20:07:25.940886 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:25.940831 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98t24\" (UniqueName: \"kubernetes.io/projected/811db7c8-11d5-4e9d-8c8b-254c17906e84-kube-api-access-98t24\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf\" (UID: \"811db7c8-11d5-4e9d-8c8b-254c17906e84\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" Oct 30 20:07:25.940962 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:25.940940 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/811db7c8-11d5-4e9d-8c8b-254c17906e84-istio-token\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf\" (UID: \"811db7c8-11d5-4e9d-8c8b-254c17906e84\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" Oct 30 20:07:25.941395 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:25.941368 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/811db7c8-11d5-4e9d-8c8b-254c17906e84-istio-data\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf\" (UID: \"811db7c8-11d5-4e9d-8c8b-254c17906e84\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" Oct 30 20:07:25.941680 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:25.941649 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/811db7c8-11d5-4e9d-8c8b-254c17906e84-credential-socket\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf\" (UID: \"811db7c8-11d5-4e9d-8c8b-254c17906e84\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" Oct 30 20:07:25.941912 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:25.941895 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/811db7c8-11d5-4e9d-8c8b-254c17906e84-workload-socket\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf\" (UID: \"811db7c8-11d5-4e9d-8c8b-254c17906e84\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" Oct 30 20:07:25.942533 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:25.942511 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/811db7c8-11d5-4e9d-8c8b-254c17906e84-istiod-ca-cert\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf\" (UID: \"811db7c8-11d5-4e9d-8c8b-254c17906e84\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" Oct 30 20:07:25.943001 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:25.942969 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/811db7c8-11d5-4e9d-8c8b-254c17906e84-workload-certs\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf\" (UID: \"811db7c8-11d5-4e9d-8c8b-254c17906e84\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" Oct 30 20:07:25.945280 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:25.945254 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/811db7c8-11d5-4e9d-8c8b-254c17906e84-istio-envoy\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf\" (UID: \"811db7c8-11d5-4e9d-8c8b-254c17906e84\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" Oct 30 20:07:25.946486 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:25.946462 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/811db7c8-11d5-4e9d-8c8b-254c17906e84-istio-podinfo\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf\" (UID: \"811db7c8-11d5-4e9d-8c8b-254c17906e84\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" Oct 30 20:07:25.968970 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:25.968933 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/811db7c8-11d5-4e9d-8c8b-254c17906e84-istio-token\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf\" (UID: \"811db7c8-11d5-4e9d-8c8b-254c17906e84\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" Oct 30 20:07:25.976643 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:25.976590 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98t24\" (UniqueName: \"kubernetes.io/projected/811db7c8-11d5-4e9d-8c8b-254c17906e84-kube-api-access-98t24\") pod \"e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf\" (UID: \"811db7c8-11d5-4e9d-8c8b-254c17906e84\") " pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" Oct 30 20:07:26.035373 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:26.035341 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aac2627-nz2jn" Oct 30 20:07:26.110158 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:26.110115 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" Oct 30 20:07:26.144232 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:26.144203 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7754459-8e35-4d81-a0e8-325f39cd7c8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7754459-8e35-4d81-a0e8-325f39cd7c8b-maas-api-openshift-ca\") pod \"f7754459-8e35-4d81-a0e8-325f39cd7c8b\" (UID: \"f7754459-8e35-4d81-a0e8-325f39cd7c8b\") " Oct 30 20:07:26.144411 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:26.144273 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7754459-8e35-4d81-a0e8-325f39cd7c8b-kube-api-access-zp5pm\" (UniqueName: \"kubernetes.io/projected/f7754459-8e35-4d81-a0e8-325f39cd7c8b-kube-api-access-zp5pm\") pod \"f7754459-8e35-4d81-a0e8-325f39cd7c8b\" (UID: \"f7754459-8e35-4d81-a0e8-325f39cd7c8b\") " Oct 30 20:07:26.144490 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:26.144474 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7754459-8e35-4d81-a0e8-325f39cd7c8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7754459-8e35-4d81-a0e8-325f39cd7c8b-maas-api-xks-ca\") pod \"f7754459-8e35-4d81-a0e8-325f39cd7c8b\" (UID: \"f7754459-8e35-4d81-a0e8-325f39cd7c8b\") " Oct 30 20:07:26.145401 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:26.145363 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7754459-8e35-4d81-a0e8-325f39cd7c8b-maas-api-openshift-ca" pod "f7754459-8e35-4d81-a0e8-325f39cd7c8b" (UID: "f7754459-8e35-4d81-a0e8-325f39cd7c8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:07:26.147283 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:26.147257 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7754459-8e35-4d81-a0e8-325f39cd7c8b-maas-api-xks-ca" pod "f7754459-8e35-4d81-a0e8-325f39cd7c8b" (UID: "f7754459-8e35-4d81-a0e8-325f39cd7c8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:07:26.147597 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:26.147566 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7754459-8e35-4d81-a0e8-325f39cd7c8b-kube-api-access-zp5pm" pod "f7754459-8e35-4d81-a0e8-325f39cd7c8b" (UID: "f7754459-8e35-4d81-a0e8-325f39cd7c8b"). InnerVolumeSpecName "kube-api-access-zp5pm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:07:26.247674 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:26.247637 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7754459-8e35-4d81-a0e8-325f39cd7c8b-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:07:26.247674 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:26.247673 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp5pm\" (UniqueName: \"kubernetes.io/projected/f7754459-8e35-4d81-a0e8-325f39cd7c8b-kube-api-access-zp5pm\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:07:26.247932 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:26.247689 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7754459-8e35-4d81-a0e8-325f39cd7c8b-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:07:26.473021 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:07:26.472990 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod811db7c8_11d5_4e9d_8c8b_254c17906e84.slice/crio-e6990407150c0016c13b03b54d0cfaeadc6ae4d38c28904bbe60b2589eb611e9 WatchSource:0}: Error finding container e6990407150c0016c13b03b54d0cfaeadc6ae4d38c28904bbe60b2589eb611e9: Status 404 returned error can't find the container with id e6990407150c0016c13b03b54d0cfaeadc6ae4d38c28904bbe60b2589eb611e9 Oct 30 20:07:26.476210 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:26.476172 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:07:26.476332 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:26.476243 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:07:26.476332 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:26.476274 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:07:26.478166 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:26.478137 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf"] Oct 30 20:07:26.644181 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:26.644133 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" containerID="cri-o://f6298a28e1fd184800beed4f30d68e12df45ab4e8de89f98a576b58dbafc33ad" gracePeriod=45 Oct 30 20:07:26.649250 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:26.649216 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": EOF" Oct 30 20:07:26.874222 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:26.874193 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aac2627-nz2jn" Oct 30 20:07:26.874763 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:26.874218 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aac2627-nz2jn" event={"ID":"f7754459-8e35-4d81-a0e8-325f39cd7c8b","Type":"ContainerDied","Data":"1b15a5153f404f94dbcc529e88af09998f69506f690eccd875aff60f4c6e1e15"} Oct 30 20:07:26.874763 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:26.874285 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b15a5153f404f94dbcc529e88af09998f69506f690eccd875aff60f4c6e1e15" Oct 30 20:07:26.876687 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:26.876655 5088 generic.go:356] "Generic (PLEG): container finished" podID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerID="f6298a28e1fd184800beed4f30d68e12df45ab4e8de89f98a576b58dbafc33ad" exitCode=0 Oct 30 20:07:26.876819 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:26.876806 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" event={"ID":"f8d25fcd-f343-445b-ac68-b2248f619ee4","Type":"ContainerDied","Data":"f6298a28e1fd184800beed4f30d68e12df45ab4e8de89f98a576b58dbafc33ad"} Oct 30 20:07:26.879641 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:26.879590 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" event={"ID":"811db7c8-11d5-4e9d-8c8b-254c17906e84","Type":"ContainerStarted","Data":"daeffcde17c313caceb0a68272537ab468c4b529d8d60af9e473a9c6ffd1ffd3"} Oct 30 20:07:26.879890 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:26.879849 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" event={"ID":"811db7c8-11d5-4e9d-8c8b-254c17906e84","Type":"ContainerStarted","Data":"e6990407150c0016c13b03b54d0cfaeadc6ae4d38c28904bbe60b2589eb611e9"} Oct 30 20:07:26.949320 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:26.949250 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" podStartSLOduration=1.94923287 podStartE2EDuration="1.94923287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:26.947710092 +0000 UTC m=+1208.360423897" watchObservedRunningTime="2026-10-30 20:07:26.94923287 +0000 UTC m=+1208.361946648" Oct 30 20:07:27.002929 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.002505 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" Oct 30 20:07:27.065570 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.065502 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8d25fcd-f343-445b-ac68-b2248f619ee4-home\" (UniqueName: \"kubernetes.io/empty-dir/f8d25fcd-f343-445b-ac68-b2248f619ee4-home\") pod \"f8d25fcd-f343-445b-ac68-b2248f619ee4\" (UID: \"f8d25fcd-f343-445b-ac68-b2248f619ee4\") " Oct 30 20:07:27.065958 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.065926 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8d25fcd-f343-445b-ac68-b2248f619ee4-home" pod "f8d25fcd-f343-445b-ac68-b2248f619ee4" (UID: "f8d25fcd-f343-445b-ac68-b2248f619ee4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:07:27.066098 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.066078 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8d25fcd-f343-445b-ac68-b2248f619ee4-tls-certs\" (UniqueName: \"kubernetes.io/secret/f8d25fcd-f343-445b-ac68-b2248f619ee4-tls-certs\") pod \"f8d25fcd-f343-445b-ac68-b2248f619ee4\" (UID: \"f8d25fcd-f343-445b-ac68-b2248f619ee4\") " Oct 30 20:07:27.066196 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.066184 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8d25fcd-f343-445b-ac68-b2248f619ee4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8d25fcd-f343-445b-ac68-b2248f619ee4-dshm\") pod \"f8d25fcd-f343-445b-ac68-b2248f619ee4\" (UID: \"f8d25fcd-f343-445b-ac68-b2248f619ee4\") " Oct 30 20:07:27.066305 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.066291 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8d25fcd-f343-445b-ac68-b2248f619ee4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8d25fcd-f343-445b-ac68-b2248f619ee4-model-cache\") pod \"f8d25fcd-f343-445b-ac68-b2248f619ee4\" (UID: \"f8d25fcd-f343-445b-ac68-b2248f619ee4\") " Oct 30 20:07:27.066658 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.066639 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8d25fcd-f343-445b-ac68-b2248f619ee4-kube-api-access-4wsxf\" (UniqueName: \"kubernetes.io/projected/f8d25fcd-f343-445b-ac68-b2248f619ee4-kube-api-access-4wsxf\") pod \"f8d25fcd-f343-445b-ac68-b2248f619ee4\" (UID: \"f8d25fcd-f343-445b-ac68-b2248f619ee4\") " Oct 30 20:07:27.066786 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.066773 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8d25fcd-f343-445b-ac68-b2248f619ee4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8d25fcd-f343-445b-ac68-b2248f619ee4-tmp-dir\") pod \"f8d25fcd-f343-445b-ac68-b2248f619ee4\" (UID: \"f8d25fcd-f343-445b-ac68-b2248f619ee4\") " Oct 30 20:07:27.067641 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.067602 5088 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8d25fcd-f343-445b-ac68-b2248f619ee4-home\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:07:27.067999 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.067981 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8d25fcd-f343-445b-ac68-b2248f619ee4-tmp-dir" pod "f8d25fcd-f343-445b-ac68-b2248f619ee4" (UID: "f8d25fcd-f343-445b-ac68-b2248f619ee4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:07:27.068276 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.068254 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8d25fcd-f343-445b-ac68-b2248f619ee4-model-cache" pod "f8d25fcd-f343-445b-ac68-b2248f619ee4" (UID: "f8d25fcd-f343-445b-ac68-b2248f619ee4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:07:27.075698 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.075664 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8d25fcd-f343-445b-ac68-b2248f619ee4-dshm" pod "f8d25fcd-f343-445b-ac68-b2248f619ee4" (UID: "f8d25fcd-f343-445b-ac68-b2248f619ee4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:07:27.082129 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.079399 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8d25fcd-f343-445b-ac68-b2248f619ee4-kube-api-access-4wsxf" pod "f8d25fcd-f343-445b-ac68-b2248f619ee4" (UID: "f8d25fcd-f343-445b-ac68-b2248f619ee4"). InnerVolumeSpecName "kube-api-access-4wsxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:07:27.082129 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.081540 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8d25fcd-f343-445b-ac68-b2248f619ee4-tls-certs" pod "f8d25fcd-f343-445b-ac68-b2248f619ee4" (UID: "f8d25fcd-f343-445b-ac68-b2248f619ee4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:07:27.118991 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.112416 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" Oct 30 20:07:27.118991 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.112570 5088 patch_prober.go:29] interesting pod/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 20:07:27.118991 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.112632 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" podUID="811db7c8-11d5-4e9d-8c8b-254c17906e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 20:07:27.181056 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.181014 5088 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8d25fcd-f343-445b-ac68-b2248f619ee4-tls-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:07:27.181056 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.181057 5088 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8d25fcd-f343-445b-ac68-b2248f619ee4-dshm\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:07:27.181262 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.181069 5088 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8d25fcd-f343-445b-ac68-b2248f619ee4-model-cache\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:07:27.181262 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.181085 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wsxf\" (UniqueName: \"kubernetes.io/projected/f8d25fcd-f343-445b-ac68-b2248f619ee4-kube-api-access-4wsxf\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:07:27.181262 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.181098 5088 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8d25fcd-f343-445b-ac68-b2248f619ee4-tmp-dir\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:07:27.191082 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.190522 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2891b522-8gd2t" Oct 30 20:07:27.199108 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.199010 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe47f907-snhsh" Oct 30 20:07:27.281834 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.281734 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b45a2b75-a359-49fd-8f33-3086717b7c39-kube-api-access-dhgk8\" (UniqueName: \"kubernetes.io/projected/b45a2b75-a359-49fd-8f33-3086717b7c39-kube-api-access-dhgk8\") pod \"b45a2b75-a359-49fd-8f33-3086717b7c39\" (UID: \"b45a2b75-a359-49fd-8f33-3086717b7c39\") " Oct 30 20:07:27.282016 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.281874 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b45a2b75-a359-49fd-8f33-3086717b7c39-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b45a2b75-a359-49fd-8f33-3086717b7c39-maas-api-openshift-ca\") pod \"b45a2b75-a359-49fd-8f33-3086717b7c39\" (UID: \"b45a2b75-a359-49fd-8f33-3086717b7c39\") " Oct 30 20:07:27.282016 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.281930 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea2533f2-6a01-44f1-a528-7b5e3eb8dde4-kube-api-access-9r99h\" (UniqueName: \"kubernetes.io/projected/ea2533f2-6a01-44f1-a528-7b5e3eb8dde4-kube-api-access-9r99h\") pod \"ea2533f2-6a01-44f1-a528-7b5e3eb8dde4\" (UID: \"ea2533f2-6a01-44f1-a528-7b5e3eb8dde4\") " Oct 30 20:07:27.282144 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.282027 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea2533f2-6a01-44f1-a528-7b5e3eb8dde4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea2533f2-6a01-44f1-a528-7b5e3eb8dde4-maas-api-openshift-ca\") pod \"ea2533f2-6a01-44f1-a528-7b5e3eb8dde4\" (UID: \"ea2533f2-6a01-44f1-a528-7b5e3eb8dde4\") " Oct 30 20:07:27.282144 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.282091 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b45a2b75-a359-49fd-8f33-3086717b7c39-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b45a2b75-a359-49fd-8f33-3086717b7c39-maas-api-xks-ca\") pod \"b45a2b75-a359-49fd-8f33-3086717b7c39\" (UID: \"b45a2b75-a359-49fd-8f33-3086717b7c39\") " Oct 30 20:07:27.282243 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.282177 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea2533f2-6a01-44f1-a528-7b5e3eb8dde4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea2533f2-6a01-44f1-a528-7b5e3eb8dde4-maas-api-xks-ca\") pod \"ea2533f2-6a01-44f1-a528-7b5e3eb8dde4\" (UID: \"ea2533f2-6a01-44f1-a528-7b5e3eb8dde4\") " Oct 30 20:07:27.282759 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.282721 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea2533f2-6a01-44f1-a528-7b5e3eb8dde4-maas-api-openshift-ca" pod "ea2533f2-6a01-44f1-a528-7b5e3eb8dde4" (UID: "ea2533f2-6a01-44f1-a528-7b5e3eb8dde4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:07:27.283085 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.283062 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b45a2b75-a359-49fd-8f33-3086717b7c39-maas-api-openshift-ca" pod "b45a2b75-a359-49fd-8f33-3086717b7c39" (UID: "b45a2b75-a359-49fd-8f33-3086717b7c39"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:07:27.283206 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.283096 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea2533f2-6a01-44f1-a528-7b5e3eb8dde4-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:07:27.286593 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.286546 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea2533f2-6a01-44f1-a528-7b5e3eb8dde4-maas-api-xks-ca" pod "ea2533f2-6a01-44f1-a528-7b5e3eb8dde4" (UID: "ea2533f2-6a01-44f1-a528-7b5e3eb8dde4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:07:27.330372 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.330301 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b45a2b75-a359-49fd-8f33-3086717b7c39-maas-api-xks-ca" pod "b45a2b75-a359-49fd-8f33-3086717b7c39" (UID: "b45a2b75-a359-49fd-8f33-3086717b7c39"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:07:27.330372 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.330357 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b45a2b75-a359-49fd-8f33-3086717b7c39-kube-api-access-dhgk8" pod "b45a2b75-a359-49fd-8f33-3086717b7c39" (UID: "b45a2b75-a359-49fd-8f33-3086717b7c39"). InnerVolumeSpecName "kube-api-access-dhgk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:07:27.332163 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.332124 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea2533f2-6a01-44f1-a528-7b5e3eb8dde4-kube-api-access-9r99h" pod "ea2533f2-6a01-44f1-a528-7b5e3eb8dde4" (UID: "ea2533f2-6a01-44f1-a528-7b5e3eb8dde4"). InnerVolumeSpecName "kube-api-access-9r99h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:07:27.385536 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.385488 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhgk8\" (UniqueName: \"kubernetes.io/projected/b45a2b75-a359-49fd-8f33-3086717b7c39-kube-api-access-dhgk8\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:07:27.385815 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.385795 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b45a2b75-a359-49fd-8f33-3086717b7c39-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:07:27.385917 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.385904 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r99h\" (UniqueName: \"kubernetes.io/projected/ea2533f2-6a01-44f1-a528-7b5e3eb8dde4-kube-api-access-9r99h\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:07:27.386007 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.385994 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b45a2b75-a359-49fd-8f33-3086717b7c39-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:07:27.386093 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.386083 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea2533f2-6a01-44f1-a528-7b5e3eb8dde4-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:07:27.728513 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.728405 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:07:27.953394 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.953349 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" event={"ID":"f8d25fcd-f343-445b-ac68-b2248f619ee4","Type":"ContainerDied","Data":"ec546d234c701b753d70fb4ecb8075c340fddbdb7cdbab19faf2bf26cfedcaae"} Oct 30 20:07:27.953871 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.953425 5088 scope.go:122] "RemoveContainer" containerID="f6298a28e1fd184800beed4f30d68e12df45ab4e8de89f98a576b58dbafc33ad" Oct 30 20:07:27.953871 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.953646 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r" Oct 30 20:07:27.961710 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.961682 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2891b522-8gd2t" Oct 30 20:07:27.961844 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.961723 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2891b522-8gd2t" event={"ID":"b45a2b75-a359-49fd-8f33-3086717b7c39","Type":"ContainerDied","Data":"15dc028c33dc914b776b4cb3781d8589192420e5a5b89cf73c1537002accf857"} Oct 30 20:07:27.961844 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.961816 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15dc028c33dc914b776b4cb3781d8589192420e5a5b89cf73c1537002accf857" Oct 30 20:07:27.965538 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.964196 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe47f907-snhsh" Oct 30 20:07:27.965538 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.964228 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe47f907-snhsh" event={"ID":"ea2533f2-6a01-44f1-a528-7b5e3eb8dde4","Type":"ContainerDied","Data":"f5967f3894b76b16546433ec7b29ec772a03d110f60d405e6a9815bc809f1242"} Oct 30 20:07:27.965538 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:27.964275 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5967f3894b76b16546433ec7b29ec772a03d110f60d405e6a9815bc809f1242" Oct 30 20:07:28.035569 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:28.035532 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r"] Oct 30 20:07:28.047759 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:28.047729 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-912f24/premium-w3-912f24-kserve-56d6b776d7-mbb7r"] Oct 30 20:07:28.111925 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:28.111890 5088 patch_prober.go:29] interesting pod/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 20:07:28.112107 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:28.111952 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" podUID="811db7c8-11d5-4e9d-8c8b-254c17906e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 20:07:28.840941 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:28.840899 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" path="/var/lib/kubelet/pods/f8d25fcd-f343-445b-ac68-b2248f619ee4/volumes" Oct 30 20:07:29.126871 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:29.126776 5088 patch_prober.go:29] interesting pod/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 20:07:29.127466 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:29.127429 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" podUID="811db7c8-11d5-4e9d-8c8b-254c17906e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 20:07:29.572779 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:29.572733 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" podUID="fec493c7-3f5a-459e-99e0-a4c816d5a907" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": context deadline exceeded" Oct 30 20:07:29.577756 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:29.577696 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:29.707640 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:29.706750 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:29.732411 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:29.726693 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6e06370-nwczs"] Oct 30 20:07:29.732411 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:29.727906 5088 cpu_manager.go:414] "containerMap: removing container" podUID="f7754459-8e35-4d81-a0e8-325f39cd7c8b" containerName="revoke-keys" Oct 30 20:07:29.732411 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:29.727929 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7754459-8e35-4d81-a0e8-325f39cd7c8b" containerName="revoke-keys" Oct 30 20:07:29.732411 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:29.727968 5088 cpu_manager.go:414] "containerMap: removing container" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" Oct 30 20:07:29.732411 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:29.727976 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" Oct 30 20:07:29.732411 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:29.728005 5088 cpu_manager.go:414] "containerMap: removing container" podUID="ea2533f2-6a01-44f1-a528-7b5e3eb8dde4" containerName="revoke-keys" Oct 30 20:07:29.732411 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:29.728013 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea2533f2-6a01-44f1-a528-7b5e3eb8dde4" containerName="revoke-keys" Oct 30 20:07:29.732411 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:29.728049 5088 cpu_manager.go:414] "containerMap: removing container" podUID="b45a2b75-a359-49fd-8f33-3086717b7c39" containerName="revoke-keys" Oct 30 20:07:29.732411 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:29.728055 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b45a2b75-a359-49fd-8f33-3086717b7c39" containerName="revoke-keys" Oct 30 20:07:29.732411 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:29.728232 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7754459-8e35-4d81-a0e8-325f39cd7c8b" containerName="revoke-keys" Oct 30 20:07:29.732411 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:29.728249 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8d25fcd-f343-445b-ac68-b2248f619ee4" containerName="main" Oct 30 20:07:29.732411 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:29.728264 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea2533f2-6a01-44f1-a528-7b5e3eb8dde4" containerName="revoke-keys" Oct 30 20:07:29.732411 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:29.728276 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="b45a2b75-a359-49fd-8f33-3086717b7c39" containerName="revoke-keys" Oct 30 20:07:29.766637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:29.751753 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e06370-nwczs" Oct 30 20:07:29.800722 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:29.785879 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:07:29.851692 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:29.805399 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6e06370-nwczs"] Oct 30 20:07:29.914075 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:29.857306 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 20:07:29.914350 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:29.893199 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:07:30.089299 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:29.894506 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5faeded-3cb2-4229-9710-4579376f59cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6e06370-nwczs\" (UID: \"e5faeded-3cb2-4229-9710-4579376f59cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e06370-nwczs" Oct 30 20:07:30.089659 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:30.089633 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26rxt\" (UniqueName: \"kubernetes.io/projected/e5faeded-3cb2-4229-9710-4579376f59cc-kube-api-access-26rxt\") pod \"maas-api-revoke-sub-b6e06370-nwczs\" (UID: \"e5faeded-3cb2-4229-9710-4579376f59cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e06370-nwczs" Oct 30 20:07:30.089966 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:30.089946 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5faeded-3cb2-4229-9710-4579376f59cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6e06370-nwczs\" (UID: \"e5faeded-3cb2-4229-9710-4579376f59cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e06370-nwczs" Oct 30 20:07:30.091331 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:30.090369 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:30.217944 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:30.194756 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5faeded-3cb2-4229-9710-4579376f59cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6e06370-nwczs\" (UID: \"e5faeded-3cb2-4229-9710-4579376f59cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e06370-nwczs" Oct 30 20:07:30.217944 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:30.195139 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5faeded-3cb2-4229-9710-4579376f59cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6e06370-nwczs\" (UID: \"e5faeded-3cb2-4229-9710-4579376f59cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e06370-nwczs" Oct 30 20:07:30.217944 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:30.195202 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26rxt\" (UniqueName: \"kubernetes.io/projected/e5faeded-3cb2-4229-9710-4579376f59cc-kube-api-access-26rxt\") pod \"maas-api-revoke-sub-b6e06370-nwczs\" (UID: \"e5faeded-3cb2-4229-9710-4579376f59cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e06370-nwczs" Oct 30 20:07:30.217944 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:30.196419 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5faeded-3cb2-4229-9710-4579376f59cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6e06370-nwczs\" (UID: \"e5faeded-3cb2-4229-9710-4579376f59cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e06370-nwczs" Oct 30 20:07:30.217944 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:30.212794 5088 patch_prober.go:29] interesting pod/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 20:07:30.217944 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:30.212856 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" podUID="811db7c8-11d5-4e9d-8c8b-254c17906e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 20:07:30.317637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:30.313315 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 30 20:07:30.425639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:30.399399 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26rxt\" (UniqueName: \"kubernetes.io/projected/e5faeded-3cb2-4229-9710-4579376f59cc-kube-api-access-26rxt\") pod \"maas-api-revoke-sub-b6e06370-nwczs\" (UID: \"e5faeded-3cb2-4229-9710-4579376f59cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e06370-nwczs" Oct 30 20:07:30.444752 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:30.434123 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5faeded-3cb2-4229-9710-4579376f59cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6e06370-nwczs\" (UID: \"e5faeded-3cb2-4229-9710-4579376f59cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e06370-nwczs" Oct 30 20:07:30.474892 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:30.468597 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 20:07:30.580013 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:30.577159 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" podUID="fec493c7-3f5a-459e-99e0-a4c816d5a907" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": context deadline exceeded" Oct 30 20:07:30.580013 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:30.577967 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" Oct 30 20:07:30.580013 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:30.578269 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 20:07:30.580013 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:30.579051 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:07:30.716930 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:30.716887 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:30.770702 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:30.730994 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 20:07:31.087733 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:30.742822 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e06370-nwczs" Oct 30 20:07:31.109638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:31.091492 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:31.161013 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:31.123516 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 20:07:31.249431 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:31.184784 5088 patch_prober.go:29] interesting pod/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 20:07:31.249431 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:31.220135 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" podUID="811db7c8-11d5-4e9d-8c8b-254c17906e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 20:07:31.313750 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:31.313704 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 30 20:07:31.556639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:31.553870 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 20:07:31.614750 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:31.579775 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" podUID="fec493c7-3f5a-459e-99e0-a4c816d5a907" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": context deadline exceeded" Oct 30 20:07:31.643884 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:31.615198 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:32.031578 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:32.005967 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:32.194821 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:32.177286 5088 patch_prober.go:29] interesting pod/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 20:07:32.194821 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:32.193464 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" podUID="811db7c8-11d5-4e9d-8c8b-254c17906e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 20:07:32.230640 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:32.227985 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:32.349732 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:32.331962 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 20:07:32.613723 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:32.582599 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" podUID="fec493c7-3f5a-459e-99e0-a4c816d5a907" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": context deadline exceeded" Oct 30 20:07:32.613723 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:32.583993 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:32.623319 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:32.623262 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 20:07:32.723637 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:07:32.713950 5088 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.752s" Oct 30 20:07:32.723637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:32.714202 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" Oct 30 20:07:32.723637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:32.714278 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:07:32.723637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:32.714355 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" Oct 30 20:07:32.723637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:32.714644 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:07:32.760049 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:32.759995 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 20:07:32.879421 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:32.879377 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" containerID="cri-o://75803e24c40db568bfb7548584770fa5234fa42ec240f96e5397c06422744f0e" gracePeriod=44 Oct 30 20:07:32.963101 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:32.963058 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:07:32.963366 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:32.963334 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" podUID="baca0e89-a102-4354-af54-c14ccc418923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:33.052327 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:33.052281 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:33.264649 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:33.245996 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:33.608743 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:33.595873 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:33.656018 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:33.655875 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:33.743676 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:33.726723 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" podUID="fec493c7-3f5a-459e-99e0-a4c816d5a907" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": context deadline exceeded" Oct 30 20:07:33.782637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:33.747639 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 20:07:33.809267 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:33.809207 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": dial tcp 10.132.0.62:8000: connect: connection refused" Oct 30 20:07:34.269740 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:33.857881 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:34.269740 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:33.943733 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:34.269740 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:34.097792 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f4snn" podUID="5ec64166-d2fc-47f6-96d1-48ea0ee1b70c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:34.269740 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:34.241975 5088 patch_prober.go:29] interesting pod/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 20:07:34.269740 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:34.252675 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" podUID="811db7c8-11d5-4e9d-8c8b-254c17906e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 20:07:34.291513 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:34.273724 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:34.291513 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:34.275519 5088 patch_prober.go:29] interesting pod/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 20:07:34.291513 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:34.275574 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" podUID="811db7c8-11d5-4e9d-8c8b-254c17906e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 20:07:34.291513 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:34.276151 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": dial tcp 10.132.0.62:8000: connect: connection refused" Oct 30 20:07:34.291513 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:34.278049 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:34.291513 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:34.280445 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9" podUID="1fa4e381-a575-4bf9-a65d-33bdc3093587" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded" Oct 30 20:07:34.363638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:34.363583 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:07:34.363822 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:34.363670 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" podUID="05a46855-a143-4c6d-8c87-404dc271bdb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:34.403675 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:34.392172 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wtlm7" podUID="419ffdb0-c223-4426-872a-50e3073812e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded" Oct 30 20:07:34.480076 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:34.432029 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": context deadline exceeded" Oct 30 20:07:34.490075 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:34.490027 5088 patch_prober.go:29] interesting pod/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:07:34.490343 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:34.490311 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" podUID="ff410c2e-9048-4e5c-965b-1261336561bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:34.613758 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:34.612992 5088 patch_prober.go:29] interesting pod/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:07:34.613758 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:34.613072 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" podUID="fc3ed9fa-b7f0-418f-8ac9-422f9e765027" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:34.688299 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:34.688246 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 30 20:07:34.727770 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:34.727714 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" podUID="fec493c7-3f5a-459e-99e0-a4c816d5a907" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:34.927797 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:34.927690 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 20:07:34.950728 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:34.937888 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:07:34.950728 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:34.937951 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" podUID="419aecf7-a250-42ad-995b-e7ebd9630a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:35.071640 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:34.991284 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:07:35.071640 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:35.029874 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" podUID="b55ec6e2-9392-4d8d-8340-ef12f674970c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:35.071640 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:34.997736 5088 patch_prober.go:29] interesting pod/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:07:35.071640 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:35.030139 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" podUID="d209c20c-000f-4cc4-9792-5a71eeb0a773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:35.131639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:35.115509 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": dial tcp 10.132.0.62:8000: connect: connection refused" Oct 30 20:07:35.131639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:35.115632 5088 patch_prober.go:29] interesting pod/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 20:07:35.131639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:35.115663 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" podUID="811db7c8-11d5-4e9d-8c8b-254c17906e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 20:07:35.272717 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:35.272671 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 20:07:35.305649 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:35.303115 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:35.333092 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:35.333024 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:35.729564 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:35.727724 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:07:35.729564 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:35.727795 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" podUID="0b36276d-bc5b-4415-8cf9-461aee572f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:35.740982 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:35.730848 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:35.752684 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:07:35.752645 5088 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.872s" Oct 30 20:07:35.810639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:35.785255 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e06370-nwczs" event={"ID":"e5faeded-3cb2-4229-9710-4579376f59cc","Type":"ContainerStarted","Data":"ffdb1c28ba095299eedde45329a0d98c1b60f7aa4028087592f2b3468d6dee19"} Oct 30 20:07:35.835640 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:35.793723 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" podUID="fec493c7-3f5a-459e-99e0-a4c816d5a907" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": context deadline exceeded" Oct 30 20:07:35.860680 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:35.846772 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:07:35.860680 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:35.846860 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" podUID="b56fca64-1150-4198-945e-7e830300c11d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:35.930992 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:35.930885 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:36.135634 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:36.126765 5088 patch_prober.go:29] interesting pod/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 20:07:36.135634 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:36.126833 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" podUID="811db7c8-11d5-4e9d-8c8b-254c17906e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 20:07:36.135634 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:36.128808 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": dial tcp 10.132.0.62:8000: connect: connection refused" Oct 30 20:07:36.425687 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:36.278569 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:36.596637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:36.589268 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:36.621865 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:36.621812 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded" Oct 30 20:07:36.622093 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:36.605887 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" podUID="fec493c7-3f5a-459e-99e0-a4c816d5a907" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/health\": context deadline exceeded" Oct 30 20:07:36.642637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:36.605849 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:36.858640 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:36.857623 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" podUID="fec493c7-3f5a-459e-99e0-a4c816d5a907" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": context deadline exceeded" Oct 30 20:07:37.020276 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:37.020226 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 20:07:37.110562 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:37.110466 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f4snn" podUID="5ec64166-d2fc-47f6-96d1-48ea0ee1b70c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:37.111801 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:37.111748 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 30 20:07:37.571522 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:37.571479 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 20:07:37.572105 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:37.128552 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wtlm7" podUID="419ffdb0-c223-4426-872a-50e3073812e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded" Oct 30 20:07:37.572753 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:37.137760 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": dial tcp 10.132.0.62:8000: connect: connection refused" Oct 30 20:07:37.572889 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:37.148574 5088 patch_prober.go:29] interesting pod/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 20:07:37.573053 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:37.573018 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" podUID="811db7c8-11d5-4e9d-8c8b-254c17906e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 20:07:37.602690 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:07:37.599389 5088 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.787s" Oct 30 20:07:37.643629 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:37.638374 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:37.653222 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:37.128377 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9" podUID="1fa4e381-a575-4bf9-a65d-33bdc3093587" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded" Oct 30 20:07:37.683329 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:37.681513 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:07:37.683329 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:37.681556 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6e06370-nwczs"] Oct 30 20:07:37.683329 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:37.681594 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-62f4a1-4vztc"] Oct 30 20:07:37.719929 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:37.719877 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-62f4a1-4vztc"] Oct 30 20:07:37.720182 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:37.720166 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3d36dcc-bsrzf"] Oct 30 20:07:37.720285 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:37.720276 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3d36dcc-bsrzf"] Oct 30 20:07:37.720494 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:37.720478 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" Oct 30 20:07:37.720675 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:37.720664 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:07:37.720824 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:37.720813 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" Oct 30 20:07:37.720981 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:37.720968 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 20:07:37.721144 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:37.721130 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:07:37.723595 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:37.723571 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-62f4a1-4vztc" Oct 30 20:07:37.736455 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:07:37.735573 5088 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d117d60_d768_4d67_bb48_cd0522fe2a57.slice/crio-conmon-75803e24c40db568bfb7548584770fa5234fa42ec240f96e5397c06422744f0e.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d117d60_d768_4d67_bb48_cd0522fe2a57.slice/crio-75803e24c40db568bfb7548584770fa5234fa42ec240f96e5397c06422744f0e.scope\": RecentStats: unable to find data in memory cache]" Oct 30 20:07:37.736455 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:07:37.736206 5088 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d117d60_d768_4d67_bb48_cd0522fe2a57.slice/crio-75803e24c40db568bfb7548584770fa5234fa42ec240f96e5397c06422744f0e.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d117d60_d768_4d67_bb48_cd0522fe2a57.slice/crio-conmon-75803e24c40db568bfb7548584770fa5234fa42ec240f96e5397c06422744f0e.scope\": RecentStats: unable to find data in memory cache]" Oct 30 20:07:37.917674 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:37.917631 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6mcz\" (UniqueName: \"kubernetes.io/projected/5f60e22d-5bad-4327-b28c-4f618abce710-kube-api-access-f6mcz\") pod \"maas-api-revoke-keys-e2e-worker-w0-62f4a1-4vztc\" (UID: \"5f60e22d-5bad-4327-b28c-4f618abce710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-62f4a1-4vztc" Oct 30 20:07:37.918149 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:37.917897 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f60e22d-5bad-4327-b28c-4f618abce710-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-62f4a1-4vztc\" (UID: \"5f60e22d-5bad-4327-b28c-4f618abce710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-62f4a1-4vztc" Oct 30 20:07:37.918468 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:37.918439 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f60e22d-5bad-4327-b28c-4f618abce710-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-62f4a1-4vztc\" (UID: \"5f60e22d-5bad-4327-b28c-4f618abce710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-62f4a1-4vztc" Oct 30 20:07:38.020872 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:38.020836 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f60e22d-5bad-4327-b28c-4f618abce710-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-62f4a1-4vztc\" (UID: \"5f60e22d-5bad-4327-b28c-4f618abce710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-62f4a1-4vztc" Oct 30 20:07:38.021157 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:38.021140 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f60e22d-5bad-4327-b28c-4f618abce710-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-62f4a1-4vztc\" (UID: \"5f60e22d-5bad-4327-b28c-4f618abce710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-62f4a1-4vztc" Oct 30 20:07:38.021432 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:38.021416 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6mcz\" (UniqueName: \"kubernetes.io/projected/5f60e22d-5bad-4327-b28c-4f618abce710-kube-api-access-f6mcz\") pod \"maas-api-revoke-keys-e2e-worker-w0-62f4a1-4vztc\" (UID: \"5f60e22d-5bad-4327-b28c-4f618abce710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-62f4a1-4vztc" Oct 30 20:07:38.022021 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:38.021651 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f60e22d-5bad-4327-b28c-4f618abce710-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-62f4a1-4vztc\" (UID: \"5f60e22d-5bad-4327-b28c-4f618abce710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-62f4a1-4vztc" Oct 30 20:07:38.033694 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:38.033648 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f60e22d-5bad-4327-b28c-4f618abce710-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-62f4a1-4vztc\" (UID: \"5f60e22d-5bad-4327-b28c-4f618abce710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-62f4a1-4vztc" Oct 30 20:07:38.051452 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:38.051423 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6mcz\" (UniqueName: \"kubernetes.io/projected/5f60e22d-5bad-4327-b28c-4f618abce710-kube-api-access-f6mcz\") pod \"maas-api-revoke-keys-e2e-worker-w0-62f4a1-4vztc\" (UID: \"5f60e22d-5bad-4327-b28c-4f618abce710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-62f4a1-4vztc" Oct 30 20:07:38.068869 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:38.068842 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-62f4a1-4vztc" Oct 30 20:07:38.117733 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:38.117122 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": dial tcp 10.132.0.62:8000: connect: connection refused" Oct 30 20:07:38.117733 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:38.117232 5088 patch_prober.go:29] interesting pod/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 20:07:38.117733 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:38.117258 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" podUID="811db7c8-11d5-4e9d-8c8b-254c17906e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 20:07:38.362312 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:38.362277 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" Oct 30 20:07:38.442371 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:38.429496 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d117d60-d768-4d67-bb48-cd0522fe2a57-home\" (UniqueName: \"kubernetes.io/empty-dir/3d117d60-d768-4d67-bb48-cd0522fe2a57-home\") pod \"3d117d60-d768-4d67-bb48-cd0522fe2a57\" (UID: \"3d117d60-d768-4d67-bb48-cd0522fe2a57\") " Oct 30 20:07:38.442371 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:38.429589 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d117d60-d768-4d67-bb48-cd0522fe2a57-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d117d60-d768-4d67-bb48-cd0522fe2a57-model-cache\") pod \"3d117d60-d768-4d67-bb48-cd0522fe2a57\" (UID: \"3d117d60-d768-4d67-bb48-cd0522fe2a57\") " Oct 30 20:07:38.442371 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:38.429643 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d117d60-d768-4d67-bb48-cd0522fe2a57-kube-api-access-g8bhj\" (UniqueName: \"kubernetes.io/projected/3d117d60-d768-4d67-bb48-cd0522fe2a57-kube-api-access-g8bhj\") pod \"3d117d60-d768-4d67-bb48-cd0522fe2a57\" (UID: \"3d117d60-d768-4d67-bb48-cd0522fe2a57\") " Oct 30 20:07:38.442371 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:38.429810 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d117d60-d768-4d67-bb48-cd0522fe2a57-tls-certs\" (UniqueName: \"kubernetes.io/secret/3d117d60-d768-4d67-bb48-cd0522fe2a57-tls-certs\") pod \"3d117d60-d768-4d67-bb48-cd0522fe2a57\" (UID: \"3d117d60-d768-4d67-bb48-cd0522fe2a57\") " Oct 30 20:07:38.442371 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:38.429957 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d117d60-d768-4d67-bb48-cd0522fe2a57-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d117d60-d768-4d67-bb48-cd0522fe2a57-dshm\") pod \"3d117d60-d768-4d67-bb48-cd0522fe2a57\" (UID: \"3d117d60-d768-4d67-bb48-cd0522fe2a57\") " Oct 30 20:07:38.442371 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:38.429992 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d117d60-d768-4d67-bb48-cd0522fe2a57-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d117d60-d768-4d67-bb48-cd0522fe2a57-tmp-dir\") pod \"3d117d60-d768-4d67-bb48-cd0522fe2a57\" (UID: \"3d117d60-d768-4d67-bb48-cd0522fe2a57\") " Oct 30 20:07:38.454390 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:38.450826 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d117d60-d768-4d67-bb48-cd0522fe2a57-model-cache" pod "3d117d60-d768-4d67-bb48-cd0522fe2a57" (UID: "3d117d60-d768-4d67-bb48-cd0522fe2a57"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:07:38.454390 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:38.452456 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d117d60-d768-4d67-bb48-cd0522fe2a57-home" pod "3d117d60-d768-4d67-bb48-cd0522fe2a57" (UID: "3d117d60-d768-4d67-bb48-cd0522fe2a57"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:07:38.454390 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:38.453061 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d117d60-d768-4d67-bb48-cd0522fe2a57-tmp-dir" pod "3d117d60-d768-4d67-bb48-cd0522fe2a57" (UID: "3d117d60-d768-4d67-bb48-cd0522fe2a57"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:07:38.462458 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:38.455758 5088 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d117d60-d768-4d67-bb48-cd0522fe2a57-tmp-dir\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:07:38.462458 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:38.455783 5088 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d117d60-d768-4d67-bb48-cd0522fe2a57-home\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:07:38.462458 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:38.455794 5088 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d117d60-d768-4d67-bb48-cd0522fe2a57-model-cache\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:07:38.557665 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:38.554877 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d117d60-d768-4d67-bb48-cd0522fe2a57-dshm" pod "3d117d60-d768-4d67-bb48-cd0522fe2a57" (UID: "3d117d60-d768-4d67-bb48-cd0522fe2a57"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:07:38.557665 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:38.557576 5088 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d117d60-d768-4d67-bb48-cd0522fe2a57-dshm\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:07:38.639640 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:38.637164 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d117d60-d768-4d67-bb48-cd0522fe2a57-kube-api-access-g8bhj" pod "3d117d60-d768-4d67-bb48-cd0522fe2a57" (UID: "3d117d60-d768-4d67-bb48-cd0522fe2a57"). InnerVolumeSpecName "kube-api-access-g8bhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:07:38.677687 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:38.674972 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d117d60-d768-4d67-bb48-cd0522fe2a57-tls-certs" pod "3d117d60-d768-4d67-bb48-cd0522fe2a57" (UID: "3d117d60-d768-4d67-bb48-cd0522fe2a57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:07:38.677687 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:38.675285 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d117d60-d768-4d67-bb48-cd0522fe2a57-tls-certs\" (UniqueName: \"kubernetes.io/secret/3d117d60-d768-4d67-bb48-cd0522fe2a57-tls-certs\") pod \"3d117d60-d768-4d67-bb48-cd0522fe2a57\" (UID: \"3d117d60-d768-4d67-bb48-cd0522fe2a57\") " Oct 30 20:07:38.677687 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:38.676140 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8bhj\" (UniqueName: \"kubernetes.io/projected/3d117d60-d768-4d67-bb48-cd0522fe2a57-kube-api-access-g8bhj\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:07:38.677687 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:07:38.676245 5088 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/3d117d60-d768-4d67-bb48-cd0522fe2a57/volumes/kubernetes.io~secret/tls-certs Oct 30 20:07:38.677687 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:38.676261 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d117d60-d768-4d67-bb48-cd0522fe2a57-tls-certs" pod "3d117d60-d768-4d67-bb48-cd0522fe2a57" (UID: "3d117d60-d768-4d67-bb48-cd0522fe2a57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:07:38.745655 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:38.745523 5088 generic.go:356] "Generic (PLEG): container finished" podID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerID="75803e24c40db568bfb7548584770fa5234fa42ec240f96e5397c06422744f0e" exitCode=0 Oct 30 20:07:38.754786 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:38.754755 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" Oct 30 20:07:38.760453 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:38.760408 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e30d967b-a226-43a9-98ba-acb2bcc23e4c" path="/var/lib/kubelet/pods/e30d967b-a226-43a9-98ba-acb2bcc23e4c/volumes" Oct 30 20:07:38.761244 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:38.761227 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-62f4a1-4vztc"] Oct 30 20:07:38.761379 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:38.761364 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" event={"ID":"3d117d60-d768-4d67-bb48-cd0522fe2a57","Type":"ContainerDied","Data":"75803e24c40db568bfb7548584770fa5234fa42ec240f96e5397c06422744f0e"} Oct 30 20:07:38.761468 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:38.761454 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" event={"ID":"3d117d60-d768-4d67-bb48-cd0522fe2a57","Type":"ContainerDied","Data":"30f62f5e367eb5f09c24c6fbbc93375860604d5ab1ff77fca155fafc6ca299e2"} Oct 30 20:07:38.761535 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:38.761513 5088 scope.go:122] "RemoveContainer" containerID="75803e24c40db568bfb7548584770fa5234fa42ec240f96e5397c06422744f0e" Oct 30 20:07:38.761678 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:38.761527 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-62f4a1-4vztc" event={"ID":"5f60e22d-5bad-4327-b28c-4f618abce710","Type":"ContainerStarted","Data":"cbfa1f3a3259c0f7f8a6b6628f0f86bc2fa9068861f5bfb4fcca4227ab6a8688"} Oct 30 20:07:38.761825 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:38.761789 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e06370-nwczs" event={"ID":"e5faeded-3cb2-4229-9710-4579376f59cc","Type":"ContainerStarted","Data":"ab73a40d0355bf1339ca35b0625d6f80b3b92822a256bbae2f71117d46d72940"} Oct 30 20:07:38.777823 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:38.777802 5088 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d117d60-d768-4d67-bb48-cd0522fe2a57-tls-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:07:38.781049 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:38.781023 5088 scope.go:122] "RemoveContainer" containerID="75803e24c40db568bfb7548584770fa5234fa42ec240f96e5397c06422744f0e" Oct 30 20:07:38.781367 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:07:38.781344 5088 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75803e24c40db568bfb7548584770fa5234fa42ec240f96e5397c06422744f0e\": container with ID starting with 75803e24c40db568bfb7548584770fa5234fa42ec240f96e5397c06422744f0e not found: ID does not exist" containerID="75803e24c40db568bfb7548584770fa5234fa42ec240f96e5397c06422744f0e" Oct 30 20:07:38.781437 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:38.781376 5088 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75803e24c40db568bfb7548584770fa5234fa42ec240f96e5397c06422744f0e"} err="failed to get container status \"75803e24c40db568bfb7548584770fa5234fa42ec240f96e5397c06422744f0e\": rpc error: code = NotFound desc = could not find container \"75803e24c40db568bfb7548584770fa5234fa42ec240f96e5397c06422744f0e\": container with ID starting with 75803e24c40db568bfb7548584770fa5234fa42ec240f96e5397c06422744f0e not found: ID does not exist" Oct 30 20:07:38.837351 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:38.837297 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e06370-nwczs" podStartSLOduration=9.837280215 podStartE2EDuration="9.837280215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:38.836580241 +0000 UTC m=+1220.249294039" watchObservedRunningTime="2026-10-30 20:07:38.837280215 +0000 UTC m=+1220.249993995" Oct 30 20:07:39.110934 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:39.110861 5088 patch_prober.go:29] interesting pod/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 20:07:39.110934 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:39.110926 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" podUID="811db7c8-11d5-4e9d-8c8b-254c17906e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 20:07:39.770462 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:39.767264 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-62f4a1-4vztc" event={"ID":"5f60e22d-5bad-4327-b28c-4f618abce710","Type":"ContainerStarted","Data":"f68c3a10ea537704b7cf0aed2ccae53e6226800a25be8dd807b3f3ddc7330cbb"} Oct 30 20:07:39.770462 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:39.769388 5088 generic.go:356] "Generic (PLEG): container finished" podID="e5faeded-3cb2-4229-9710-4579376f59cc" containerID="ab73a40d0355bf1339ca35b0625d6f80b3b92822a256bbae2f71117d46d72940" exitCode=0 Oct 30 20:07:39.770462 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:39.769514 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e06370-nwczs" event={"ID":"e5faeded-3cb2-4229-9710-4579376f59cc","Type":"ContainerDied","Data":"ab73a40d0355bf1339ca35b0625d6f80b3b92822a256bbae2f71117d46d72940"} Oct 30 20:07:39.831911 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:39.831858 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-62f4a1-4vztc" podStartSLOduration=4.831844118 podStartE2EDuration="4.831844118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:39.827064287 +0000 UTC m=+1221.239778065" watchObservedRunningTime="2026-10-30 20:07:39.831844118 +0000 UTC m=+1221.244557896" Oct 30 20:07:40.046686 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:40.046554 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq"] Oct 30 20:07:40.081787 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:40.081738 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c36234df-r4njh"] Oct 30 20:07:40.088032 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:40.088001 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c36234df-r4njh"] Oct 30 20:07:40.104201 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:40.104167 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk"] Oct 30 20:07:40.111261 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:40.111227 5088 patch_prober.go:29] interesting pod/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 20:07:40.111681 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:40.111285 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" podUID="811db7c8-11d5-4e9d-8c8b-254c17906e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 20:07:40.744410 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:40.744377 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77f149f0-fa20-41b0-919f-baf1395d6879" path="/var/lib/kubelet/pods/77f149f0-fa20-41b0-919f-baf1395d6879/volumes" Oct 30 20:07:40.779827 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:40.779790 5088 generic.go:356] "Generic (PLEG): container finished" podID="5f60e22d-5bad-4327-b28c-4f618abce710" containerID="f68c3a10ea537704b7cf0aed2ccae53e6226800a25be8dd807b3f3ddc7330cbb" exitCode=0 Oct 30 20:07:40.780012 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:40.779864 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-62f4a1-4vztc" event={"ID":"5f60e22d-5bad-4327-b28c-4f618abce710","Type":"ContainerDied","Data":"f68c3a10ea537704b7cf0aed2ccae53e6226800a25be8dd807b3f3ddc7330cbb"} Oct 30 20:07:40.997346 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:40.997267 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e06370-nwczs" Oct 30 20:07:41.042279 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:41.042242 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5faeded-3cb2-4229-9710-4579376f59cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5faeded-3cb2-4229-9710-4579376f59cc-maas-api-xks-ca\") pod \"e5faeded-3cb2-4229-9710-4579376f59cc\" (UID: \"e5faeded-3cb2-4229-9710-4579376f59cc\") " Oct 30 20:07:41.042466 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:41.042386 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5faeded-3cb2-4229-9710-4579376f59cc-kube-api-access-26rxt\" (UniqueName: \"kubernetes.io/projected/e5faeded-3cb2-4229-9710-4579376f59cc-kube-api-access-26rxt\") pod \"e5faeded-3cb2-4229-9710-4579376f59cc\" (UID: \"e5faeded-3cb2-4229-9710-4579376f59cc\") " Oct 30 20:07:41.042561 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:41.042519 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5faeded-3cb2-4229-9710-4579376f59cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5faeded-3cb2-4229-9710-4579376f59cc-maas-api-openshift-ca\") pod \"e5faeded-3cb2-4229-9710-4579376f59cc\" (UID: \"e5faeded-3cb2-4229-9710-4579376f59cc\") " Oct 30 20:07:41.043096 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:41.043043 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5faeded-3cb2-4229-9710-4579376f59cc-maas-api-openshift-ca" pod "e5faeded-3cb2-4229-9710-4579376f59cc" (UID: "e5faeded-3cb2-4229-9710-4579376f59cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:07:41.043381 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:41.043331 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5faeded-3cb2-4229-9710-4579376f59cc-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:07:41.046066 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:41.045665 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5faeded-3cb2-4229-9710-4579376f59cc-kube-api-access-26rxt" pod "e5faeded-3cb2-4229-9710-4579376f59cc" (UID: "e5faeded-3cb2-4229-9710-4579376f59cc"). InnerVolumeSpecName "kube-api-access-26rxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:07:41.046450 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:41.046425 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5faeded-3cb2-4229-9710-4579376f59cc-maas-api-xks-ca" pod "e5faeded-3cb2-4229-9710-4579376f59cc" (UID: "e5faeded-3cb2-4229-9710-4579376f59cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:07:41.113718 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:41.112432 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" Oct 30 20:07:41.145432 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:41.145389 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5faeded-3cb2-4229-9710-4579376f59cc-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:07:41.145432 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:41.145428 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26rxt\" (UniqueName: \"kubernetes.io/projected/e5faeded-3cb2-4229-9710-4579376f59cc-kube-api-access-26rxt\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:07:41.469312 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:41.469068 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" Oct 30 20:07:41.790990 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:41.790954 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e06370-nwczs" Oct 30 20:07:41.791200 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:41.791003 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e06370-nwczs" event={"ID":"e5faeded-3cb2-4229-9710-4579376f59cc","Type":"ContainerDied","Data":"ffdb1c28ba095299eedde45329a0d98c1b60f7aa4028087592f2b3468d6dee19"} Oct 30 20:07:41.791200 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:41.791050 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffdb1c28ba095299eedde45329a0d98c1b60f7aa4028087592f2b3468d6dee19" Oct 30 20:07:41.793257 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:41.793237 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" Oct 30 20:07:42.056952 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:42.056846 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-62f4a1-4vztc" Oct 30 20:07:42.194429 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:42.194395 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f60e22d-5bad-4327-b28c-4f618abce710-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f60e22d-5bad-4327-b28c-4f618abce710-maas-api-xks-ca\") pod \"5f60e22d-5bad-4327-b28c-4f618abce710\" (UID: \"5f60e22d-5bad-4327-b28c-4f618abce710\") " Oct 30 20:07:42.194855 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:42.194591 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f60e22d-5bad-4327-b28c-4f618abce710-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f60e22d-5bad-4327-b28c-4f618abce710-maas-api-openshift-ca\") pod \"5f60e22d-5bad-4327-b28c-4f618abce710\" (UID: \"5f60e22d-5bad-4327-b28c-4f618abce710\") " Oct 30 20:07:42.194855 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:42.194681 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f60e22d-5bad-4327-b28c-4f618abce710-kube-api-access-f6mcz\" (UniqueName: \"kubernetes.io/projected/5f60e22d-5bad-4327-b28c-4f618abce710-kube-api-access-f6mcz\") pod \"5f60e22d-5bad-4327-b28c-4f618abce710\" (UID: \"5f60e22d-5bad-4327-b28c-4f618abce710\") " Oct 30 20:07:42.194945 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:42.194888 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f60e22d-5bad-4327-b28c-4f618abce710-maas-api-openshift-ca" pod "5f60e22d-5bad-4327-b28c-4f618abce710" (UID: "5f60e22d-5bad-4327-b28c-4f618abce710"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:07:42.195407 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:42.195384 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f60e22d-5bad-4327-b28c-4f618abce710-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:07:42.196548 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:42.196525 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f60e22d-5bad-4327-b28c-4f618abce710-maas-api-xks-ca" pod "5f60e22d-5bad-4327-b28c-4f618abce710" (UID: "5f60e22d-5bad-4327-b28c-4f618abce710"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:07:42.196833 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:42.196815 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f60e22d-5bad-4327-b28c-4f618abce710-kube-api-access-f6mcz" pod "5f60e22d-5bad-4327-b28c-4f618abce710" (UID: "5f60e22d-5bad-4327-b28c-4f618abce710"). InnerVolumeSpecName "kube-api-access-f6mcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:07:42.326693 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:42.322330 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6mcz\" (UniqueName: \"kubernetes.io/projected/5f60e22d-5bad-4327-b28c-4f618abce710-kube-api-access-f6mcz\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:07:42.326693 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:42.322367 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f60e22d-5bad-4327-b28c-4f618abce710-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:07:42.800870 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:42.800840 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-62f4a1-4vztc" Oct 30 20:07:42.800870 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:42.800854 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-62f4a1-4vztc" event={"ID":"5f60e22d-5bad-4327-b28c-4f618abce710","Type":"ContainerDied","Data":"cbfa1f3a3259c0f7f8a6b6628f0f86bc2fa9068861f5bfb4fcca4227ab6a8688"} Oct 30 20:07:42.801123 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:42.800908 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbfa1f3a3259c0f7f8a6b6628f0f86bc2fa9068861f5bfb4fcca4227ab6a8688" Oct 30 20:07:45.510791 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:45.510707 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-626870-7fcn9"] Oct 30 20:07:45.562971 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:45.562935 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-626870-7fcn9"] Oct 30 20:07:46.135849 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:46.135811 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5f10d2-58d98b4775-jq5gd"] Oct 30 20:07:46.749514 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:46.749470 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74c24a9b-9591-4478-a762-58940c1438f3" path="/var/lib/kubelet/pods/74c24a9b-9591-4478-a762-58940c1438f3/volumes" Oct 30 20:07:47.125313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:47.125224 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-62f4a1-645b9d8589-k6rx9"] Oct 30 20:07:47.975285 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:47.975008 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0350007e-v6mjg"] Oct 30 20:07:47.979709 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:47.976902 5088 cpu_manager.go:414] "containerMap: removing container" podUID="5f60e22d-5bad-4327-b28c-4f618abce710" containerName="revoke-keys" Oct 30 20:07:47.979709 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:47.976935 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f60e22d-5bad-4327-b28c-4f618abce710" containerName="revoke-keys" Oct 30 20:07:47.979709 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:47.976972 5088 cpu_manager.go:414] "containerMap: removing container" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" Oct 30 20:07:47.979709 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:47.976980 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" Oct 30 20:07:47.979709 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:47.977030 5088 cpu_manager.go:414] "containerMap: removing container" podUID="e5faeded-3cb2-4229-9710-4579376f59cc" containerName="revoke-keys" Oct 30 20:07:47.979709 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:47.977037 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5faeded-3cb2-4229-9710-4579376f59cc" containerName="revoke-keys" Oct 30 20:07:47.979709 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:47.977220 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" containerName="main" Oct 30 20:07:47.979709 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:47.977240 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5faeded-3cb2-4229-9710-4579376f59cc" containerName="revoke-keys" Oct 30 20:07:47.979709 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:47.977255 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f60e22d-5bad-4327-b28c-4f618abce710" containerName="revoke-keys" Oct 30 20:07:48.036190 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:47.996898 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0350007e-v6mjg" Oct 30 20:07:48.112767 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:48.112722 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b07cf7ca-5ebb-49c6-a757-1243524af40f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0350007e-v6mjg\" (UID: \"b07cf7ca-5ebb-49c6-a757-1243524af40f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0350007e-v6mjg" Oct 30 20:07:48.113286 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:48.113258 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b07cf7ca-5ebb-49c6-a757-1243524af40f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0350007e-v6mjg\" (UID: \"b07cf7ca-5ebb-49c6-a757-1243524af40f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0350007e-v6mjg" Oct 30 20:07:48.113458 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:48.113441 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv9zx\" (UniqueName: \"kubernetes.io/projected/b07cf7ca-5ebb-49c6-a757-1243524af40f-kube-api-access-hv9zx\") pod \"maas-api-revoke-sub-0350007e-v6mjg\" (UID: \"b07cf7ca-5ebb-49c6-a757-1243524af40f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0350007e-v6mjg" Oct 30 20:07:48.178636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:48.174328 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:07:48.178636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:48.174398 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:07:48.259730 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:48.233447 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b07cf7ca-5ebb-49c6-a757-1243524af40f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0350007e-v6mjg\" (UID: \"b07cf7ca-5ebb-49c6-a757-1243524af40f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0350007e-v6mjg" Oct 30 20:07:48.259730 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:48.233690 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b07cf7ca-5ebb-49c6-a757-1243524af40f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0350007e-v6mjg\" (UID: \"b07cf7ca-5ebb-49c6-a757-1243524af40f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0350007e-v6mjg" Oct 30 20:07:48.259730 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:48.233774 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv9zx\" (UniqueName: \"kubernetes.io/projected/b07cf7ca-5ebb-49c6-a757-1243524af40f-kube-api-access-hv9zx\") pod \"maas-api-revoke-sub-0350007e-v6mjg\" (UID: \"b07cf7ca-5ebb-49c6-a757-1243524af40f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0350007e-v6mjg" Oct 30 20:07:48.259730 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:48.235076 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b07cf7ca-5ebb-49c6-a757-1243524af40f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0350007e-v6mjg\" (UID: \"b07cf7ca-5ebb-49c6-a757-1243524af40f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0350007e-v6mjg" Oct 30 20:07:48.259730 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:48.238023 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b07cf7ca-5ebb-49c6-a757-1243524af40f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0350007e-v6mjg\" (UID: \"b07cf7ca-5ebb-49c6-a757-1243524af40f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0350007e-v6mjg" Oct 30 20:07:48.263113 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:48.263081 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0350007e-v6mjg"] Oct 30 20:07:48.272594 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:48.272562 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6237426-dcvxw"] Oct 30 20:07:48.272939 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:48.272921 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv9zx\" (UniqueName: \"kubernetes.io/projected/b07cf7ca-5ebb-49c6-a757-1243524af40f-kube-api-access-hv9zx\") pod \"maas-api-revoke-sub-0350007e-v6mjg\" (UID: \"b07cf7ca-5ebb-49c6-a757-1243524af40f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0350007e-v6mjg" Oct 30 20:07:48.281704 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:48.281671 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6237426-dcvxw"] Oct 30 20:07:48.305744 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:48.305708 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0350007e-v6mjg" Oct 30 20:07:48.647573 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:07:48.647536 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb07cf7ca_5ebb_49c6_a757_1243524af40f.slice/crio-8e72d11e5dc85203e99289acfcad566d891f8533e9d63115a698a77867530655 WatchSource:0}: Error finding container 8e72d11e5dc85203e99289acfcad566d891f8533e9d63115a698a77867530655: Status 404 returned error can't find the container with id 8e72d11e5dc85203e99289acfcad566d891f8533e9d63115a698a77867530655 Oct 30 20:07:48.648903 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:48.648876 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0350007e-v6mjg"] Oct 30 20:07:48.806708 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:48.804948 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb43b116-cc3d-4d6b-a0c0-06dd170822ed" path="/var/lib/kubelet/pods/fb43b116-cc3d-4d6b-a0c0-06dd170822ed/volumes" Oct 30 20:07:49.348175 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:49.348135 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0350007e-v6mjg" event={"ID":"b07cf7ca-5ebb-49c6-a757-1243524af40f","Type":"ContainerStarted","Data":"725d8881240d39c401290b472f11172ec5422a72d839bd432b4511d02f60a0ba"} Oct 30 20:07:49.348811 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:49.348780 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0350007e-v6mjg" event={"ID":"b07cf7ca-5ebb-49c6-a757-1243524af40f","Type":"ContainerStarted","Data":"8e72d11e5dc85203e99289acfcad566d891f8533e9d63115a698a77867530655"} Oct 30 20:07:49.418913 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:49.408899 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0350007e-v6mjg" podStartSLOduration=2.408875222 podStartE2EDuration="2.408875222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:49.40037636 +0000 UTC m=+1230.813090180" watchObservedRunningTime="2026-10-30 20:07:49.408875222 +0000 UTC m=+1230.821589002" Oct 30 20:07:50.112885 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:50.112840 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41ff877d-ps998"] Oct 30 20:07:50.155316 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:50.155235 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41ff877d-ps998"] Oct 30 20:07:50.399634 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:50.388722 5088 generic.go:356] "Generic (PLEG): container finished" podID="b07cf7ca-5ebb-49c6-a757-1243524af40f" containerID="725d8881240d39c401290b472f11172ec5422a72d839bd432b4511d02f60a0ba" exitCode=0 Oct 30 20:07:50.399634 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:50.389072 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0350007e-v6mjg" event={"ID":"b07cf7ca-5ebb-49c6-a757-1243524af40f","Type":"ContainerDied","Data":"725d8881240d39c401290b472f11172ec5422a72d839bd432b4511d02f60a0ba"} Oct 30 20:07:50.896474 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:50.895806 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8d2c833-630a-4bcd-a943-f576ca32d634" path="/var/lib/kubelet/pods/d8d2c833-630a-4bcd-a943-f576ca32d634/volumes" Oct 30 20:07:51.086425 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:51.086371 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:51.145147 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:51.145102 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f10d2-58d98b4775-jq5gd" podUID="24f9b32b-8908-4adf-bda3-5cfc8cf34bc5" containerName="payload-pre-processing" containerID="cri-o://489b0a7cc51dbb214bc5e596a066ac15f465bf372ee1bc62015e49f46dc7ae03" gracePeriod=25 Oct 30 20:07:51.323702 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:51.321984 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:51.461638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:51.460802 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:51.617696 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:51.609833 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" podUID="fec493c7-3f5a-459e-99e0-a4c816d5a907" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:51.717594 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:51.717537 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 20:07:51.718255 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:51.717658 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:52.120724 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:52.115243 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:52.120724 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:52.115953 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 20:07:52.132648 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:52.128253 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-62f4a1-645b9d8589-k6rx9" podUID="933b9fd2-86f8-4f62-a3e5-48636487d9b0" containerName="payload-processing" containerID="cri-o://133ce1c35e98bbdfce32864378f02f84461a01e8d0bd8ae7160251bde03e9ece" gracePeriod=25 Oct 30 20:07:52.398646 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:52.398517 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 20:07:52.399830 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:52.399806 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:07:52.584886 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:52.584840 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 20:07:52.613769 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:52.602215 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:07:52.613769 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:52.612733 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" podUID="fec493c7-3f5a-459e-99e0-a4c816d5a907" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:52.613769 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:52.613694 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" Oct 30 20:07:52.765775 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:52.765733 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:52.771641 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:52.768070 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:07:52.813812 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:52.785151 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 30 20:07:53.239472 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:53.202523 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 30 20:07:53.415635 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:53.410722 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 20:07:53.587718 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:07:53.582712 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:07:53.600643 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:53.591734 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:53.667643 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:53.664100 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" podUID="fec493c7-3f5a-459e-99e0-a4c816d5a907" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:53.785294 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:53.785253 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:53.879372 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:53.786142 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:53.881461 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:53.881428 5088 patch_prober.go:29] interesting pod/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:07:53.881755 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:53.881711 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" podUID="e402d580-e9a6-4e35-a2ff-e5ef2d78eb90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:54.212829 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:54.212717 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:54.388645 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:07:54.385296 5088 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.6s" Oct 30 20:07:54.480848 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:54.426879 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:54.481380 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:54.481356 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" Oct 30 20:07:54.515721 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:54.515687 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt"] Oct 30 20:07:54.518452 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:54.518416 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" podUID="cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980" containerName="istio-proxy" containerID="cri-o://af78a77d20af1212cd0cbe7378f61b37a49b69e6b21ab698149d0670432b8a81" gracePeriod=30 Oct 30 20:07:54.606044 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:54.605999 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:54.681967 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:54.681923 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl"] Oct 30 20:07:54.684455 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:54.684416 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" podUID="4789c6d9-e328-4890-a2b3-a2fdbfdb04b9" containerName="istio-proxy" containerID="cri-o://8df290432378123b70d4427acfe82432d938ea3a85aa1b705c55e635cde745c5" gracePeriod=30 Oct 30 20:07:54.686119 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:54.682285 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" podUID="fec493c7-3f5a-459e-99e0-a4c816d5a907" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": context deadline exceeded" Oct 30 20:07:54.901634 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:54.882021 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:54.901634 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:54.882418 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 30 20:07:55.250713 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:55.229434 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 30 20:07:55.284657 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:55.253480 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" containerID="cri-o://108f700feb46b39497c4425638937653dc8316b3ea9c281460a178cf185138d2" gracePeriod=45 Oct 30 20:07:55.338800 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:55.338688 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" containerID="cri-o://509d90b132ac33c180da539c5def59572e449b8c1dd393a8ba5dbb9f327b4c73" gracePeriod=45 Oct 30 20:07:55.428679 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:55.427508 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7"] Oct 30 20:07:55.429681 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:55.429639 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" podUID="f2757424-fb74-48ff-82c3-ccb79a27fd2a" containerName="istio-proxy" containerID="cri-o://15c727db7065a1567294b0f8b8556b22587ec02b41504624305b564a569fbb00" gracePeriod=30 Oct 30 20:07:55.508909 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:55.502970 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:55.791706 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:55.784191 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz"] Oct 30 20:07:55.791706 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:55.784511 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" podUID="678518a9-8731-4f6f-a7a1-4b0e64070e92" containerName="istio-proxy" containerID="cri-o://8e3731c9ad46f1c7da2c1fed08fddf94087b20095f37969b667fa8d70bfbcb89" gracePeriod=30 Oct 30 20:07:55.808647 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:55.798641 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:55.887383 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:55.887337 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 30 20:07:55.888985 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:55.888949 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 20:07:55.890100 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:55.890074 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" podUID="fec493c7-3f5a-459e-99e0-a4c816d5a907" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": context deadline exceeded" Oct 30 20:07:55.933261 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:55.931757 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": dial tcp 10.132.0.64:8000: connect: connection refused" Oct 30 20:07:56.265785 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:56.258027 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:56.465815 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:56.463744 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" podUID="fec493c7-3f5a-459e-99e0-a4c816d5a907" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:56.597722 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:56.597674 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 20:07:56.677645 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:56.660817 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": dial tcp 10.132.0.83:8000: connect: connection refused" Oct 30 20:07:56.747635 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:56.715128 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded" Oct 30 20:07:56.938705 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:56.865337 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:56.992641 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:56.953052 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": dial tcp 10.132.0.64:8000: connect: connection refused" Oct 30 20:07:57.093640 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:56.980945 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 30 20:07:57.129703 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:57.100442 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" podUID="fec493c7-3f5a-459e-99e0-a4c816d5a907" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": context deadline exceeded" Oct 30 20:07:57.258085 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:57.258050 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0350007e-v6mjg" Oct 30 20:07:57.312854 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:57.312792 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 30 20:07:57.388879 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:57.388822 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": dial tcp 10.132.0.83:8000: connect: connection refused" Oct 30 20:07:57.562740 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:57.536466 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-efbac3-66db987bc9-ptftf" podUID="0ee0e296-2cf3-4cdc-9340-d202e75c6a14" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.56:50052\" within 1s: context deadline exceeded" Oct 30 20:07:57.867951 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:57.867831 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:57.899639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:57.894310 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b07cf7ca-5ebb-49c6-a757-1243524af40f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b07cf7ca-5ebb-49c6-a757-1243524af40f-maas-api-openshift-ca\") pod \"b07cf7ca-5ebb-49c6-a757-1243524af40f\" (UID: \"b07cf7ca-5ebb-49c6-a757-1243524af40f\") " Oct 30 20:07:57.899639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:57.894477 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b07cf7ca-5ebb-49c6-a757-1243524af40f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b07cf7ca-5ebb-49c6-a757-1243524af40f-maas-api-xks-ca\") pod \"b07cf7ca-5ebb-49c6-a757-1243524af40f\" (UID: \"b07cf7ca-5ebb-49c6-a757-1243524af40f\") " Oct 30 20:07:57.899639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:57.894540 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b07cf7ca-5ebb-49c6-a757-1243524af40f-kube-api-access-hv9zx\" (UniqueName: \"kubernetes.io/projected/b07cf7ca-5ebb-49c6-a757-1243524af40f-kube-api-access-hv9zx\") pod \"b07cf7ca-5ebb-49c6-a757-1243524af40f\" (UID: \"b07cf7ca-5ebb-49c6-a757-1243524af40f\") " Oct 30 20:07:57.951645 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:07:57.926989 5088 logging.go:55] [core] [Channel #668 SubChannel #669]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.56:50052", ServerName: "10.132.0.56:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.56:50052: operation was canceled" Oct 30 20:07:57.978736 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:57.928314 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b07cf7ca-5ebb-49c6-a757-1243524af40f-maas-api-openshift-ca" pod "b07cf7ca-5ebb-49c6-a757-1243524af40f" (UID: "b07cf7ca-5ebb-49c6-a757-1243524af40f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:07:58.085647 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:58.048066 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 30 20:07:58.100975 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:58.052250 5088 patch_prober.go:29] interesting pod/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:07:58.147633 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:58.101746 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" podUID="fec493c7-3f5a-459e-99e0-a4c816d5a907" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:58.148002 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:58.052430 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b07cf7ca-5ebb-49c6-a757-1243524af40f-kube-api-access-hv9zx" pod "b07cf7ca-5ebb-49c6-a757-1243524af40f" (UID: "b07cf7ca-5ebb-49c6-a757-1243524af40f"). InnerVolumeSpecName "kube-api-access-hv9zx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:07:58.212832 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:58.212752 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" podUID="1f2b960b-8db8-4b84-b6d1-9ac12823e278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:58.231893 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:58.226831 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b07cf7ca-5ebb-49c6-a757-1243524af40f-maas-api-xks-ca" pod "b07cf7ca-5ebb-49c6-a757-1243524af40f" (UID: "b07cf7ca-5ebb-49c6-a757-1243524af40f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:07:58.289638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:58.269828 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": dial tcp 10.132.0.64:8000: connect: connection refused" Oct 30 20:07:58.379931 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:58.368389 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b07cf7ca-5ebb-49c6-a757-1243524af40f-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:07:58.380209 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:58.380175 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b07cf7ca-5ebb-49c6-a757-1243524af40f-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:07:58.380344 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:58.380329 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv9zx\" (UniqueName: \"kubernetes.io/projected/b07cf7ca-5ebb-49c6-a757-1243524af40f-kube-api-access-hv9zx\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:07:58.380725 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:58.380698 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:58.543653 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:58.540448 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": dial tcp 10.132.0.83:8000: connect: connection refused" Oct 30 20:07:58.576639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:58.573225 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-62f4a1-645b9d8589-k6rx9" podUID="933b9fd2-86f8-4f62-a3e5-48636487d9b0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:07:58.618645 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:58.609116 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wtlm7" podUID="419ffdb0-c223-4426-872a-50e3073812e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:58.671056 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:58.647215 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0350007e-v6mjg" Oct 30 20:07:58.694641 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:58.688725 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f10d2-58d98b4775-jq5gd" podUID="24f9b32b-8908-4adf-bda3-5cfc8cf34bc5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:07:58.723641 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:58.700135 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": dial tcp 10.132.0.64:8000: connect: connection refused" Oct 30 20:07:58.739320 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:07:58.739279 5088 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.606s" Oct 30 20:07:58.739602 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:58.739560 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0350007e-v6mjg" event={"ID":"b07cf7ca-5ebb-49c6-a757-1243524af40f","Type":"ContainerDied","Data":"8e72d11e5dc85203e99289acfcad566d891f8533e9d63115a698a77867530655"} Oct 30 20:07:58.747968 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:58.747938 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e72d11e5dc85203e99289acfcad566d891f8533e9d63115a698a77867530655" Oct 30 20:07:58.879635 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:58.876549 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:58.948269 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:58.948168 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:59.108645 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:59.103026 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 30 20:07:59.152685 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:59.152628 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" podUID="fec493c7-3f5a-459e-99e0-a4c816d5a907" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:59.388648 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:59.370807 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": dial tcp 10.132.0.83:8000: connect: connection refused" Oct 30 20:07:59.410873 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:59.410777 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:59.452663 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:59.436987 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq"] Oct 30 20:07:59.452663 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:59.437350 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" podUID="a255c4a5-c090-49d2-84ba-536dfb9b8a69" containerName="istio-proxy" containerID="cri-o://500b9e0dbc558d86839f26b86c64e2b7e81db3e1fda3e37f9dcbb093e05578d4" gracePeriod=30 Oct 30 20:07:59.542459 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:59.542415 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": context deadline exceeded" Oct 30 20:07:59.905776 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:59.902121 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": dial tcp 10.132.0.64:8000: connect: connection refused" Oct 30 20:07:59.971639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:07:59.937989 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z"] Oct 30 20:08:00.220695 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.204317 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" podUID="f4b44c6e-891a-4073-9192-035aeaad10a4" containerName="istio-proxy" containerID="cri-o://3a6fcdf62cfa94bb23a47d8095c3f93e63a21035e5267a54f9439fb5482243d5" gracePeriod=30 Oct 30 20:08:00.220695 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.109748 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:00.220695 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.145739 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:08:00.220695 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.207490 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" podUID="14b2ad56-7181-420c-8ad1-bdc00f25e928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:00.256028 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.183225 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" podUID="fec493c7-3f5a-459e-99e0-a4c816d5a907" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:00.273868 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.273765 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 20:08:00.320603 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.320559 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": dial tcp 10.132.0.83:8000: connect: connection refused" Oct 30 20:08:00.395145 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.395109 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 20:08:00.420968 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.420935 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:08:00.424683 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.424657 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp"] Oct 30 20:08:00.425084 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.425052 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" podUID="14b2ad56-7181-420c-8ad1-bdc00f25e928" containerName="istio-proxy" containerID="cri-o://4358d77f290648c02af2f9a40e5a7ed64a0a45b4e236f0c73a7c725314a93bd1" gracePeriod=30 Oct 30 20:08:00.428181 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.428154 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" Oct 30 20:08:00.439184 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:08:00.439129 5088 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb07cf7ca_5ebb_49c6_a757_1243524af40f.slice\": RecentStats: unable to find data in memory cache]" Oct 30 20:08:00.483863 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.483575 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" Oct 30 20:08:00.624933 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.613183 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8"] Oct 30 20:08:00.624933 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.613544 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" podUID="ab112498-5d12-44e0-85b3-0c9f5dc003d6" containerName="istio-proxy" containerID="cri-o://e68213262edc0b90dd827831d6b42294e4f6c2e96e19943a2dbf1a2894f0065b" gracePeriod=30 Oct 30 20:08:00.663239 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.663203 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a199d8f-hqq5n"] Oct 30 20:08:00.667898 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.667862 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a199d8f-hqq5n"] Oct 30 20:08:00.677384 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.677352 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m"] Oct 30 20:08:00.677656 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.677607 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" podUID="b56fca64-1150-4198-945e-7e830300c11d" containerName="istio-proxy" containerID="cri-o://9eaf99979ceb984cf41aa039dda144ebbde0304a942ce7350ed64c04c8537d33" gracePeriod=30 Oct 30 20:08:00.683041 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.683009 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk"] Oct 30 20:08:00.683261 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.683236 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" podUID="0b36276d-bc5b-4415-8cf9-461aee572f3a" containerName="istio-proxy" containerID="cri-o://d53c6f364c7241cc108097bf75f911b385021a7733320a1ccf7a4a977d146d8d" gracePeriod=30 Oct 30 20:08:00.687975 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.687951 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw"] Oct 30 20:08:00.688193 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.688167 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" podUID="baca0e89-a102-4354-af54-c14ccc418923" containerName="istio-proxy" containerID="cri-o://945cf965ae8ee1f5738db7747631301dc305f411de087b51d4c826da927c53a1" gracePeriod=30 Oct 30 20:08:00.688370 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.688346 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": dial tcp 10.132.0.64:8000: connect: connection refused" Oct 30 20:08:00.693107 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.693082 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s"] Oct 30 20:08:00.693327 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.693303 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" podUID="05a46855-a143-4c6d-8c87-404dc271bdb3" containerName="istio-proxy" containerID="cri-o://e2852ce9442a037ded3c3a1a2d61a71a7261186d4e0bd39c8dbc326bacd5a9b7" gracePeriod=30 Oct 30 20:08:00.697960 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.697937 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bebf4566-g9fwj"] Oct 30 20:08:00.698777 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.698760 5088 cpu_manager.go:414] "containerMap: removing container" podUID="b07cf7ca-5ebb-49c6-a757-1243524af40f" containerName="revoke-keys" Oct 30 20:08:00.698841 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.698778 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b07cf7ca-5ebb-49c6-a757-1243524af40f" containerName="revoke-keys" Oct 30 20:08:00.698909 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.698900 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="b07cf7ca-5ebb-49c6-a757-1243524af40f" containerName="revoke-keys" Oct 30 20:08:00.702723 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.702708 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bebf4566-g9fwj"] Oct 30 20:08:00.702801 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.702791 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebf4566-g9fwj" Oct 30 20:08:00.719559 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.719535 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:08:00.719716 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.719668 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:08:00.762096 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.762015 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a6dccf5-7f1b-42c8-a974-1cf5510af915" path="/var/lib/kubelet/pods/7a6dccf5-7f1b-42c8-a974-1cf5510af915/volumes" Oct 30 20:08:00.840761 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.838992 5088 patch_prober.go:29] interesting pod/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:08:00.840761 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.839076 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" podUID="ab112498-5d12-44e0-85b3-0c9f5dc003d6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:08:00.894358 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.894322 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28hpx\" (UniqueName: \"kubernetes.io/projected/ad2f9107-20fe-4ae3-ae9a-e92c315e5a45-kube-api-access-28hpx\") pod \"maas-api-revoke-sub-bebf4566-g9fwj\" (UID: \"ad2f9107-20fe-4ae3-ae9a-e92c315e5a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebf4566-g9fwj" Oct 30 20:08:00.894541 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.894502 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad2f9107-20fe-4ae3-ae9a-e92c315e5a45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bebf4566-g9fwj\" (UID: \"ad2f9107-20fe-4ae3-ae9a-e92c315e5a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebf4566-g9fwj" Oct 30 20:08:00.894682 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.894635 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad2f9107-20fe-4ae3-ae9a-e92c315e5a45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bebf4566-g9fwj\" (UID: \"ad2f9107-20fe-4ae3-ae9a-e92c315e5a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebf4566-g9fwj" Oct 30 20:08:00.996145 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.996106 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad2f9107-20fe-4ae3-ae9a-e92c315e5a45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bebf4566-g9fwj\" (UID: \"ad2f9107-20fe-4ae3-ae9a-e92c315e5a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebf4566-g9fwj" Oct 30 20:08:00.996367 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.996230 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad2f9107-20fe-4ae3-ae9a-e92c315e5a45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bebf4566-g9fwj\" (UID: \"ad2f9107-20fe-4ae3-ae9a-e92c315e5a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebf4566-g9fwj" Oct 30 20:08:00.996367 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.996334 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28hpx\" (UniqueName: \"kubernetes.io/projected/ad2f9107-20fe-4ae3-ae9a-e92c315e5a45-kube-api-access-28hpx\") pod \"maas-api-revoke-sub-bebf4566-g9fwj\" (UID: \"ad2f9107-20fe-4ae3-ae9a-e92c315e5a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebf4566-g9fwj" Oct 30 20:08:00.996924 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.996898 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad2f9107-20fe-4ae3-ae9a-e92c315e5a45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bebf4566-g9fwj\" (UID: \"ad2f9107-20fe-4ae3-ae9a-e92c315e5a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebf4566-g9fwj" Oct 30 20:08:00.998767 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:00.998720 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad2f9107-20fe-4ae3-ae9a-e92c315e5a45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bebf4566-g9fwj\" (UID: \"ad2f9107-20fe-4ae3-ae9a-e92c315e5a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebf4566-g9fwj" Oct 30 20:08:01.022857 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.022784 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28hpx\" (UniqueName: \"kubernetes.io/projected/ad2f9107-20fe-4ae3-ae9a-e92c315e5a45-kube-api-access-28hpx\") pod \"maas-api-revoke-sub-bebf4566-g9fwj\" (UID: \"ad2f9107-20fe-4ae3-ae9a-e92c315e5a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebf4566-g9fwj" Oct 30 20:08:01.045449 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.045412 5088 generic.go:356] "Generic (PLEG): container finished" podID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerID="108f700feb46b39497c4425638937653dc8316b3ea9c281460a178cf185138d2" exitCode=0 Oct 30 20:08:01.045630 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.045490 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" event={"ID":"c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21","Type":"ContainerDied","Data":"108f700feb46b39497c4425638937653dc8316b3ea9c281460a178cf185138d2"} Oct 30 20:08:01.047337 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.047313 5088 generic.go:356] "Generic (PLEG): container finished" podID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerID="509d90b132ac33c180da539c5def59572e449b8c1dd393a8ba5dbb9f327b4c73" exitCode=0 Oct 30 20:08:01.047426 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.047412 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" event={"ID":"682a764a-19ca-49e8-a1d4-1258b8d74bf4","Type":"ContainerDied","Data":"509d90b132ac33c180da539c5def59572e449b8c1dd393a8ba5dbb9f327b4c73"} Oct 30 20:08:01.303713 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.303582 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": dial tcp 10.132.0.83:8000: connect: connection refused" Oct 30 20:08:01.317898 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.317861 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebf4566-g9fwj" Oct 30 20:08:01.406667 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.404824 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Oct 30 20:08:01.406667 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.404885 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" podUID="baca0e89-a102-4354-af54-c14ccc418923" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:08:01.518804 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.518764 5088 patch_prober.go:29] interesting pod/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:08:01.518962 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.518840 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" podUID="f2757424-fb74-48ff-82c3-ccb79a27fd2a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:08:01.574670 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.573146 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:08:01.607813 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.607778 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21-kube-api-access-4qqfn\" (UniqueName: \"kubernetes.io/projected/c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21-kube-api-access-4qqfn\") pod \"c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21\" (UID: \"c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21\") " Oct 30 20:08:01.608020 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.607842 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21-model-cache\") pod \"c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21\" (UID: \"c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21\") " Oct 30 20:08:01.608020 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.607886 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21-dshm\") pod \"c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21\" (UID: \"c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21\") " Oct 30 20:08:01.608020 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.608007 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21-home\" (UniqueName: \"kubernetes.io/empty-dir/c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21-home\") pod \"c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21\" (UID: \"c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21\") " Oct 30 20:08:01.608590 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.608249 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21-tls-certs\" (UniqueName: \"kubernetes.io/secret/c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21-tls-certs\") pod \"c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21\" (UID: \"c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21\") " Oct 30 20:08:01.608590 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.608463 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21-home" pod "c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" (UID: "c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:01.608590 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.608482 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21-tmp-dir\") pod \"c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21\" (UID: \"c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21\") " Oct 30 20:08:01.608846 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.608746 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21-tmp-dir" pod "c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" (UID: "c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:01.608920 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.608899 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21-model-cache" pod "c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" (UID: "c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:01.609497 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.609423 5088 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21-tmp-dir\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:01.609497 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.609454 5088 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21-model-cache\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:01.609497 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.609467 5088 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21-home\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:01.620656 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.614338 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21-kube-api-access-4qqfn" pod "c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" (UID: "c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21"). InnerVolumeSpecName "kube-api-access-4qqfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:01.620656 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.615905 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21-tls-certs" pod "c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" (UID: "c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:08:01.620656 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.617702 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21-dshm" pod "c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" (UID: "c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:01.641841 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.641805 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bebf4566-g9fwj"] Oct 30 20:08:01.668348 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.668325 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:08:01.710183 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.710149 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/682a764a-19ca-49e8-a1d4-1258b8d74bf4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/682a764a-19ca-49e8-a1d4-1258b8d74bf4-model-cache\") pod \"682a764a-19ca-49e8-a1d4-1258b8d74bf4\" (UID: \"682a764a-19ca-49e8-a1d4-1258b8d74bf4\") " Oct 30 20:08:01.710377 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.710207 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/682a764a-19ca-49e8-a1d4-1258b8d74bf4-tls-certs\" (UniqueName: \"kubernetes.io/secret/682a764a-19ca-49e8-a1d4-1258b8d74bf4-tls-certs\") pod \"682a764a-19ca-49e8-a1d4-1258b8d74bf4\" (UID: \"682a764a-19ca-49e8-a1d4-1258b8d74bf4\") " Oct 30 20:08:01.710377 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.710250 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/682a764a-19ca-49e8-a1d4-1258b8d74bf4-kube-api-access-dtf95\" (UniqueName: \"kubernetes.io/projected/682a764a-19ca-49e8-a1d4-1258b8d74bf4-kube-api-access-dtf95\") pod \"682a764a-19ca-49e8-a1d4-1258b8d74bf4\" (UID: \"682a764a-19ca-49e8-a1d4-1258b8d74bf4\") " Oct 30 20:08:01.710377 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.710366 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/682a764a-19ca-49e8-a1d4-1258b8d74bf4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/682a764a-19ca-49e8-a1d4-1258b8d74bf4-tmp-dir\") pod \"682a764a-19ca-49e8-a1d4-1258b8d74bf4\" (UID: \"682a764a-19ca-49e8-a1d4-1258b8d74bf4\") " Oct 30 20:08:01.710559 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.710473 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/682a764a-19ca-49e8-a1d4-1258b8d74bf4-model-cache" pod "682a764a-19ca-49e8-a1d4-1258b8d74bf4" (UID: "682a764a-19ca-49e8-a1d4-1258b8d74bf4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:01.710559 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.710547 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/682a764a-19ca-49e8-a1d4-1258b8d74bf4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/682a764a-19ca-49e8-a1d4-1258b8d74bf4-dshm\") pod \"682a764a-19ca-49e8-a1d4-1258b8d74bf4\" (UID: \"682a764a-19ca-49e8-a1d4-1258b8d74bf4\") " Oct 30 20:08:01.710690 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.710643 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/682a764a-19ca-49e8-a1d4-1258b8d74bf4-home\" (UniqueName: \"kubernetes.io/empty-dir/682a764a-19ca-49e8-a1d4-1258b8d74bf4-home\") pod \"682a764a-19ca-49e8-a1d4-1258b8d74bf4\" (UID: \"682a764a-19ca-49e8-a1d4-1258b8d74bf4\") " Oct 30 20:08:01.710888 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.710855 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/682a764a-19ca-49e8-a1d4-1258b8d74bf4-tmp-dir" pod "682a764a-19ca-49e8-a1d4-1258b8d74bf4" (UID: "682a764a-19ca-49e8-a1d4-1258b8d74bf4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:01.711116 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.711094 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/682a764a-19ca-49e8-a1d4-1258b8d74bf4-home" pod "682a764a-19ca-49e8-a1d4-1258b8d74bf4" (UID: "682a764a-19ca-49e8-a1d4-1258b8d74bf4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:01.711556 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.711531 5088 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/682a764a-19ca-49e8-a1d4-1258b8d74bf4-home\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:01.711714 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.711561 5088 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/682a764a-19ca-49e8-a1d4-1258b8d74bf4-model-cache\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:01.711714 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.711581 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qqfn\" (UniqueName: \"kubernetes.io/projected/c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21-kube-api-access-4qqfn\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:01.711714 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.711594 5088 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21-dshm\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:01.711714 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.711627 5088 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/682a764a-19ca-49e8-a1d4-1258b8d74bf4-tmp-dir\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:01.711714 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.711641 5088 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21-tls-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:01.712815 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.712788 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/682a764a-19ca-49e8-a1d4-1258b8d74bf4-tls-certs" pod "682a764a-19ca-49e8-a1d4-1258b8d74bf4" (UID: "682a764a-19ca-49e8-a1d4-1258b8d74bf4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:08:01.712909 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.712853 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/682a764a-19ca-49e8-a1d4-1258b8d74bf4-kube-api-access-dtf95" pod "682a764a-19ca-49e8-a1d4-1258b8d74bf4" (UID: "682a764a-19ca-49e8-a1d4-1258b8d74bf4"). InnerVolumeSpecName "kube-api-access-dtf95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:01.713098 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.713083 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/682a764a-19ca-49e8-a1d4-1258b8d74bf4-dshm" pod "682a764a-19ca-49e8-a1d4-1258b8d74bf4" (UID: "682a764a-19ca-49e8-a1d4-1258b8d74bf4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:01.813201 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.813130 5088 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/682a764a-19ca-49e8-a1d4-1258b8d74bf4-dshm\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:01.813201 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.813157 5088 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/682a764a-19ca-49e8-a1d4-1258b8d74bf4-tls-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:01.813201 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.813167 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtf95\" (UniqueName: \"kubernetes.io/projected/682a764a-19ca-49e8-a1d4-1258b8d74bf4-kube-api-access-dtf95\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:01.952706 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.952660 5088 patch_prober.go:29] interesting pod/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:08:01.952875 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:01.952714 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" podUID="678518a9-8731-4f6f-a7a1-4b0e64070e92" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:08:02.072689 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:02.072588 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" event={"ID":"c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21","Type":"ContainerDied","Data":"5495d66b685d82a2e53bdc781873b30d6823504c503d3e26f1fd7691c3119811"} Oct 30 20:08:02.072967 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:02.072946 5088 scope.go:122] "RemoveContainer" containerID="108f700feb46b39497c4425638937653dc8316b3ea9c281460a178cf185138d2" Oct 30 20:08:02.073241 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:02.073225 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq" Oct 30 20:08:02.102486 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:02.102450 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" event={"ID":"682a764a-19ca-49e8-a1d4-1258b8d74bf4","Type":"ContainerDied","Data":"4572181a77380ac8a27d95ed5858169d915b823a20275780c6429bd52f81fa20"} Oct 30 20:08:02.102676 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:02.102543 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk" Oct 30 20:08:02.104444 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:02.104418 5088 scope.go:122] "RemoveContainer" containerID="509d90b132ac33c180da539c5def59572e449b8c1dd393a8ba5dbb9f327b4c73" Oct 30 20:08:02.107668 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:02.107553 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebf4566-g9fwj" event={"ID":"ad2f9107-20fe-4ae3-ae9a-e92c315e5a45","Type":"ContainerStarted","Data":"2258c8627d35cec8ffa54f122716d85681ddf3076bc12d00d6e01ae08750152b"} Oct 30 20:08:02.155255 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:02.155225 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq"] Oct 30 20:08:02.161285 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:02.161260 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-62f4a1/simulator-w0-62f4a1-kserve-7df66b8675-wmkxq"] Oct 30 20:08:02.200481 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:02.200449 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk"] Oct 30 20:08:02.209869 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:02.209840 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5f10d2/distinct-2-w1-5f10d2-kserve-586bf549b7-fglwk"] Oct 30 20:08:02.249950 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:02.249894 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebf4566-g9fwj" podStartSLOduration=3.249880259 podStartE2EDuration="3.249880259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:02.248902694 +0000 UTC m=+1243.661616502" watchObservedRunningTime="2026-10-30 20:08:02.249880259 +0000 UTC m=+1243.662594060" Oct 30 20:08:02.353654 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:02.353558 5088 patch_prober.go:29] interesting pod/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:08:02.353654 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:02.353628 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" podUID="4789c6d9-e328-4890-a2b3-a2fdbfdb04b9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:08:02.740829 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:02.740794 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" path="/var/lib/kubelet/pods/682a764a-19ca-49e8-a1d4-1258b8d74bf4/volumes" Oct 30 20:08:02.741326 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:02.741311 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" path="/var/lib/kubelet/pods/c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21/volumes" Oct 30 20:08:02.911355 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:02.911270 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Oct 30 20:08:02.911355 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:02.911329 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" podUID="05a46855-a143-4c6d-8c87-404dc271bdb3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:08:03.120568 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:03.120531 5088 generic.go:356] "Generic (PLEG): container finished" podID="ad2f9107-20fe-4ae3-ae9a-e92c315e5a45" containerID="1fa04071a7cfb0edc7dcfaee6d0d27db5d0ee935ee8812a7b7dc5d0d4f1612ed" exitCode=0 Oct 30 20:08:03.120779 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:03.120570 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebf4566-g9fwj" event={"ID":"ad2f9107-20fe-4ae3-ae9a-e92c315e5a45","Type":"ContainerDied","Data":"1fa04071a7cfb0edc7dcfaee6d0d27db5d0ee935ee8812a7b7dc5d0d4f1612ed"} Oct 30 20:08:03.122157 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:03.122133 5088 generic.go:356] "Generic (PLEG): container finished" podID="a4af0759-e61a-48f6-821a-e1ad34261f11" containerID="d6efd7473c24c5e49dc2f91caa389a0281d1692701d86ef43538945ce2d50786" exitCode=0 Oct 30 20:08:03.122290 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:03.122176 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-j6v5l-pmxpq" event={"ID":"a4af0759-e61a-48f6-821a-e1ad34261f11","Type":"ContainerDied","Data":"d6efd7473c24c5e49dc2f91caa389a0281d1692701d86ef43538945ce2d50786"} Oct 30 20:08:03.593179 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:03.593139 5088 patch_prober.go:29] interesting pod/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:08:03.593550 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:03.593196 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" podUID="cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:08:04.057404 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.057365 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c990cbdd-229pr"] Oct 30 20:08:04.063748 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.063719 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e38036ba-75zcz"] Oct 30 20:08:04.069602 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.069574 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c990cbdd-229pr"] Oct 30 20:08:04.074636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.074596 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e38036ba-75zcz"] Oct 30 20:08:04.132511 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.132482 5088 generic.go:356] "Generic (PLEG): container finished" podID="cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980" containerID="af78a77d20af1212cd0cbe7378f61b37a49b69e6b21ab698149d0670432b8a81" exitCode=0 Oct 30 20:08:04.132692 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.132567 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" event={"ID":"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980","Type":"ContainerDied","Data":"af78a77d20af1212cd0cbe7378f61b37a49b69e6b21ab698149d0670432b8a81"} Oct 30 20:08:04.158191 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.158151 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Oct 30 20:08:04.158409 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.158222 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" podUID="0b36276d-bc5b-4415-8cf9-461aee572f3a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:08:04.283238 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.283201 5088 patch_prober.go:29] interesting pod/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:08:04.283413 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.283267 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" podUID="b56fca64-1150-4198-945e-7e830300c11d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:08:04.373861 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.373832 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-j6v5l-pmxpq" Oct 30 20:08:04.378509 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.378478 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebf4566-g9fwj" Oct 30 20:08:04.397090 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.397066 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" Oct 30 20:08:04.466844 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.466804 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-istiod-ca-cert\") pod \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\" (UID: \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\") " Oct 30 20:08:04.467052 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.466937 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad2f9107-20fe-4ae3-ae9a-e92c315e5a45-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad2f9107-20fe-4ae3-ae9a-e92c315e5a45-maas-api-xks-ca\") pod \"ad2f9107-20fe-4ae3-ae9a-e92c315e5a45\" (UID: \"ad2f9107-20fe-4ae3-ae9a-e92c315e5a45\") " Oct 30 20:08:04.467052 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.466974 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-istio-envoy\") pod \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\" (UID: \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\") " Oct 30 20:08:04.467052 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.467027 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4af0759-e61a-48f6-821a-e1ad34261f11-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4af0759-e61a-48f6-821a-e1ad34261f11-service-ca-bundle\") pod \"a4af0759-e61a-48f6-821a-e1ad34261f11\" (UID: \"a4af0759-e61a-48f6-821a-e1ad34261f11\") " Oct 30 20:08:04.467216 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.467141 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4af0759-e61a-48f6-821a-e1ad34261f11-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4af0759-e61a-48f6-821a-e1ad34261f11-trusted-ca-bundle\") pod \"a4af0759-e61a-48f6-821a-e1ad34261f11\" (UID: \"a4af0759-e61a-48f6-821a-e1ad34261f11\") " Oct 30 20:08:04.467216 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.467189 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-istiod-ca-cert" pod "cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980" (UID: "cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:04.467326 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.467249 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-workload-certs\") pod \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\" (UID: \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\") " Oct 30 20:08:04.467424 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.467404 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-workload-socket\") pod \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\" (UID: \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\") " Oct 30 20:08:04.467643 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.467498 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4af0759-e61a-48f6-821a-e1ad34261f11-service-ca-bundle" pod "a4af0759-e61a-48f6-821a-e1ad34261f11" (UID: "a4af0759-e61a-48f6-821a-e1ad34261f11"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:04.467643 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.467512 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad2f9107-20fe-4ae3-ae9a-e92c315e5a45-kube-api-access-28hpx\" (UniqueName: \"kubernetes.io/projected/ad2f9107-20fe-4ae3-ae9a-e92c315e5a45-kube-api-access-28hpx\") pod \"ad2f9107-20fe-4ae3-ae9a-e92c315e5a45\" (UID: \"ad2f9107-20fe-4ae3-ae9a-e92c315e5a45\") " Oct 30 20:08:04.467643 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.467533 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-workload-certs" pod "cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980" (UID: "cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:04.467643 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.467569 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4af0759-e61a-48f6-821a-e1ad34261f11-trusted-ca-bundle" pod "a4af0759-e61a-48f6-821a-e1ad34261f11" (UID: "a4af0759-e61a-48f6-821a-e1ad34261f11"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:04.467643 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.467590 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-workload-socket" pod "cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980" (UID: "cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:04.467862 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.467838 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4af0759-e61a-48f6-821a-e1ad34261f11-kube-api-access-gvw4q\" (UniqueName: \"kubernetes.io/projected/a4af0759-e61a-48f6-821a-e1ad34261f11-kube-api-access-gvw4q\") pod \"a4af0759-e61a-48f6-821a-e1ad34261f11\" (UID: \"a4af0759-e61a-48f6-821a-e1ad34261f11\") " Oct 30 20:08:04.467950 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.467927 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-kube-api-access-jmdrb\" (UniqueName: \"kubernetes.io/projected/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-kube-api-access-jmdrb\") pod \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\" (UID: \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\") " Oct 30 20:08:04.468099 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.467991 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-istio-podinfo\") pod \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\" (UID: \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\") " Oct 30 20:08:04.468099 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.468084 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4af0759-e61a-48f6-821a-e1ad34261f11-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a4af0759-e61a-48f6-821a-e1ad34261f11-archives-path\") pod \"a4af0759-e61a-48f6-821a-e1ad34261f11\" (UID: \"a4af0759-e61a-48f6-821a-e1ad34261f11\") " Oct 30 20:08:04.468229 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.468117 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-istio-token\" (UniqueName: \"kubernetes.io/projected/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-istio-token\") pod \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\" (UID: \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\") " Oct 30 20:08:04.468229 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.468156 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-credential-socket\") pod \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\" (UID: \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\") " Oct 30 20:08:04.468229 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.468197 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-istio-data\") pod \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\" (UID: \"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980\") " Oct 30 20:08:04.468384 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.468253 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad2f9107-20fe-4ae3-ae9a-e92c315e5a45-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad2f9107-20fe-4ae3-ae9a-e92c315e5a45-maas-api-openshift-ca\") pod \"ad2f9107-20fe-4ae3-ae9a-e92c315e5a45\" (UID: \"ad2f9107-20fe-4ae3-ae9a-e92c315e5a45\") " Oct 30 20:08:04.469184 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.469158 5088 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4af0759-e61a-48f6-821a-e1ad34261f11-trusted-ca-bundle\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:04.469284 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.469238 5088 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:04.469284 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.469254 5088 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:04.469284 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.469267 5088 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:04.469461 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.469281 5088 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4af0759-e61a-48f6-821a-e1ad34261f11-service-ca-bundle\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:04.470005 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.469980 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-istio-envoy" pod "cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980" (UID: "cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:04.470005 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.469994 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad2f9107-20fe-4ae3-ae9a-e92c315e5a45-maas-api-openshift-ca" pod "ad2f9107-20fe-4ae3-ae9a-e92c315e5a45" (UID: "ad2f9107-20fe-4ae3-ae9a-e92c315e5a45"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:04.470406 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.470273 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-credential-socket" pod "cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980" (UID: "cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:04.470506 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.470486 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-istio-data" pod "cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980" (UID: "cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:04.472951 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.470510 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad2f9107-20fe-4ae3-ae9a-e92c315e5a45-maas-api-xks-ca" pod "ad2f9107-20fe-4ae3-ae9a-e92c315e5a45" (UID: "ad2f9107-20fe-4ae3-ae9a-e92c315e5a45"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:08:04.472951 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.470690 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4af0759-e61a-48f6-821a-e1ad34261f11-archives-path" pod "a4af0759-e61a-48f6-821a-e1ad34261f11" (UID: "a4af0759-e61a-48f6-821a-e1ad34261f11"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:04.472951 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.471380 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4af0759-e61a-48f6-821a-e1ad34261f11-kube-api-access-gvw4q" pod "a4af0759-e61a-48f6-821a-e1ad34261f11" (UID: "a4af0759-e61a-48f6-821a-e1ad34261f11"). InnerVolumeSpecName "kube-api-access-gvw4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:04.472951 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.471444 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-istio-podinfo" pod "cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980" (UID: "cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:08:04.472951 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.471573 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-kube-api-access-jmdrb" pod "cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980" (UID: "cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980"). InnerVolumeSpecName "kube-api-access-jmdrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:04.472951 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.471660 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad2f9107-20fe-4ae3-ae9a-e92c315e5a45-kube-api-access-28hpx" pod "ad2f9107-20fe-4ae3-ae9a-e92c315e5a45" (UID: "ad2f9107-20fe-4ae3-ae9a-e92c315e5a45"). InnerVolumeSpecName "kube-api-access-28hpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:04.472951 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.472020 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-istio-token" pod "cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980" (UID: "cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:04.570919 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.570838 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28hpx\" (UniqueName: \"kubernetes.io/projected/ad2f9107-20fe-4ae3-ae9a-e92c315e5a45-kube-api-access-28hpx\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:04.570919 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.570880 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvw4q\" (UniqueName: \"kubernetes.io/projected/a4af0759-e61a-48f6-821a-e1ad34261f11-kube-api-access-gvw4q\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:04.570919 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.570895 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmdrb\" (UniqueName: \"kubernetes.io/projected/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-kube-api-access-jmdrb\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:04.570919 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.570909 5088 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:04.570919 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.570922 5088 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a4af0759-e61a-48f6-821a-e1ad34261f11-archives-path\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:04.571248 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.570936 5088 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:04.571248 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.570948 5088 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:04.571248 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.570960 5088 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:04.571248 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.570972 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad2f9107-20fe-4ae3-ae9a-e92c315e5a45-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:04.571248 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.570984 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad2f9107-20fe-4ae3-ae9a-e92c315e5a45-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:04.571248 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.570999 5088 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:04.744237 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.744192 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c713054-69fb-4a8d-8670-19264c9c8e0c" path="/var/lib/kubelet/pods/5c713054-69fb-4a8d-8670-19264c9c8e0c/volumes" Oct 30 20:08:04.744796 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:04.744778 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a" path="/var/lib/kubelet/pods/c7f49ac2-73d1-4789-a8f0-cdab1a9cf52a/volumes" Oct 30 20:08:05.143678 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.143649 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebf4566-g9fwj" Oct 30 20:08:05.143678 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.143665 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebf4566-g9fwj" event={"ID":"ad2f9107-20fe-4ae3-ae9a-e92c315e5a45","Type":"ContainerDied","Data":"2258c8627d35cec8ffa54f122716d85681ddf3076bc12d00d6e01ae08750152b"} Oct 30 20:08:05.143933 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.143725 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2258c8627d35cec8ffa54f122716d85681ddf3076bc12d00d6e01ae08750152b" Oct 30 20:08:05.145191 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.145171 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-j6v5l-pmxpq" Oct 30 20:08:05.145339 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.145227 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-j6v5l-pmxpq" event={"ID":"a4af0759-e61a-48f6-821a-e1ad34261f11","Type":"ContainerDied","Data":"f8f5e6e5c59f943f241c9fd72105d3fdd1426ac1369a14c29d5653f53fc545af"} Oct 30 20:08:05.145339 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.145276 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8f5e6e5c59f943f241c9fd72105d3fdd1426ac1369a14c29d5653f53fc545af" Oct 30 20:08:05.147009 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.146986 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" event={"ID":"cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980","Type":"ContainerDied","Data":"3ea6babf52d5fdb46c7999a8d376ec03c02109bc5ba7036b0572eb937561cf39"} Oct 30 20:08:05.147148 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.147039 5088 scope.go:122] "RemoveContainer" containerID="af78a77d20af1212cd0cbe7378f61b37a49b69e6b21ab698149d0670432b8a81" Oct 30 20:08:05.147148 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.146996 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt" Oct 30 20:08:05.228059 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.228024 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt"] Oct 30 20:08:05.243584 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.243556 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-5vldt"] Oct 30 20:08:05.644111 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.643379 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" Oct 30 20:08:05.654294 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.654221 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" Oct 30 20:08:05.727159 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.698158 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/678518a9-8731-4f6f-a7a1-4b0e64070e92-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/678518a9-8731-4f6f-a7a1-4b0e64070e92-credential-socket\") pod \"678518a9-8731-4f6f-a7a1-4b0e64070e92\" (UID: \"678518a9-8731-4f6f-a7a1-4b0e64070e92\") " Oct 30 20:08:05.727159 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.698281 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/678518a9-8731-4f6f-a7a1-4b0e64070e92-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/678518a9-8731-4f6f-a7a1-4b0e64070e92-istio-podinfo\") pod \"678518a9-8731-4f6f-a7a1-4b0e64070e92\" (UID: \"678518a9-8731-4f6f-a7a1-4b0e64070e92\") " Oct 30 20:08:05.727159 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.698317 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/678518a9-8731-4f6f-a7a1-4b0e64070e92-istio-token\" (UniqueName: \"kubernetes.io/projected/678518a9-8731-4f6f-a7a1-4b0e64070e92-istio-token\") pod \"678518a9-8731-4f6f-a7a1-4b0e64070e92\" (UID: \"678518a9-8731-4f6f-a7a1-4b0e64070e92\") " Oct 30 20:08:05.727159 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.698347 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-credential-socket\") pod \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\" (UID: \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\") " Oct 30 20:08:05.727159 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.698412 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/678518a9-8731-4f6f-a7a1-4b0e64070e92-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/678518a9-8731-4f6f-a7a1-4b0e64070e92-istio-envoy\") pod \"678518a9-8731-4f6f-a7a1-4b0e64070e92\" (UID: \"678518a9-8731-4f6f-a7a1-4b0e64070e92\") " Oct 30 20:08:05.727159 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.698486 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-workload-socket\") pod \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\" (UID: \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\") " Oct 30 20:08:05.727159 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.698528 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/678518a9-8731-4f6f-a7a1-4b0e64070e92-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/678518a9-8731-4f6f-a7a1-4b0e64070e92-workload-socket\") pod \"678518a9-8731-4f6f-a7a1-4b0e64070e92\" (UID: \"678518a9-8731-4f6f-a7a1-4b0e64070e92\") " Oct 30 20:08:05.727159 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.698605 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-kube-api-access-zmt7j\" (UniqueName: \"kubernetes.io/projected/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-kube-api-access-zmt7j\") pod \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\" (UID: \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\") " Oct 30 20:08:05.727159 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.698653 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-istiod-ca-cert\") pod \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\" (UID: \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\") " Oct 30 20:08:05.727159 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.698718 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-istio-token\" (UniqueName: \"kubernetes.io/projected/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-istio-token\") pod \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\" (UID: \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\") " Oct 30 20:08:05.727159 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.719524 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-credential-socket" pod "4789c6d9-e328-4890-a2b3-a2fdbfdb04b9" (UID: "4789c6d9-e328-4890-a2b3-a2fdbfdb04b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:05.727159 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.719877 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/678518a9-8731-4f6f-a7a1-4b0e64070e92-credential-socket" pod "678518a9-8731-4f6f-a7a1-4b0e64070e92" (UID: "678518a9-8731-4f6f-a7a1-4b0e64070e92"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:05.727159 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.719923 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/678518a9-8731-4f6f-a7a1-4b0e64070e92-istio-podinfo" pod "678518a9-8731-4f6f-a7a1-4b0e64070e92" (UID: "678518a9-8731-4f6f-a7a1-4b0e64070e92"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:08:05.727159 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.720300 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-istiod-ca-cert" pod "4789c6d9-e328-4890-a2b3-a2fdbfdb04b9" (UID: "4789c6d9-e328-4890-a2b3-a2fdbfdb04b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:05.727159 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.720324 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-istio-data\") pod \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\" (UID: \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\") " Oct 30 20:08:05.727159 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.720370 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/678518a9-8731-4f6f-a7a1-4b0e64070e92-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/678518a9-8731-4f6f-a7a1-4b0e64070e92-istio-data\") pod \"678518a9-8731-4f6f-a7a1-4b0e64070e92\" (UID: \"678518a9-8731-4f6f-a7a1-4b0e64070e92\") " Oct 30 20:08:05.727159 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.720443 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/678518a9-8731-4f6f-a7a1-4b0e64070e92-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/678518a9-8731-4f6f-a7a1-4b0e64070e92-workload-certs\") pod \"678518a9-8731-4f6f-a7a1-4b0e64070e92\" (UID: \"678518a9-8731-4f6f-a7a1-4b0e64070e92\") " Oct 30 20:08:05.727159 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.720495 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-istio-podinfo\") pod \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\" (UID: \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\") " Oct 30 20:08:05.727159 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.720563 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-workload-certs\") pod \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\" (UID: \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\") " Oct 30 20:08:05.727159 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.720603 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/678518a9-8731-4f6f-a7a1-4b0e64070e92-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/678518a9-8731-4f6f-a7a1-4b0e64070e92-istiod-ca-cert\") pod \"678518a9-8731-4f6f-a7a1-4b0e64070e92\" (UID: \"678518a9-8731-4f6f-a7a1-4b0e64070e92\") " Oct 30 20:08:05.727159 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.720705 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/678518a9-8731-4f6f-a7a1-4b0e64070e92-kube-api-access-b9s8x\" (UniqueName: \"kubernetes.io/projected/678518a9-8731-4f6f-a7a1-4b0e64070e92-kube-api-access-b9s8x\") pod \"678518a9-8731-4f6f-a7a1-4b0e64070e92\" (UID: \"678518a9-8731-4f6f-a7a1-4b0e64070e92\") " Oct 30 20:08:05.727159 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.720760 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-istio-envoy\") pod \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\" (UID: \"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9\") " Oct 30 20:08:05.727159 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.721947 5088 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/678518a9-8731-4f6f-a7a1-4b0e64070e92-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:05.727159 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.721972 5088 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/678518a9-8731-4f6f-a7a1-4b0e64070e92-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:05.727159 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.721987 5088 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:05.727159 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.722001 5088 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:05.727159 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.722529 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-workload-socket" pod "4789c6d9-e328-4890-a2b3-a2fdbfdb04b9" (UID: "4789c6d9-e328-4890-a2b3-a2fdbfdb04b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:05.727159 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.724915 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/678518a9-8731-4f6f-a7a1-4b0e64070e92-istio-envoy" pod "678518a9-8731-4f6f-a7a1-4b0e64070e92" (UID: "678518a9-8731-4f6f-a7a1-4b0e64070e92"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:05.727159 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.725809 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/678518a9-8731-4f6f-a7a1-4b0e64070e92-istio-data" pod "678518a9-8731-4f6f-a7a1-4b0e64070e92" (UID: "678518a9-8731-4f6f-a7a1-4b0e64070e92"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:05.727159 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.725858 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/678518a9-8731-4f6f-a7a1-4b0e64070e92-workload-certs" pod "678518a9-8731-4f6f-a7a1-4b0e64070e92" (UID: "678518a9-8731-4f6f-a7a1-4b0e64070e92"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:05.727159 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.725904 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-workload-certs" pod "4789c6d9-e328-4890-a2b3-a2fdbfdb04b9" (UID: "4789c6d9-e328-4890-a2b3-a2fdbfdb04b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:05.727159 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.726036 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/678518a9-8731-4f6f-a7a1-4b0e64070e92-istiod-ca-cert" pod "678518a9-8731-4f6f-a7a1-4b0e64070e92" (UID: "678518a9-8731-4f6f-a7a1-4b0e64070e92"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:05.727159 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.726081 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-istio-data" pod "4789c6d9-e328-4890-a2b3-a2fdbfdb04b9" (UID: "4789c6d9-e328-4890-a2b3-a2fdbfdb04b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:05.737437 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.727217 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/678518a9-8731-4f6f-a7a1-4b0e64070e92-istio-token" pod "678518a9-8731-4f6f-a7a1-4b0e64070e92" (UID: "678518a9-8731-4f6f-a7a1-4b0e64070e92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:05.737437 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.730517 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/678518a9-8731-4f6f-a7a1-4b0e64070e92-workload-socket" pod "678518a9-8731-4f6f-a7a1-4b0e64070e92" (UID: "678518a9-8731-4f6f-a7a1-4b0e64070e92"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:05.739803 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.739770 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-istio-token" pod "4789c6d9-e328-4890-a2b3-a2fdbfdb04b9" (UID: "4789c6d9-e328-4890-a2b3-a2fdbfdb04b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:05.741476 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.741446 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-kube-api-access-zmt7j" pod "4789c6d9-e328-4890-a2b3-a2fdbfdb04b9" (UID: "4789c6d9-e328-4890-a2b3-a2fdbfdb04b9"). InnerVolumeSpecName "kube-api-access-zmt7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:05.742994 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.742957 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-istio-envoy" pod "4789c6d9-e328-4890-a2b3-a2fdbfdb04b9" (UID: "4789c6d9-e328-4890-a2b3-a2fdbfdb04b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:05.743780 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.743740 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-istio-podinfo" pod "4789c6d9-e328-4890-a2b3-a2fdbfdb04b9" (UID: "4789c6d9-e328-4890-a2b3-a2fdbfdb04b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:08:05.744039 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.744008 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/678518a9-8731-4f6f-a7a1-4b0e64070e92-kube-api-access-b9s8x" pod "678518a9-8731-4f6f-a7a1-4b0e64070e92" (UID: "678518a9-8731-4f6f-a7a1-4b0e64070e92"). InnerVolumeSpecName "kube-api-access-b9s8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:05.863873 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.839291 5088 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/678518a9-8731-4f6f-a7a1-4b0e64070e92-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:05.863873 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.839329 5088 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:05.863873 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.839341 5088 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:05.863873 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.839355 5088 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/678518a9-8731-4f6f-a7a1-4b0e64070e92-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:05.863873 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.839369 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9s8x\" (UniqueName: \"kubernetes.io/projected/678518a9-8731-4f6f-a7a1-4b0e64070e92-kube-api-access-b9s8x\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:05.863873 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.839393 5088 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:05.863873 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.839407 5088 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/678518a9-8731-4f6f-a7a1-4b0e64070e92-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:05.863873 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.839418 5088 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/678518a9-8731-4f6f-a7a1-4b0e64070e92-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:05.863873 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.839432 5088 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:05.863873 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.839454 5088 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/678518a9-8731-4f6f-a7a1-4b0e64070e92-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:05.863873 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.839468 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmt7j\" (UniqueName: \"kubernetes.io/projected/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-kube-api-access-zmt7j\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:05.863873 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.839484 5088 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:05.863873 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.839499 5088 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:05.863873 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:05.839511 5088 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/678518a9-8731-4f6f-a7a1-4b0e64070e92-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.008742 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.008181 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" Oct 30 20:08:06.110128 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.110102 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" Oct 30 20:08:06.119140 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.119112 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" Oct 30 20:08:06.156295 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.156256 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a255c4a5-c090-49d2-84ba-536dfb9b8a69-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a255c4a5-c090-49d2-84ba-536dfb9b8a69-istio-envoy\") pod \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\" (UID: \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\") " Oct 30 20:08:06.156696 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.156669 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a255c4a5-c090-49d2-84ba-536dfb9b8a69-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a255c4a5-c090-49d2-84ba-536dfb9b8a69-credential-socket\") pod \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\" (UID: \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\") " Oct 30 20:08:06.156890 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.156872 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a255c4a5-c090-49d2-84ba-536dfb9b8a69-kube-api-access-4zkdz\" (UniqueName: \"kubernetes.io/projected/a255c4a5-c090-49d2-84ba-536dfb9b8a69-kube-api-access-4zkdz\") pod \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\" (UID: \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\") " Oct 30 20:08:06.157070 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.157051 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a255c4a5-c090-49d2-84ba-536dfb9b8a69-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a255c4a5-c090-49d2-84ba-536dfb9b8a69-istiod-ca-cert\") pod \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\" (UID: \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\") " Oct 30 20:08:06.157229 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.157213 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a255c4a5-c090-49d2-84ba-536dfb9b8a69-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a255c4a5-c090-49d2-84ba-536dfb9b8a69-istio-data\") pod \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\" (UID: \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\") " Oct 30 20:08:06.157376 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.157357 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a255c4a5-c090-49d2-84ba-536dfb9b8a69-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a255c4a5-c090-49d2-84ba-536dfb9b8a69-workload-socket\") pod \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\" (UID: \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\") " Oct 30 20:08:06.157648 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.157628 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a255c4a5-c090-49d2-84ba-536dfb9b8a69-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a255c4a5-c090-49d2-84ba-536dfb9b8a69-workload-certs\") pod \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\" (UID: \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\") " Oct 30 20:08:06.157791 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.157775 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a255c4a5-c090-49d2-84ba-536dfb9b8a69-istio-token\" (UniqueName: \"kubernetes.io/projected/a255c4a5-c090-49d2-84ba-536dfb9b8a69-istio-token\") pod \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\" (UID: \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\") " Oct 30 20:08:06.158053 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.158014 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a255c4a5-c090-49d2-84ba-536dfb9b8a69-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a255c4a5-c090-49d2-84ba-536dfb9b8a69-istio-podinfo\") pod \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\" (UID: \"a255c4a5-c090-49d2-84ba-536dfb9b8a69\") " Oct 30 20:08:06.171235 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.171197 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a255c4a5-c090-49d2-84ba-536dfb9b8a69-istio-envoy" pod "a255c4a5-c090-49d2-84ba-536dfb9b8a69" (UID: "a255c4a5-c090-49d2-84ba-536dfb9b8a69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.171450 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.171428 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a255c4a5-c090-49d2-84ba-536dfb9b8a69-credential-socket" pod "a255c4a5-c090-49d2-84ba-536dfb9b8a69" (UID: "a255c4a5-c090-49d2-84ba-536dfb9b8a69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.173037 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.172961 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a255c4a5-c090-49d2-84ba-536dfb9b8a69-workload-certs" pod "a255c4a5-c090-49d2-84ba-536dfb9b8a69" (UID: "a255c4a5-c090-49d2-84ba-536dfb9b8a69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.173172 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.173153 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a255c4a5-c090-49d2-84ba-536dfb9b8a69-istio-data" pod "a255c4a5-c090-49d2-84ba-536dfb9b8a69" (UID: "a255c4a5-c090-49d2-84ba-536dfb9b8a69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.173305 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.173267 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a255c4a5-c090-49d2-84ba-536dfb9b8a69-workload-socket" pod "a255c4a5-c090-49d2-84ba-536dfb9b8a69" (UID: "a255c4a5-c090-49d2-84ba-536dfb9b8a69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.173668 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.173633 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a255c4a5-c090-49d2-84ba-536dfb9b8a69-istiod-ca-cert" pod "a255c4a5-c090-49d2-84ba-536dfb9b8a69" (UID: "a255c4a5-c090-49d2-84ba-536dfb9b8a69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:06.177963 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.177887 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a255c4a5-c090-49d2-84ba-536dfb9b8a69-kube-api-access-4zkdz" pod "a255c4a5-c090-49d2-84ba-536dfb9b8a69" (UID: "a255c4a5-c090-49d2-84ba-536dfb9b8a69"). InnerVolumeSpecName "kube-api-access-4zkdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:06.184432 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.184398 5088 generic.go:356] "Generic (PLEG): container finished" podID="05a46855-a143-4c6d-8c87-404dc271bdb3" containerID="e2852ce9442a037ded3c3a1a2d61a71a7261186d4e0bd39c8dbc326bacd5a9b7" exitCode=0 Oct 30 20:08:06.184554 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.184506 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" event={"ID":"05a46855-a143-4c6d-8c87-404dc271bdb3","Type":"ContainerDied","Data":"e2852ce9442a037ded3c3a1a2d61a71a7261186d4e0bd39c8dbc326bacd5a9b7"} Oct 30 20:08:06.184636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.184554 5088 scope.go:122] "RemoveContainer" containerID="ab404a6d0bb243a2362e993df58be2d5b55d6c8958bcc2f6e364ccc15da2c8f7" Oct 30 20:08:06.185600 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.185576 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a255c4a5-c090-49d2-84ba-536dfb9b8a69-istio-podinfo" pod "a255c4a5-c090-49d2-84ba-536dfb9b8a69" (UID: "a255c4a5-c090-49d2-84ba-536dfb9b8a69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:08:06.188234 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.188203 5088 generic.go:356] "Generic (PLEG): container finished" podID="0b36276d-bc5b-4415-8cf9-461aee572f3a" containerID="d53c6f364c7241cc108097bf75f911b385021a7733320a1ccf7a4a977d146d8d" exitCode=0 Oct 30 20:08:06.188359 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.188339 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" event={"ID":"0b36276d-bc5b-4415-8cf9-461aee572f3a","Type":"ContainerDied","Data":"d53c6f364c7241cc108097bf75f911b385021a7733320a1ccf7a4a977d146d8d"} Oct 30 20:08:06.191773 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.191341 5088 generic.go:356] "Generic (PLEG): container finished" podID="14b2ad56-7181-420c-8ad1-bdc00f25e928" containerID="4358d77f290648c02af2f9a40e5a7ed64a0a45b4e236f0c73a7c725314a93bd1" exitCode=0 Oct 30 20:08:06.191773 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.191488 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" Oct 30 20:08:06.191773 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.191692 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" event={"ID":"14b2ad56-7181-420c-8ad1-bdc00f25e928","Type":"ContainerDied","Data":"4358d77f290648c02af2f9a40e5a7ed64a0a45b4e236f0c73a7c725314a93bd1"} Oct 30 20:08:06.191773 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.191734 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp" event={"ID":"14b2ad56-7181-420c-8ad1-bdc00f25e928","Type":"ContainerDied","Data":"1993c6ed548f07135ac27b4feba9506113207bfe9eeb62258dddd5da744eb82e"} Oct 30 20:08:06.192419 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.192384 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a255c4a5-c090-49d2-84ba-536dfb9b8a69-istio-token" pod "a255c4a5-c090-49d2-84ba-536dfb9b8a69" (UID: "a255c4a5-c090-49d2-84ba-536dfb9b8a69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:06.193885 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.193854 5088 generic.go:356] "Generic (PLEG): container finished" podID="4789c6d9-e328-4890-a2b3-a2fdbfdb04b9" containerID="8df290432378123b70d4427acfe82432d938ea3a85aa1b705c55e635cde745c5" exitCode=0 Oct 30 20:08:06.193973 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.193938 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" event={"ID":"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9","Type":"ContainerDied","Data":"8df290432378123b70d4427acfe82432d938ea3a85aa1b705c55e635cde745c5"} Oct 30 20:08:06.194032 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.193980 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" event={"ID":"4789c6d9-e328-4890-a2b3-a2fdbfdb04b9","Type":"ContainerDied","Data":"04f356b1c755e58eef8f654d2c69f81d87eb02bab82174166836131fa77ad2d1"} Oct 30 20:08:06.194103 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.194052 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl" Oct 30 20:08:06.204473 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.204437 5088 generic.go:356] "Generic (PLEG): container finished" podID="a255c4a5-c090-49d2-84ba-536dfb9b8a69" containerID="500b9e0dbc558d86839f26b86c64e2b7e81db3e1fda3e37f9dcbb093e05578d4" exitCode=0 Oct 30 20:08:06.205289 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.205270 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" Oct 30 20:08:06.209893 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.204827 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" event={"ID":"a255c4a5-c090-49d2-84ba-536dfb9b8a69","Type":"ContainerDied","Data":"500b9e0dbc558d86839f26b86c64e2b7e81db3e1fda3e37f9dcbb093e05578d4"} Oct 30 20:08:06.210090 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.210055 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq" event={"ID":"a255c4a5-c090-49d2-84ba-536dfb9b8a69","Type":"ContainerDied","Data":"e4a1b171f325420d687994df96ba8900e2b68544df4e5579ff279092b25c1fda"} Oct 30 20:08:06.210819 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.210795 5088 generic.go:356] "Generic (PLEG): container finished" podID="f2757424-fb74-48ff-82c3-ccb79a27fd2a" containerID="15c727db7065a1567294b0f8b8556b22587ec02b41504624305b564a569fbb00" exitCode=0 Oct 30 20:08:06.210996 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.210978 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" event={"ID":"f2757424-fb74-48ff-82c3-ccb79a27fd2a","Type":"ContainerDied","Data":"15c727db7065a1567294b0f8b8556b22587ec02b41504624305b564a569fbb00"} Oct 30 20:08:06.215829 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.215805 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" Oct 30 20:08:06.218825 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.216218 5088 generic.go:356] "Generic (PLEG): container finished" podID="ab112498-5d12-44e0-85b3-0c9f5dc003d6" containerID="e68213262edc0b90dd827831d6b42294e4f6c2e96e19943a2dbf1a2894f0065b" exitCode=0 Oct 30 20:08:06.219136 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.216243 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" event={"ID":"ab112498-5d12-44e0-85b3-0c9f5dc003d6","Type":"ContainerDied","Data":"e68213262edc0b90dd827831d6b42294e4f6c2e96e19943a2dbf1a2894f0065b"} Oct 30 20:08:06.234087 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.233979 5088 scope.go:122] "RemoveContainer" containerID="4358d77f290648c02af2f9a40e5a7ed64a0a45b4e236f0c73a7c725314a93bd1" Oct 30 20:08:06.252777 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.251386 5088 generic.go:356] "Generic (PLEG): container finished" podID="f4b44c6e-891a-4073-9192-035aeaad10a4" containerID="3a6fcdf62cfa94bb23a47d8095c3f93e63a21035e5267a54f9439fb5482243d5" exitCode=0 Oct 30 20:08:06.252777 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.251522 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" event={"ID":"f4b44c6e-891a-4073-9192-035aeaad10a4","Type":"ContainerDied","Data":"3a6fcdf62cfa94bb23a47d8095c3f93e63a21035e5267a54f9439fb5482243d5"} Oct 30 20:08:06.252777 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.251576 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" event={"ID":"f4b44c6e-891a-4073-9192-035aeaad10a4","Type":"ContainerDied","Data":"20e7dfeb87f6609d1712e902592e8a9b0e68189ad91a712c72c7bad4ab52cc03"} Oct 30 20:08:06.252777 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.251826 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z" Oct 30 20:08:06.256800 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.256574 5088 generic.go:356] "Generic (PLEG): container finished" podID="b56fca64-1150-4198-945e-7e830300c11d" containerID="9eaf99979ceb984cf41aa039dda144ebbde0304a942ce7350ed64c04c8537d33" exitCode=0 Oct 30 20:08:06.256800 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.256717 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" event={"ID":"b56fca64-1150-4198-945e-7e830300c11d","Type":"ContainerDied","Data":"9eaf99979ceb984cf41aa039dda144ebbde0304a942ce7350ed64c04c8537d33"} Oct 30 20:08:06.259394 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.259361 5088 generic.go:356] "Generic (PLEG): container finished" podID="baca0e89-a102-4354-af54-c14ccc418923" containerID="945cf965ae8ee1f5738db7747631301dc305f411de087b51d4c826da927c53a1" exitCode=0 Oct 30 20:08:06.259607 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.259582 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" event={"ID":"baca0e89-a102-4354-af54-c14ccc418923","Type":"ContainerDied","Data":"945cf965ae8ee1f5738db7747631301dc305f411de087b51d4c826da927c53a1"} Oct 30 20:08:06.262116 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.260991 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b2ad56-7181-420c-8ad1-bdc00f25e928-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14b2ad56-7181-420c-8ad1-bdc00f25e928-istio-envoy\") pod \"14b2ad56-7181-420c-8ad1-bdc00f25e928\" (UID: \"14b2ad56-7181-420c-8ad1-bdc00f25e928\") " Oct 30 20:08:06.262209 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.262166 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b2ad56-7181-420c-8ad1-bdc00f25e928-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14b2ad56-7181-420c-8ad1-bdc00f25e928-workload-certs\") pod \"14b2ad56-7181-420c-8ad1-bdc00f25e928\" (UID: \"14b2ad56-7181-420c-8ad1-bdc00f25e928\") " Oct 30 20:08:06.262265 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.262215 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b2ad56-7181-420c-8ad1-bdc00f25e928-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14b2ad56-7181-420c-8ad1-bdc00f25e928-istio-data\") pod \"14b2ad56-7181-420c-8ad1-bdc00f25e928\" (UID: \"14b2ad56-7181-420c-8ad1-bdc00f25e928\") " Oct 30 20:08:06.262331 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.262281 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4b44c6e-891a-4073-9192-035aeaad10a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4b44c6e-891a-4073-9192-035aeaad10a4-istio-data\") pod \"f4b44c6e-891a-4073-9192-035aeaad10a4\" (UID: \"f4b44c6e-891a-4073-9192-035aeaad10a4\") " Oct 30 20:08:06.262331 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.262318 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14b2ad56-7181-420c-8ad1-bdc00f25e928-kube-api-access-qszrk\" (UniqueName: \"kubernetes.io/projected/14b2ad56-7181-420c-8ad1-bdc00f25e928-kube-api-access-qszrk\") pod \"14b2ad56-7181-420c-8ad1-bdc00f25e928\" (UID: \"14b2ad56-7181-420c-8ad1-bdc00f25e928\") " Oct 30 20:08:06.262434 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.262368 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14b2ad56-7181-420c-8ad1-bdc00f25e928-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14b2ad56-7181-420c-8ad1-bdc00f25e928-istiod-ca-cert\") pod \"14b2ad56-7181-420c-8ad1-bdc00f25e928\" (UID: \"14b2ad56-7181-420c-8ad1-bdc00f25e928\") " Oct 30 20:08:06.262434 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.262411 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4b44c6e-891a-4073-9192-035aeaad10a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4b44c6e-891a-4073-9192-035aeaad10a4-workload-certs\") pod \"f4b44c6e-891a-4073-9192-035aeaad10a4\" (UID: \"f4b44c6e-891a-4073-9192-035aeaad10a4\") " Oct 30 20:08:06.262533 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.262499 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4b44c6e-891a-4073-9192-035aeaad10a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4b44c6e-891a-4073-9192-035aeaad10a4-istio-podinfo\") pod \"f4b44c6e-891a-4073-9192-035aeaad10a4\" (UID: \"f4b44c6e-891a-4073-9192-035aeaad10a4\") " Oct 30 20:08:06.262795 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.262598 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14b2ad56-7181-420c-8ad1-bdc00f25e928-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14b2ad56-7181-420c-8ad1-bdc00f25e928-istio-podinfo\") pod \"14b2ad56-7181-420c-8ad1-bdc00f25e928\" (UID: \"14b2ad56-7181-420c-8ad1-bdc00f25e928\") " Oct 30 20:08:06.262795 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.262727 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4b44c6e-891a-4073-9192-035aeaad10a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4b44c6e-891a-4073-9192-035aeaad10a4-istiod-ca-cert\") pod \"f4b44c6e-891a-4073-9192-035aeaad10a4\" (UID: \"f4b44c6e-891a-4073-9192-035aeaad10a4\") " Oct 30 20:08:06.263678 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.263539 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b44c6e-891a-4073-9192-035aeaad10a4-workload-certs" pod "f4b44c6e-891a-4073-9192-035aeaad10a4" (UID: "f4b44c6e-891a-4073-9192-035aeaad10a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.264265 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.264060 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14b2ad56-7181-420c-8ad1-bdc00f25e928-istiod-ca-cert" pod "14b2ad56-7181-420c-8ad1-bdc00f25e928" (UID: "14b2ad56-7181-420c-8ad1-bdc00f25e928"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:06.264409 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.264390 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b2ad56-7181-420c-8ad1-bdc00f25e928-istio-data" pod "14b2ad56-7181-420c-8ad1-bdc00f25e928" (UID: "14b2ad56-7181-420c-8ad1-bdc00f25e928"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.264578 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.264551 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b44c6e-891a-4073-9192-035aeaad10a4-istio-data" pod "f4b44c6e-891a-4073-9192-035aeaad10a4" (UID: "f4b44c6e-891a-4073-9192-035aeaad10a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.264790 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.264770 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b2ad56-7181-420c-8ad1-bdc00f25e928-workload-certs" pod "14b2ad56-7181-420c-8ad1-bdc00f25e928" (UID: "14b2ad56-7181-420c-8ad1-bdc00f25e928"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.265231 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.265213 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b2ad56-7181-420c-8ad1-bdc00f25e928-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b2ad56-7181-420c-8ad1-bdc00f25e928-credential-socket\") pod \"14b2ad56-7181-420c-8ad1-bdc00f25e928\" (UID: \"14b2ad56-7181-420c-8ad1-bdc00f25e928\") " Oct 30 20:08:06.265431 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.265411 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b2ad56-7181-420c-8ad1-bdc00f25e928-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b2ad56-7181-420c-8ad1-bdc00f25e928-workload-socket\") pod \"14b2ad56-7181-420c-8ad1-bdc00f25e928\" (UID: \"14b2ad56-7181-420c-8ad1-bdc00f25e928\") " Oct 30 20:08:06.265554 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.265539 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4b44c6e-891a-4073-9192-035aeaad10a4-kube-api-access-2bhbp\" (UniqueName: \"kubernetes.io/projected/f4b44c6e-891a-4073-9192-035aeaad10a4-kube-api-access-2bhbp\") pod \"f4b44c6e-891a-4073-9192-035aeaad10a4\" (UID: \"f4b44c6e-891a-4073-9192-035aeaad10a4\") " Oct 30 20:08:06.265718 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.265701 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4b44c6e-891a-4073-9192-035aeaad10a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b44c6e-891a-4073-9192-035aeaad10a4-workload-socket\") pod \"f4b44c6e-891a-4073-9192-035aeaad10a4\" (UID: \"f4b44c6e-891a-4073-9192-035aeaad10a4\") " Oct 30 20:08:06.265874 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.265855 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4b44c6e-891a-4073-9192-035aeaad10a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4b44c6e-891a-4073-9192-035aeaad10a4-istio-envoy\") pod \"f4b44c6e-891a-4073-9192-035aeaad10a4\" (UID: \"f4b44c6e-891a-4073-9192-035aeaad10a4\") " Oct 30 20:08:06.266022 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.266006 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4b44c6e-891a-4073-9192-035aeaad10a4-istio-token\" (UniqueName: \"kubernetes.io/projected/f4b44c6e-891a-4073-9192-035aeaad10a4-istio-token\") pod \"f4b44c6e-891a-4073-9192-035aeaad10a4\" (UID: \"f4b44c6e-891a-4073-9192-035aeaad10a4\") " Oct 30 20:08:06.266164 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.266148 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4b44c6e-891a-4073-9192-035aeaad10a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b44c6e-891a-4073-9192-035aeaad10a4-credential-socket\") pod \"f4b44c6e-891a-4073-9192-035aeaad10a4\" (UID: \"f4b44c6e-891a-4073-9192-035aeaad10a4\") " Oct 30 20:08:06.266274 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.266260 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14b2ad56-7181-420c-8ad1-bdc00f25e928-istio-token\" (UniqueName: \"kubernetes.io/projected/14b2ad56-7181-420c-8ad1-bdc00f25e928-istio-token\") pod \"14b2ad56-7181-420c-8ad1-bdc00f25e928\" (UID: \"14b2ad56-7181-420c-8ad1-bdc00f25e928\") " Oct 30 20:08:06.266842 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.266811 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4b44c6e-891a-4073-9192-035aeaad10a4-istiod-ca-cert" pod "f4b44c6e-891a-4073-9192-035aeaad10a4" (UID: "f4b44c6e-891a-4073-9192-035aeaad10a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:06.267119 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.267099 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b2ad56-7181-420c-8ad1-bdc00f25e928-credential-socket" pod "14b2ad56-7181-420c-8ad1-bdc00f25e928" (UID: "14b2ad56-7181-420c-8ad1-bdc00f25e928"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.267345 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.267327 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b2ad56-7181-420c-8ad1-bdc00f25e928-workload-socket" pod "14b2ad56-7181-420c-8ad1-bdc00f25e928" (UID: "14b2ad56-7181-420c-8ad1-bdc00f25e928"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.267598 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.267572 5088 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a255c4a5-c090-49d2-84ba-536dfb9b8a69-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.267730 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.267716 5088 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a255c4a5-c090-49d2-84ba-536dfb9b8a69-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.267824 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.267812 5088 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4b44c6e-891a-4073-9192-035aeaad10a4-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.267905 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.267894 5088 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a255c4a5-c090-49d2-84ba-536dfb9b8a69-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.267989 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.267977 5088 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b2ad56-7181-420c-8ad1-bdc00f25e928-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.268441 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.268191 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14b2ad56-7181-420c-8ad1-bdc00f25e928-kube-api-access-qszrk" pod "14b2ad56-7181-420c-8ad1-bdc00f25e928" (UID: "14b2ad56-7181-420c-8ad1-bdc00f25e928"). InnerVolumeSpecName "kube-api-access-qszrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:06.268578 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.268560 5088 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b2ad56-7181-420c-8ad1-bdc00f25e928-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.269028 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.268942 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b44c6e-891a-4073-9192-035aeaad10a4-credential-socket" pod "f4b44c6e-891a-4073-9192-035aeaad10a4" (UID: "f4b44c6e-891a-4073-9192-035aeaad10a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.269138 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.269121 5088 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a255c4a5-c090-49d2-84ba-536dfb9b8a69-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.269231 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.269219 5088 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a255c4a5-c090-49d2-84ba-536dfb9b8a69-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.269336 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.269324 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zkdz\" (UniqueName: \"kubernetes.io/projected/a255c4a5-c090-49d2-84ba-536dfb9b8a69-kube-api-access-4zkdz\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.269416 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.269405 5088 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14b2ad56-7181-420c-8ad1-bdc00f25e928-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.269493 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.269484 5088 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14b2ad56-7181-420c-8ad1-bdc00f25e928-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.269576 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.269566 5088 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a255c4a5-c090-49d2-84ba-536dfb9b8a69-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.269672 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.269660 5088 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4b44c6e-891a-4073-9192-035aeaad10a4-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.269760 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.269750 5088 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a255c4a5-c090-49d2-84ba-536dfb9b8a69-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.269844 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.269833 5088 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14b2ad56-7181-420c-8ad1-bdc00f25e928-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.269920 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.269910 5088 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4b44c6e-891a-4073-9192-035aeaad10a4-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.270000 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.269989 5088 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a255c4a5-c090-49d2-84ba-536dfb9b8a69-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.270327 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.270308 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b44c6e-891a-4073-9192-035aeaad10a4-workload-socket" pod "f4b44c6e-891a-4073-9192-035aeaad10a4" (UID: "f4b44c6e-891a-4073-9192-035aeaad10a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.281855 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.277970 5088 generic.go:356] "Generic (PLEG): container finished" podID="678518a9-8731-4f6f-a7a1-4b0e64070e92" containerID="8e3731c9ad46f1c7da2c1fed08fddf94087b20095f37969b667fa8d70bfbcb89" exitCode=0 Oct 30 20:08:06.281855 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.278037 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" event={"ID":"678518a9-8731-4f6f-a7a1-4b0e64070e92","Type":"ContainerDied","Data":"8e3731c9ad46f1c7da2c1fed08fddf94087b20095f37969b667fa8d70bfbcb89"} Oct 30 20:08:06.281855 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.278082 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" event={"ID":"678518a9-8731-4f6f-a7a1-4b0e64070e92","Type":"ContainerDied","Data":"ab441d6fa6aef2eedee3d13c580b44456a212a40006513cc2229ca20a9b9ec1b"} Oct 30 20:08:06.281855 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.278159 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz" Oct 30 20:08:06.291691 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.285950 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b44c6e-891a-4073-9192-035aeaad10a4-istio-envoy" pod "f4b44c6e-891a-4073-9192-035aeaad10a4" (UID: "f4b44c6e-891a-4073-9192-035aeaad10a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.291691 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.286344 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4b44c6e-891a-4073-9192-035aeaad10a4-istio-token" pod "f4b44c6e-891a-4073-9192-035aeaad10a4" (UID: "f4b44c6e-891a-4073-9192-035aeaad10a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:06.300372 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.294815 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4b44c6e-891a-4073-9192-035aeaad10a4-istio-podinfo" pod "f4b44c6e-891a-4073-9192-035aeaad10a4" (UID: "f4b44c6e-891a-4073-9192-035aeaad10a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:08:06.300372 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.294940 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b2ad56-7181-420c-8ad1-bdc00f25e928-istio-envoy" pod "14b2ad56-7181-420c-8ad1-bdc00f25e928" (UID: "14b2ad56-7181-420c-8ad1-bdc00f25e928"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.300372 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.295799 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14b2ad56-7181-420c-8ad1-bdc00f25e928-istio-token" pod "14b2ad56-7181-420c-8ad1-bdc00f25e928" (UID: "14b2ad56-7181-420c-8ad1-bdc00f25e928"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:06.300372 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.297254 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14b2ad56-7181-420c-8ad1-bdc00f25e928-istio-podinfo" pod "14b2ad56-7181-420c-8ad1-bdc00f25e928" (UID: "14b2ad56-7181-420c-8ad1-bdc00f25e928"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:08:06.300372 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.297414 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4b44c6e-891a-4073-9192-035aeaad10a4-kube-api-access-2bhbp" pod "f4b44c6e-891a-4073-9192-035aeaad10a4" (UID: "f4b44c6e-891a-4073-9192-035aeaad10a4"). InnerVolumeSpecName "kube-api-access-2bhbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:06.304927 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.304895 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" Oct 30 20:08:06.306640 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.306605 5088 scope.go:122] "RemoveContainer" containerID="4358d77f290648c02af2f9a40e5a7ed64a0a45b4e236f0c73a7c725314a93bd1" Oct 30 20:08:06.306928 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:08:06.306906 5088 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4358d77f290648c02af2f9a40e5a7ed64a0a45b4e236f0c73a7c725314a93bd1\": container with ID starting with 4358d77f290648c02af2f9a40e5a7ed64a0a45b4e236f0c73a7c725314a93bd1 not found: ID does not exist" containerID="4358d77f290648c02af2f9a40e5a7ed64a0a45b4e236f0c73a7c725314a93bd1" Oct 30 20:08:06.307034 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.306934 5088 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4358d77f290648c02af2f9a40e5a7ed64a0a45b4e236f0c73a7c725314a93bd1"} err="failed to get container status \"4358d77f290648c02af2f9a40e5a7ed64a0a45b4e236f0c73a7c725314a93bd1\": rpc error: code = NotFound desc = could not find container \"4358d77f290648c02af2f9a40e5a7ed64a0a45b4e236f0c73a7c725314a93bd1\": container with ID starting with 4358d77f290648c02af2f9a40e5a7ed64a0a45b4e236f0c73a7c725314a93bd1 not found: ID does not exist" Oct 30 20:08:06.307034 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.306951 5088 scope.go:122] "RemoveContainer" containerID="8df290432378123b70d4427acfe82432d938ea3a85aa1b705c55e635cde745c5" Oct 30 20:08:06.328043 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.328013 5088 scope.go:122] "RemoveContainer" containerID="8df290432378123b70d4427acfe82432d938ea3a85aa1b705c55e635cde745c5" Oct 30 20:08:06.330767 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:08:06.330739 5088 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8df290432378123b70d4427acfe82432d938ea3a85aa1b705c55e635cde745c5\": container with ID starting with 8df290432378123b70d4427acfe82432d938ea3a85aa1b705c55e635cde745c5 not found: ID does not exist" containerID="8df290432378123b70d4427acfe82432d938ea3a85aa1b705c55e635cde745c5" Oct 30 20:08:06.330881 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.330776 5088 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8df290432378123b70d4427acfe82432d938ea3a85aa1b705c55e635cde745c5"} err="failed to get container status \"8df290432378123b70d4427acfe82432d938ea3a85aa1b705c55e635cde745c5\": rpc error: code = NotFound desc = could not find container \"8df290432378123b70d4427acfe82432d938ea3a85aa1b705c55e635cde745c5\": container with ID starting with 8df290432378123b70d4427acfe82432d938ea3a85aa1b705c55e635cde745c5 not found: ID does not exist" Oct 30 20:08:06.330881 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.330800 5088 scope.go:122] "RemoveContainer" containerID="500b9e0dbc558d86839f26b86c64e2b7e81db3e1fda3e37f9dcbb093e05578d4" Oct 30 20:08:06.333352 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.333330 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl"] Oct 30 20:08:06.335779 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.335652 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" Oct 30 20:08:06.339815 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.339791 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-bncnl"] Oct 30 20:08:06.343765 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.343744 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" Oct 30 20:08:06.349927 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.349828 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" Oct 30 20:08:06.350122 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.350069 5088 scope.go:122] "RemoveContainer" containerID="500b9e0dbc558d86839f26b86c64e2b7e81db3e1fda3e37f9dcbb093e05578d4" Oct 30 20:08:06.350376 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:08:06.350351 5088 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"500b9e0dbc558d86839f26b86c64e2b7e81db3e1fda3e37f9dcbb093e05578d4\": container with ID starting with 500b9e0dbc558d86839f26b86c64e2b7e81db3e1fda3e37f9dcbb093e05578d4 not found: ID does not exist" containerID="500b9e0dbc558d86839f26b86c64e2b7e81db3e1fda3e37f9dcbb093e05578d4" Oct 30 20:08:06.350465 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.350380 5088 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"500b9e0dbc558d86839f26b86c64e2b7e81db3e1fda3e37f9dcbb093e05578d4"} err="failed to get container status \"500b9e0dbc558d86839f26b86c64e2b7e81db3e1fda3e37f9dcbb093e05578d4\": rpc error: code = NotFound desc = could not find container \"500b9e0dbc558d86839f26b86c64e2b7e81db3e1fda3e37f9dcbb093e05578d4\": container with ID starting with 500b9e0dbc558d86839f26b86c64e2b7e81db3e1fda3e37f9dcbb093e05578d4 not found: ID does not exist" Oct 30 20:08:06.350465 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.350397 5088 scope.go:122] "RemoveContainer" containerID="3a6fcdf62cfa94bb23a47d8095c3f93e63a21035e5267a54f9439fb5482243d5" Oct 30 20:08:06.367380 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.367352 5088 scope.go:122] "RemoveContainer" containerID="592772e60f2ac707fee713a5705108969e381b16166a0f80580a0819513cc1df" Oct 30 20:08:06.371275 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.371248 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2757424-fb74-48ff-82c3-ccb79a27fd2a-istio-token\" (UniqueName: \"kubernetes.io/projected/f2757424-fb74-48ff-82c3-ccb79a27fd2a-istio-token\") pod \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\" (UID: \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\") " Oct 30 20:08:06.371783 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.371749 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2757424-fb74-48ff-82c3-ccb79a27fd2a-istio-data" pod "f2757424-fb74-48ff-82c3-ccb79a27fd2a" (UID: "f2757424-fb74-48ff-82c3-ccb79a27fd2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.372942 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.372913 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2757424-fb74-48ff-82c3-ccb79a27fd2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2757424-fb74-48ff-82c3-ccb79a27fd2a-istio-data\") pod \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\" (UID: \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\") " Oct 30 20:08:06.373052 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.372987 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2757424-fb74-48ff-82c3-ccb79a27fd2a-kube-api-access-czgtn\" (UniqueName: \"kubernetes.io/projected/f2757424-fb74-48ff-82c3-ccb79a27fd2a-kube-api-access-czgtn\") pod \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\" (UID: \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\") " Oct 30 20:08:06.373052 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.373019 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2757424-fb74-48ff-82c3-ccb79a27fd2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2757424-fb74-48ff-82c3-ccb79a27fd2a-workload-certs\") pod \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\" (UID: \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\") " Oct 30 20:08:06.373237 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.373178 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2757424-fb74-48ff-82c3-ccb79a27fd2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2757424-fb74-48ff-82c3-ccb79a27fd2a-istio-envoy\") pod \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\" (UID: \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\") " Oct 30 20:08:06.373237 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.373230 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2757424-fb74-48ff-82c3-ccb79a27fd2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2757424-fb74-48ff-82c3-ccb79a27fd2a-workload-socket\") pod \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\" (UID: \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\") " Oct 30 20:08:06.373351 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.373315 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2757424-fb74-48ff-82c3-ccb79a27fd2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2757424-fb74-48ff-82c3-ccb79a27fd2a-istiod-ca-cert\") pod \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\" (UID: \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\") " Oct 30 20:08:06.373351 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.373340 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2757424-fb74-48ff-82c3-ccb79a27fd2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2757424-fb74-48ff-82c3-ccb79a27fd2a-istio-podinfo\") pod \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\" (UID: \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\") " Oct 30 20:08:06.373456 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.373384 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2757424-fb74-48ff-82c3-ccb79a27fd2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2757424-fb74-48ff-82c3-ccb79a27fd2a-credential-socket\") pod \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\" (UID: \"f2757424-fb74-48ff-82c3-ccb79a27fd2a\") " Oct 30 20:08:06.374149 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.373907 5088 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2757424-fb74-48ff-82c3-ccb79a27fd2a-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.374149 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.373929 5088 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4b44c6e-891a-4073-9192-035aeaad10a4-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.374149 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.373944 5088 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14b2ad56-7181-420c-8ad1-bdc00f25e928-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.374149 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.373955 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bhbp\" (UniqueName: \"kubernetes.io/projected/f4b44c6e-891a-4073-9192-035aeaad10a4-kube-api-access-2bhbp\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.374149 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.373965 5088 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b44c6e-891a-4073-9192-035aeaad10a4-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.374149 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.373974 5088 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4b44c6e-891a-4073-9192-035aeaad10a4-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.374149 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.373982 5088 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4b44c6e-891a-4073-9192-035aeaad10a4-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.374149 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.373989 5088 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b44c6e-891a-4073-9192-035aeaad10a4-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.374149 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.373999 5088 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14b2ad56-7181-420c-8ad1-bdc00f25e928-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.374149 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.374008 5088 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14b2ad56-7181-420c-8ad1-bdc00f25e928-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.374149 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.374016 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qszrk\" (UniqueName: \"kubernetes.io/projected/14b2ad56-7181-420c-8ad1-bdc00f25e928-kube-api-access-qszrk\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.374656 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.374536 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2757424-fb74-48ff-82c3-ccb79a27fd2a-credential-socket" pod "f2757424-fb74-48ff-82c3-ccb79a27fd2a" (UID: "f2757424-fb74-48ff-82c3-ccb79a27fd2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.374972 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.374824 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2757424-fb74-48ff-82c3-ccb79a27fd2a-workload-socket" pod "f2757424-fb74-48ff-82c3-ccb79a27fd2a" (UID: "f2757424-fb74-48ff-82c3-ccb79a27fd2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.375430 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.375093 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2757424-fb74-48ff-82c3-ccb79a27fd2a-istio-token" pod "f2757424-fb74-48ff-82c3-ccb79a27fd2a" (UID: "f2757424-fb74-48ff-82c3-ccb79a27fd2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:06.375430 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.375178 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2757424-fb74-48ff-82c3-ccb79a27fd2a-istiod-ca-cert" pod "f2757424-fb74-48ff-82c3-ccb79a27fd2a" (UID: "f2757424-fb74-48ff-82c3-ccb79a27fd2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:06.375785 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.375769 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2757424-fb74-48ff-82c3-ccb79a27fd2a-workload-certs" pod "f2757424-fb74-48ff-82c3-ccb79a27fd2a" (UID: "f2757424-fb74-48ff-82c3-ccb79a27fd2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.376959 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.376925 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2757424-fb74-48ff-82c3-ccb79a27fd2a-istio-envoy" pod "f2757424-fb74-48ff-82c3-ccb79a27fd2a" (UID: "f2757424-fb74-48ff-82c3-ccb79a27fd2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.378852 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.378830 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2757424-fb74-48ff-82c3-ccb79a27fd2a-kube-api-access-czgtn" pod "f2757424-fb74-48ff-82c3-ccb79a27fd2a" (UID: "f2757424-fb74-48ff-82c3-ccb79a27fd2a"). InnerVolumeSpecName "kube-api-access-czgtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:06.379076 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.379052 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2757424-fb74-48ff-82c3-ccb79a27fd2a-istio-podinfo" pod "f2757424-fb74-48ff-82c3-ccb79a27fd2a" (UID: "f2757424-fb74-48ff-82c3-ccb79a27fd2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:08:06.384421 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.384399 5088 scope.go:122] "RemoveContainer" containerID="3a6fcdf62cfa94bb23a47d8095c3f93e63a21035e5267a54f9439fb5482243d5" Oct 30 20:08:06.384885 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:08:06.384856 5088 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a6fcdf62cfa94bb23a47d8095c3f93e63a21035e5267a54f9439fb5482243d5\": container with ID starting with 3a6fcdf62cfa94bb23a47d8095c3f93e63a21035e5267a54f9439fb5482243d5 not found: ID does not exist" containerID="3a6fcdf62cfa94bb23a47d8095c3f93e63a21035e5267a54f9439fb5482243d5" Oct 30 20:08:06.384988 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.384915 5088 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a6fcdf62cfa94bb23a47d8095c3f93e63a21035e5267a54f9439fb5482243d5"} err="failed to get container status \"3a6fcdf62cfa94bb23a47d8095c3f93e63a21035e5267a54f9439fb5482243d5\": rpc error: code = NotFound desc = could not find container \"3a6fcdf62cfa94bb23a47d8095c3f93e63a21035e5267a54f9439fb5482243d5\": container with ID starting with 3a6fcdf62cfa94bb23a47d8095c3f93e63a21035e5267a54f9439fb5482243d5 not found: ID does not exist" Oct 30 20:08:06.384988 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.384941 5088 scope.go:122] "RemoveContainer" containerID="592772e60f2ac707fee713a5705108969e381b16166a0f80580a0819513cc1df" Oct 30 20:08:06.385293 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:08:06.385273 5088 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"592772e60f2ac707fee713a5705108969e381b16166a0f80580a0819513cc1df\": container with ID starting with 592772e60f2ac707fee713a5705108969e381b16166a0f80580a0819513cc1df not found: ID does not exist" containerID="592772e60f2ac707fee713a5705108969e381b16166a0f80580a0819513cc1df" Oct 30 20:08:06.385372 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.385296 5088 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"592772e60f2ac707fee713a5705108969e381b16166a0f80580a0819513cc1df"} err="failed to get container status \"592772e60f2ac707fee713a5705108969e381b16166a0f80580a0819513cc1df\": rpc error: code = NotFound desc = could not find container \"592772e60f2ac707fee713a5705108969e381b16166a0f80580a0819513cc1df\": container with ID starting with 592772e60f2ac707fee713a5705108969e381b16166a0f80580a0819513cc1df not found: ID does not exist" Oct 30 20:08:06.385372 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.385311 5088 scope.go:122] "RemoveContainer" containerID="8e3731c9ad46f1c7da2c1fed08fddf94087b20095f37969b667fa8d70bfbcb89" Oct 30 20:08:06.399163 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.399135 5088 scope.go:122] "RemoveContainer" containerID="8e3731c9ad46f1c7da2c1fed08fddf94087b20095f37969b667fa8d70bfbcb89" Oct 30 20:08:06.399490 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:08:06.399462 5088 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e3731c9ad46f1c7da2c1fed08fddf94087b20095f37969b667fa8d70bfbcb89\": container with ID starting with 8e3731c9ad46f1c7da2c1fed08fddf94087b20095f37969b667fa8d70bfbcb89 not found: ID does not exist" containerID="8e3731c9ad46f1c7da2c1fed08fddf94087b20095f37969b667fa8d70bfbcb89" Oct 30 20:08:06.399594 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.399504 5088 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e3731c9ad46f1c7da2c1fed08fddf94087b20095f37969b667fa8d70bfbcb89"} err="failed to get container status \"8e3731c9ad46f1c7da2c1fed08fddf94087b20095f37969b667fa8d70bfbcb89\": rpc error: code = NotFound desc = could not find container \"8e3731c9ad46f1c7da2c1fed08fddf94087b20095f37969b667fa8d70bfbcb89\": container with ID starting with 8e3731c9ad46f1c7da2c1fed08fddf94087b20095f37969b667fa8d70bfbcb89 not found: ID does not exist" Oct 30 20:08:06.471964 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.471874 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq"] Oct 30 20:08:06.474923 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.474865 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b36276d-bc5b-4415-8cf9-461aee572f3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b36276d-bc5b-4415-8cf9-461aee572f3a-workload-socket\") pod \"0b36276d-bc5b-4415-8cf9-461aee572f3a\" (UID: \"0b36276d-bc5b-4415-8cf9-461aee572f3a\") " Oct 30 20:08:06.474923 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.474912 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b56fca64-1150-4198-945e-7e830300c11d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56fca64-1150-4198-945e-7e830300c11d-workload-certs\") pod \"b56fca64-1150-4198-945e-7e830300c11d\" (UID: \"b56fca64-1150-4198-945e-7e830300c11d\") " Oct 30 20:08:06.475138 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.474961 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab112498-5d12-44e0-85b3-0c9f5dc003d6-kube-api-access-8688v\" (UniqueName: \"kubernetes.io/projected/ab112498-5d12-44e0-85b3-0c9f5dc003d6-kube-api-access-8688v\") pod \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\" (UID: \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\") " Oct 30 20:08:06.475138 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.475000 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a46855-a143-4c6d-8c87-404dc271bdb3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05a46855-a143-4c6d-8c87-404dc271bdb3-workload-certs\") pod \"05a46855-a143-4c6d-8c87-404dc271bdb3\" (UID: \"05a46855-a143-4c6d-8c87-404dc271bdb3\") " Oct 30 20:08:06.475138 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.475031 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b56fca64-1150-4198-945e-7e830300c11d-istio-token\" (UniqueName: \"kubernetes.io/projected/b56fca64-1150-4198-945e-7e830300c11d-istio-token\") pod \"b56fca64-1150-4198-945e-7e830300c11d\" (UID: \"b56fca64-1150-4198-945e-7e830300c11d\") " Oct 30 20:08:06.475138 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.475061 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05a46855-a143-4c6d-8c87-404dc271bdb3-istio-token\" (UniqueName: \"kubernetes.io/projected/05a46855-a143-4c6d-8c87-404dc271bdb3-istio-token\") pod \"05a46855-a143-4c6d-8c87-404dc271bdb3\" (UID: \"05a46855-a143-4c6d-8c87-404dc271bdb3\") " Oct 30 20:08:06.475138 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.475104 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b36276d-bc5b-4415-8cf9-461aee572f3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b36276d-bc5b-4415-8cf9-461aee572f3a-istio-envoy\") pod \"0b36276d-bc5b-4415-8cf9-461aee572f3a\" (UID: \"0b36276d-bc5b-4415-8cf9-461aee572f3a\") " Oct 30 20:08:06.475138 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.475130 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b36276d-bc5b-4415-8cf9-461aee572f3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b36276d-bc5b-4415-8cf9-461aee572f3a-credential-socket\") pod \"0b36276d-bc5b-4415-8cf9-461aee572f3a\" (UID: \"0b36276d-bc5b-4415-8cf9-461aee572f3a\") " Oct 30 20:08:06.475376 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.475177 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b36276d-bc5b-4415-8cf9-461aee572f3a-workload-socket" pod "0b36276d-bc5b-4415-8cf9-461aee572f3a" (UID: "0b36276d-bc5b-4415-8cf9-461aee572f3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.475376 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.475186 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b36276d-bc5b-4415-8cf9-461aee572f3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b36276d-bc5b-4415-8cf9-461aee572f3a-workload-certs\") pod \"0b36276d-bc5b-4415-8cf9-461aee572f3a\" (UID: \"0b36276d-bc5b-4415-8cf9-461aee572f3a\") " Oct 30 20:08:06.475376 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.475260 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b36276d-bc5b-4415-8cf9-461aee572f3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b36276d-bc5b-4415-8cf9-461aee572f3a-istio-podinfo\") pod \"0b36276d-bc5b-4415-8cf9-461aee572f3a\" (UID: \"0b36276d-bc5b-4415-8cf9-461aee572f3a\") " Oct 30 20:08:06.475376 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.475330 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b56fca64-1150-4198-945e-7e830300c11d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56fca64-1150-4198-945e-7e830300c11d-istiod-ca-cert\") pod \"b56fca64-1150-4198-945e-7e830300c11d\" (UID: \"b56fca64-1150-4198-945e-7e830300c11d\") " Oct 30 20:08:06.475376 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.475361 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b56fca64-1150-4198-945e-7e830300c11d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56fca64-1150-4198-945e-7e830300c11d-istio-data\") pod \"b56fca64-1150-4198-945e-7e830300c11d\" (UID: \"b56fca64-1150-4198-945e-7e830300c11d\") " Oct 30 20:08:06.475521 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.475413 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b56fca64-1150-4198-945e-7e830300c11d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56fca64-1150-4198-945e-7e830300c11d-credential-socket\") pod \"b56fca64-1150-4198-945e-7e830300c11d\" (UID: \"b56fca64-1150-4198-945e-7e830300c11d\") " Oct 30 20:08:06.475521 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.475451 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b36276d-bc5b-4415-8cf9-461aee572f3a-workload-certs" pod "0b36276d-bc5b-4415-8cf9-461aee572f3a" (UID: "0b36276d-bc5b-4415-8cf9-461aee572f3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.475521 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.475469 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab112498-5d12-44e0-85b3-0c9f5dc003d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab112498-5d12-44e0-85b3-0c9f5dc003d6-credential-socket\") pod \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\" (UID: \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\") " Oct 30 20:08:06.475521 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.475501 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab112498-5d12-44e0-85b3-0c9f5dc003d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab112498-5d12-44e0-85b3-0c9f5dc003d6-istiod-ca-cert\") pod \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\" (UID: \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\") " Oct 30 20:08:06.475687 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.475547 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab112498-5d12-44e0-85b3-0c9f5dc003d6-istio-token\" (UniqueName: \"kubernetes.io/projected/ab112498-5d12-44e0-85b3-0c9f5dc003d6-istio-token\") pod \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\" (UID: \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\") " Oct 30 20:08:06.475687 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.475628 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a46855-a143-4c6d-8c87-404dc271bdb3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05a46855-a143-4c6d-8c87-404dc271bdb3-istio-envoy\") pod \"05a46855-a143-4c6d-8c87-404dc271bdb3\" (UID: \"05a46855-a143-4c6d-8c87-404dc271bdb3\") " Oct 30 20:08:06.475687 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.475664 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a46855-a143-4c6d-8c87-404dc271bdb3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a46855-a143-4c6d-8c87-404dc271bdb3-workload-socket\") pod \"05a46855-a143-4c6d-8c87-404dc271bdb3\" (UID: \"05a46855-a143-4c6d-8c87-404dc271bdb3\") " Oct 30 20:08:06.475775 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.475701 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b36276d-bc5b-4415-8cf9-461aee572f3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b36276d-bc5b-4415-8cf9-461aee572f3a-istiod-ca-cert\") pod \"0b36276d-bc5b-4415-8cf9-461aee572f3a\" (UID: \"0b36276d-bc5b-4415-8cf9-461aee572f3a\") " Oct 30 20:08:06.475775 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.475707 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56fca64-1150-4198-945e-7e830300c11d-workload-certs" pod "b56fca64-1150-4198-945e-7e830300c11d" (UID: "b56fca64-1150-4198-945e-7e830300c11d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.475775 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.475746 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab112498-5d12-44e0-85b3-0c9f5dc003d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab112498-5d12-44e0-85b3-0c9f5dc003d6-istio-envoy\") pod \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\" (UID: \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\") " Oct 30 20:08:06.475869 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.475812 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b56fca64-1150-4198-945e-7e830300c11d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56fca64-1150-4198-945e-7e830300c11d-workload-socket\") pod \"b56fca64-1150-4198-945e-7e830300c11d\" (UID: \"b56fca64-1150-4198-945e-7e830300c11d\") " Oct 30 20:08:06.475900 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.475868 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab112498-5d12-44e0-85b3-0c9f5dc003d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab112498-5d12-44e0-85b3-0c9f5dc003d6-workload-socket\") pod \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\" (UID: \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\") " Oct 30 20:08:06.475942 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.475907 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b36276d-bc5b-4415-8cf9-461aee572f3a-kube-api-access-d8csc\" (UniqueName: \"kubernetes.io/projected/0b36276d-bc5b-4415-8cf9-461aee572f3a-kube-api-access-d8csc\") pod \"0b36276d-bc5b-4415-8cf9-461aee572f3a\" (UID: \"0b36276d-bc5b-4415-8cf9-461aee572f3a\") " Oct 30 20:08:06.476003 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.475960 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05a46855-a143-4c6d-8c87-404dc271bdb3-kube-api-access-jr2gs\" (UniqueName: \"kubernetes.io/projected/05a46855-a143-4c6d-8c87-404dc271bdb3-kube-api-access-jr2gs\") pod \"05a46855-a143-4c6d-8c87-404dc271bdb3\" (UID: \"05a46855-a143-4c6d-8c87-404dc271bdb3\") " Oct 30 20:08:06.476003 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.475994 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b56fca64-1150-4198-945e-7e830300c11d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56fca64-1150-4198-945e-7e830300c11d-istio-envoy\") pod \"b56fca64-1150-4198-945e-7e830300c11d\" (UID: \"b56fca64-1150-4198-945e-7e830300c11d\") " Oct 30 20:08:06.476103 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.476024 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab112498-5d12-44e0-85b3-0c9f5dc003d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab112498-5d12-44e0-85b3-0c9f5dc003d6-istio-data\") pod \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\" (UID: \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\") " Oct 30 20:08:06.476103 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.476067 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab112498-5d12-44e0-85b3-0c9f5dc003d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab112498-5d12-44e0-85b3-0c9f5dc003d6-workload-certs\") pod \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\" (UID: \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\") " Oct 30 20:08:06.476202 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.476136 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b56fca64-1150-4198-945e-7e830300c11d-kube-api-access-wmqdl\" (UniqueName: \"kubernetes.io/projected/b56fca64-1150-4198-945e-7e830300c11d-kube-api-access-wmqdl\") pod \"b56fca64-1150-4198-945e-7e830300c11d\" (UID: \"b56fca64-1150-4198-945e-7e830300c11d\") " Oct 30 20:08:06.476202 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.476170 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b36276d-bc5b-4415-8cf9-461aee572f3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b36276d-bc5b-4415-8cf9-461aee572f3a-istio-data\") pod \"0b36276d-bc5b-4415-8cf9-461aee572f3a\" (UID: \"0b36276d-bc5b-4415-8cf9-461aee572f3a\") " Oct 30 20:08:06.476294 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.476234 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b56fca64-1150-4198-945e-7e830300c11d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56fca64-1150-4198-945e-7e830300c11d-istio-podinfo\") pod \"b56fca64-1150-4198-945e-7e830300c11d\" (UID: \"b56fca64-1150-4198-945e-7e830300c11d\") " Oct 30 20:08:06.476327 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.476299 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b36276d-bc5b-4415-8cf9-461aee572f3a-istio-token\" (UniqueName: \"kubernetes.io/projected/0b36276d-bc5b-4415-8cf9-461aee572f3a-istio-token\") pod \"0b36276d-bc5b-4415-8cf9-461aee572f3a\" (UID: \"0b36276d-bc5b-4415-8cf9-461aee572f3a\") " Oct 30 20:08:06.476364 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.476345 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a46855-a143-4c6d-8c87-404dc271bdb3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a46855-a143-4c6d-8c87-404dc271bdb3-credential-socket\") pod \"05a46855-a143-4c6d-8c87-404dc271bdb3\" (UID: \"05a46855-a143-4c6d-8c87-404dc271bdb3\") " Oct 30 20:08:06.476398 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.476379 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05a46855-a143-4c6d-8c87-404dc271bdb3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05a46855-a143-4c6d-8c87-404dc271bdb3-istiod-ca-cert\") pod \"05a46855-a143-4c6d-8c87-404dc271bdb3\" (UID: \"05a46855-a143-4c6d-8c87-404dc271bdb3\") " Oct 30 20:08:06.476432 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.476411 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a46855-a143-4c6d-8c87-404dc271bdb3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05a46855-a143-4c6d-8c87-404dc271bdb3-istio-data\") pod \"05a46855-a143-4c6d-8c87-404dc271bdb3\" (UID: \"05a46855-a143-4c6d-8c87-404dc271bdb3\") " Oct 30 20:08:06.476480 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.476463 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05a46855-a143-4c6d-8c87-404dc271bdb3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05a46855-a143-4c6d-8c87-404dc271bdb3-istio-podinfo\") pod \"05a46855-a143-4c6d-8c87-404dc271bdb3\" (UID: \"05a46855-a143-4c6d-8c87-404dc271bdb3\") " Oct 30 20:08:06.476512 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.476502 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab112498-5d12-44e0-85b3-0c9f5dc003d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab112498-5d12-44e0-85b3-0c9f5dc003d6-istio-podinfo\") pod \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\" (UID: \"ab112498-5d12-44e0-85b3-0c9f5dc003d6\") " Oct 30 20:08:06.477471 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.477449 5088 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2757424-fb74-48ff-82c3-ccb79a27fd2a-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.477540 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.477477 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czgtn\" (UniqueName: \"kubernetes.io/projected/f2757424-fb74-48ff-82c3-ccb79a27fd2a-kube-api-access-czgtn\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.477540 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.477493 5088 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2757424-fb74-48ff-82c3-ccb79a27fd2a-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.477540 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.477507 5088 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2757424-fb74-48ff-82c3-ccb79a27fd2a-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.477540 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.477521 5088 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2757424-fb74-48ff-82c3-ccb79a27fd2a-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.477540 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.477535 5088 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2757424-fb74-48ff-82c3-ccb79a27fd2a-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.477734 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.477549 5088 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2757424-fb74-48ff-82c3-ccb79a27fd2a-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.477734 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.477562 5088 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b36276d-bc5b-4415-8cf9-461aee572f3a-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.477734 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.477576 5088 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56fca64-1150-4198-945e-7e830300c11d-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.477734 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.477591 5088 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2757424-fb74-48ff-82c3-ccb79a27fd2a-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.477734 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.477604 5088 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b36276d-bc5b-4415-8cf9-461aee572f3a-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.478236 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.478209 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b36276d-bc5b-4415-8cf9-461aee572f3a-credential-socket" pod "0b36276d-bc5b-4415-8cf9-461aee572f3a" (UID: "0b36276d-bc5b-4415-8cf9-461aee572f3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.478473 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.478384 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a46855-a143-4c6d-8c87-404dc271bdb3-workload-certs" pod "05a46855-a143-4c6d-8c87-404dc271bdb3" (UID: "05a46855-a143-4c6d-8c87-404dc271bdb3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.478587 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.478443 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab112498-5d12-44e0-85b3-0c9f5dc003d6-workload-socket" pod "ab112498-5d12-44e0-85b3-0c9f5dc003d6" (UID: "ab112498-5d12-44e0-85b3-0c9f5dc003d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.478664 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.478648 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab112498-5d12-44e0-85b3-0c9f5dc003d6-istiod-ca-cert" pod "ab112498-5d12-44e0-85b3-0c9f5dc003d6" (UID: "ab112498-5d12-44e0-85b3-0c9f5dc003d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:06.478907 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.478878 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56fca64-1150-4198-945e-7e830300c11d-istio-data" pod "b56fca64-1150-4198-945e-7e830300c11d" (UID: "b56fca64-1150-4198-945e-7e830300c11d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.478907 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.478892 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56fca64-1150-4198-945e-7e830300c11d-credential-socket" pod "b56fca64-1150-4198-945e-7e830300c11d" (UID: "b56fca64-1150-4198-945e-7e830300c11d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.479099 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.479079 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab112498-5d12-44e0-85b3-0c9f5dc003d6-credential-socket" pod "ab112498-5d12-44e0-85b3-0c9f5dc003d6" (UID: "ab112498-5d12-44e0-85b3-0c9f5dc003d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.479300 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.479274 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-4prxq"] Oct 30 20:08:06.479450 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.479361 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b56fca64-1150-4198-945e-7e830300c11d-istiod-ca-cert" pod "b56fca64-1150-4198-945e-7e830300c11d" (UID: "b56fca64-1150-4198-945e-7e830300c11d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:06.479636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.479599 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b36276d-bc5b-4415-8cf9-461aee572f3a-istio-data" pod "0b36276d-bc5b-4415-8cf9-461aee572f3a" (UID: "0b36276d-bc5b-4415-8cf9-461aee572f3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.479914 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.479888 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a46855-a143-4c6d-8c87-404dc271bdb3-credential-socket" pod "05a46855-a143-4c6d-8c87-404dc271bdb3" (UID: "05a46855-a143-4c6d-8c87-404dc271bdb3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.480183 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.480158 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab112498-5d12-44e0-85b3-0c9f5dc003d6-istio-podinfo" pod "ab112498-5d12-44e0-85b3-0c9f5dc003d6" (UID: "ab112498-5d12-44e0-85b3-0c9f5dc003d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:08:06.480532 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.480503 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05a46855-a143-4c6d-8c87-404dc271bdb3-istiod-ca-cert" pod "05a46855-a143-4c6d-8c87-404dc271bdb3" (UID: "05a46855-a143-4c6d-8c87-404dc271bdb3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:06.480855 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.480829 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a46855-a143-4c6d-8c87-404dc271bdb3-istio-data" pod "05a46855-a143-4c6d-8c87-404dc271bdb3" (UID: "05a46855-a143-4c6d-8c87-404dc271bdb3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.480947 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.480846 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab112498-5d12-44e0-85b3-0c9f5dc003d6-istio-token" pod "ab112498-5d12-44e0-85b3-0c9f5dc003d6" (UID: "ab112498-5d12-44e0-85b3-0c9f5dc003d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:06.480947 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.480872 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b56fca64-1150-4198-945e-7e830300c11d-istio-token" pod "b56fca64-1150-4198-945e-7e830300c11d" (UID: "b56fca64-1150-4198-945e-7e830300c11d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:06.481271 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.481246 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b36276d-bc5b-4415-8cf9-461aee572f3a-istio-podinfo" pod "0b36276d-bc5b-4415-8cf9-461aee572f3a" (UID: "0b36276d-bc5b-4415-8cf9-461aee572f3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:08:06.481496 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.481477 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab112498-5d12-44e0-85b3-0c9f5dc003d6-istio-data" pod "ab112498-5d12-44e0-85b3-0c9f5dc003d6" (UID: "ab112498-5d12-44e0-85b3-0c9f5dc003d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.481938 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.481915 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b36276d-bc5b-4415-8cf9-461aee572f3a-istio-envoy" pod "0b36276d-bc5b-4415-8cf9-461aee572f3a" (UID: "0b36276d-bc5b-4415-8cf9-461aee572f3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.482037 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.481992 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab112498-5d12-44e0-85b3-0c9f5dc003d6-kube-api-access-8688v" pod "ab112498-5d12-44e0-85b3-0c9f5dc003d6" (UID: "ab112498-5d12-44e0-85b3-0c9f5dc003d6"). InnerVolumeSpecName "kube-api-access-8688v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:06.482333 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.482312 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b36276d-bc5b-4415-8cf9-461aee572f3a-istiod-ca-cert" pod "0b36276d-bc5b-4415-8cf9-461aee572f3a" (UID: "0b36276d-bc5b-4415-8cf9-461aee572f3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:06.482771 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.482520 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b56fca64-1150-4198-945e-7e830300c11d-istio-podinfo" pod "b56fca64-1150-4198-945e-7e830300c11d" (UID: "b56fca64-1150-4198-945e-7e830300c11d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:08:06.482771 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.482548 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a46855-a143-4c6d-8c87-404dc271bdb3-workload-socket" pod "05a46855-a143-4c6d-8c87-404dc271bdb3" (UID: "05a46855-a143-4c6d-8c87-404dc271bdb3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.483201 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.482959 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab112498-5d12-44e0-85b3-0c9f5dc003d6-workload-certs" pod "ab112498-5d12-44e0-85b3-0c9f5dc003d6" (UID: "ab112498-5d12-44e0-85b3-0c9f5dc003d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.483201 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.483016 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b36276d-bc5b-4415-8cf9-461aee572f3a-istio-token" pod "0b36276d-bc5b-4415-8cf9-461aee572f3a" (UID: "0b36276d-bc5b-4415-8cf9-461aee572f3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:06.483524 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.483207 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a46855-a143-4c6d-8c87-404dc271bdb3-istio-envoy" pod "05a46855-a143-4c6d-8c87-404dc271bdb3" (UID: "05a46855-a143-4c6d-8c87-404dc271bdb3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.483524 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.483245 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56fca64-1150-4198-945e-7e830300c11d-workload-socket" pod "b56fca64-1150-4198-945e-7e830300c11d" (UID: "b56fca64-1150-4198-945e-7e830300c11d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.483683 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.483578 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05a46855-a143-4c6d-8c87-404dc271bdb3-istio-token" pod "05a46855-a143-4c6d-8c87-404dc271bdb3" (UID: "05a46855-a143-4c6d-8c87-404dc271bdb3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:06.484412 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.484365 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b36276d-bc5b-4415-8cf9-461aee572f3a-kube-api-access-d8csc" pod "0b36276d-bc5b-4415-8cf9-461aee572f3a" (UID: "0b36276d-bc5b-4415-8cf9-461aee572f3a"). InnerVolumeSpecName "kube-api-access-d8csc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:06.484569 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.484534 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05a46855-a143-4c6d-8c87-404dc271bdb3-istio-podinfo" pod "05a46855-a143-4c6d-8c87-404dc271bdb3" (UID: "05a46855-a143-4c6d-8c87-404dc271bdb3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:08:06.484835 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.484808 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab112498-5d12-44e0-85b3-0c9f5dc003d6-istio-envoy" pod "ab112498-5d12-44e0-85b3-0c9f5dc003d6" (UID: "ab112498-5d12-44e0-85b3-0c9f5dc003d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.484933 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.484867 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05a46855-a143-4c6d-8c87-404dc271bdb3-kube-api-access-jr2gs" pod "05a46855-a143-4c6d-8c87-404dc271bdb3" (UID: "05a46855-a143-4c6d-8c87-404dc271bdb3"). InnerVolumeSpecName "kube-api-access-jr2gs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:06.485143 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.485126 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56fca64-1150-4198-945e-7e830300c11d-istio-envoy" pod "b56fca64-1150-4198-945e-7e830300c11d" (UID: "b56fca64-1150-4198-945e-7e830300c11d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.485557 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.485535 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b56fca64-1150-4198-945e-7e830300c11d-kube-api-access-wmqdl" pod "b56fca64-1150-4198-945e-7e830300c11d" (UID: "b56fca64-1150-4198-945e-7e830300c11d"). InnerVolumeSpecName "kube-api-access-wmqdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:06.507486 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.507466 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" Oct 30 20:08:06.578908 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.578882 5088 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab112498-5d12-44e0-85b3-0c9f5dc003d6-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.578908 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.578907 5088 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab112498-5d12-44e0-85b3-0c9f5dc003d6-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.578908 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.578916 5088 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05a46855-a143-4c6d-8c87-404dc271bdb3-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.579111 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.578923 5088 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a46855-a143-4c6d-8c87-404dc271bdb3-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.579111 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.578932 5088 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b36276d-bc5b-4415-8cf9-461aee572f3a-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.579111 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.578942 5088 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab112498-5d12-44e0-85b3-0c9f5dc003d6-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.579111 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.578950 5088 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56fca64-1150-4198-945e-7e830300c11d-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.579111 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.578957 5088 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab112498-5d12-44e0-85b3-0c9f5dc003d6-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.579111 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.578966 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8csc\" (UniqueName: \"kubernetes.io/projected/0b36276d-bc5b-4415-8cf9-461aee572f3a-kube-api-access-d8csc\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.579111 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.578976 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr2gs\" (UniqueName: \"kubernetes.io/projected/05a46855-a143-4c6d-8c87-404dc271bdb3-kube-api-access-jr2gs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.579111 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.578984 5088 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56fca64-1150-4198-945e-7e830300c11d-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.579111 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.578992 5088 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab112498-5d12-44e0-85b3-0c9f5dc003d6-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.579111 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.579000 5088 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab112498-5d12-44e0-85b3-0c9f5dc003d6-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.579111 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.579020 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmqdl\" (UniqueName: \"kubernetes.io/projected/b56fca64-1150-4198-945e-7e830300c11d-kube-api-access-wmqdl\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.579111 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.579030 5088 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b36276d-bc5b-4415-8cf9-461aee572f3a-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.579111 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.579039 5088 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56fca64-1150-4198-945e-7e830300c11d-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.579111 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.579049 5088 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b36276d-bc5b-4415-8cf9-461aee572f3a-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.579111 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.579057 5088 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a46855-a143-4c6d-8c87-404dc271bdb3-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.579111 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.579065 5088 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05a46855-a143-4c6d-8c87-404dc271bdb3-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.579111 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.579073 5088 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05a46855-a143-4c6d-8c87-404dc271bdb3-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.579111 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.579080 5088 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05a46855-a143-4c6d-8c87-404dc271bdb3-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.579111 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.579089 5088 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab112498-5d12-44e0-85b3-0c9f5dc003d6-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.579111 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.579097 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8688v\" (UniqueName: \"kubernetes.io/projected/ab112498-5d12-44e0-85b3-0c9f5dc003d6-kube-api-access-8688v\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.579111 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.579107 5088 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05a46855-a143-4c6d-8c87-404dc271bdb3-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.579111 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.579115 5088 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b56fca64-1150-4198-945e-7e830300c11d-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.579111 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.579122 5088 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05a46855-a143-4c6d-8c87-404dc271bdb3-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.579850 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.579130 5088 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b36276d-bc5b-4415-8cf9-461aee572f3a-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.579850 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.579137 5088 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b36276d-bc5b-4415-8cf9-461aee572f3a-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.579850 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.579145 5088 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b36276d-bc5b-4415-8cf9-461aee572f3a-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.579850 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.579152 5088 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56fca64-1150-4198-945e-7e830300c11d-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.579850 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.579162 5088 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56fca64-1150-4198-945e-7e830300c11d-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.579850 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.579169 5088 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56fca64-1150-4198-945e-7e830300c11d-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.579850 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.579177 5088 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab112498-5d12-44e0-85b3-0c9f5dc003d6-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.647267 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.647219 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz"] Oct 30 20:08:06.655149 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.655118 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-w4krz"] Oct 30 20:08:06.679849 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.679814 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/baca0e89-a102-4354-af54-c14ccc418923-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baca0e89-a102-4354-af54-c14ccc418923-istio-podinfo\") pod \"baca0e89-a102-4354-af54-c14ccc418923\" (UID: \"baca0e89-a102-4354-af54-c14ccc418923\") " Oct 30 20:08:06.679998 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.679949 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baca0e89-a102-4354-af54-c14ccc418923-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baca0e89-a102-4354-af54-c14ccc418923-istiod-ca-cert\") pod \"baca0e89-a102-4354-af54-c14ccc418923\" (UID: \"baca0e89-a102-4354-af54-c14ccc418923\") " Oct 30 20:08:06.680054 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.680005 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baca0e89-a102-4354-af54-c14ccc418923-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baca0e89-a102-4354-af54-c14ccc418923-istio-envoy\") pod \"baca0e89-a102-4354-af54-c14ccc418923\" (UID: \"baca0e89-a102-4354-af54-c14ccc418923\") " Oct 30 20:08:06.680054 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.680041 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baca0e89-a102-4354-af54-c14ccc418923-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baca0e89-a102-4354-af54-c14ccc418923-workload-socket\") pod \"baca0e89-a102-4354-af54-c14ccc418923\" (UID: \"baca0e89-a102-4354-af54-c14ccc418923\") " Oct 30 20:08:06.680176 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.680094 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baca0e89-a102-4354-af54-c14ccc418923-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baca0e89-a102-4354-af54-c14ccc418923-workload-certs\") pod \"baca0e89-a102-4354-af54-c14ccc418923\" (UID: \"baca0e89-a102-4354-af54-c14ccc418923\") " Oct 30 20:08:06.680176 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.680141 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baca0e89-a102-4354-af54-c14ccc418923-kube-api-access-nrf7k\" (UniqueName: \"kubernetes.io/projected/baca0e89-a102-4354-af54-c14ccc418923-kube-api-access-nrf7k\") pod \"baca0e89-a102-4354-af54-c14ccc418923\" (UID: \"baca0e89-a102-4354-af54-c14ccc418923\") " Oct 30 20:08:06.680270 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.680223 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baca0e89-a102-4354-af54-c14ccc418923-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baca0e89-a102-4354-af54-c14ccc418923-credential-socket\") pod \"baca0e89-a102-4354-af54-c14ccc418923\" (UID: \"baca0e89-a102-4354-af54-c14ccc418923\") " Oct 30 20:08:06.680270 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.680249 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baca0e89-a102-4354-af54-c14ccc418923-istio-token\" (UniqueName: \"kubernetes.io/projected/baca0e89-a102-4354-af54-c14ccc418923-istio-token\") pod \"baca0e89-a102-4354-af54-c14ccc418923\" (UID: \"baca0e89-a102-4354-af54-c14ccc418923\") " Oct 30 20:08:06.680385 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.680338 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baca0e89-a102-4354-af54-c14ccc418923-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baca0e89-a102-4354-af54-c14ccc418923-istio-data\") pod \"baca0e89-a102-4354-af54-c14ccc418923\" (UID: \"baca0e89-a102-4354-af54-c14ccc418923\") " Oct 30 20:08:06.680445 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.680371 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baca0e89-a102-4354-af54-c14ccc418923-workload-socket" pod "baca0e89-a102-4354-af54-c14ccc418923" (UID: "baca0e89-a102-4354-af54-c14ccc418923"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.680445 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.680390 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baca0e89-a102-4354-af54-c14ccc418923-workload-certs" pod "baca0e89-a102-4354-af54-c14ccc418923" (UID: "baca0e89-a102-4354-af54-c14ccc418923"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.680560 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.680445 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baca0e89-a102-4354-af54-c14ccc418923-istiod-ca-cert" pod "baca0e89-a102-4354-af54-c14ccc418923" (UID: "baca0e89-a102-4354-af54-c14ccc418923"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:06.680664 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.680640 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baca0e89-a102-4354-af54-c14ccc418923-istio-data" pod "baca0e89-a102-4354-af54-c14ccc418923" (UID: "baca0e89-a102-4354-af54-c14ccc418923"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.680834 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.680672 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baca0e89-a102-4354-af54-c14ccc418923-credential-socket" pod "baca0e89-a102-4354-af54-c14ccc418923" (UID: "baca0e89-a102-4354-af54-c14ccc418923"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.681260 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.681235 5088 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baca0e89-a102-4354-af54-c14ccc418923-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.681336 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.681266 5088 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baca0e89-a102-4354-af54-c14ccc418923-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.681336 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.681281 5088 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baca0e89-a102-4354-af54-c14ccc418923-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.681336 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.681295 5088 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baca0e89-a102-4354-af54-c14ccc418923-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.681336 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.681309 5088 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baca0e89-a102-4354-af54-c14ccc418923-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.682403 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.682381 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/baca0e89-a102-4354-af54-c14ccc418923-istio-podinfo" pod "baca0e89-a102-4354-af54-c14ccc418923" (UID: "baca0e89-a102-4354-af54-c14ccc418923"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:08:06.682472 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.682402 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baca0e89-a102-4354-af54-c14ccc418923-istio-token" pod "baca0e89-a102-4354-af54-c14ccc418923" (UID: "baca0e89-a102-4354-af54-c14ccc418923"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:06.682595 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.682581 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baca0e89-a102-4354-af54-c14ccc418923-istio-envoy" pod "baca0e89-a102-4354-af54-c14ccc418923" (UID: "baca0e89-a102-4354-af54-c14ccc418923"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:06.682749 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.682733 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baca0e89-a102-4354-af54-c14ccc418923-kube-api-access-nrf7k" pod "baca0e89-a102-4354-af54-c14ccc418923" (UID: "baca0e89-a102-4354-af54-c14ccc418923"). InnerVolumeSpecName "kube-api-access-nrf7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:06.744754 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.744667 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4789c6d9-e328-4890-a2b3-a2fdbfdb04b9" path="/var/lib/kubelet/pods/4789c6d9-e328-4890-a2b3-a2fdbfdb04b9/volumes" Oct 30 20:08:06.745555 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.745528 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="678518a9-8731-4f6f-a7a1-4b0e64070e92" path="/var/lib/kubelet/pods/678518a9-8731-4f6f-a7a1-4b0e64070e92/volumes" Oct 30 20:08:06.746092 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.746074 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a255c4a5-c090-49d2-84ba-536dfb9b8a69" path="/var/lib/kubelet/pods/a255c4a5-c090-49d2-84ba-536dfb9b8a69/volumes" Oct 30 20:08:06.746928 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.746901 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980" path="/var/lib/kubelet/pods/cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980/volumes" Oct 30 20:08:06.748329 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.748309 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z"] Oct 30 20:08:06.758708 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.758686 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-2cs4z"] Oct 30 20:08:06.783103 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.783073 5088 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baca0e89-a102-4354-af54-c14ccc418923-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.783193 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.783107 5088 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baca0e89-a102-4354-af54-c14ccc418923-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.783193 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.783121 5088 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baca0e89-a102-4354-af54-c14ccc418923-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.783193 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.783137 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrf7k\" (UniqueName: \"kubernetes.io/projected/baca0e89-a102-4354-af54-c14ccc418923-kube-api-access-nrf7k\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:06.784651 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.784591 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp"] Oct 30 20:08:06.789820 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:06.789799 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-jsknp"] Oct 30 20:08:07.287913 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:07.287876 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" event={"ID":"0b36276d-bc5b-4415-8cf9-461aee572f3a","Type":"ContainerDied","Data":"e9b6e0bfaa6bd0ba4363efce496ebd01f6506c5a6c2b8f1a1de9f5b36106c990"} Oct 30 20:08:07.288366 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:07.287947 5088 scope.go:122] "RemoveContainer" containerID="d53c6f364c7241cc108097bf75f911b385021a7733320a1ccf7a4a977d146d8d" Oct 30 20:08:07.288366 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:07.287887 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk" Oct 30 20:08:07.293078 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:07.293056 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" Oct 30 20:08:07.293215 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:07.293058 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7" event={"ID":"f2757424-fb74-48ff-82c3-ccb79a27fd2a","Type":"ContainerDied","Data":"2827f0489f7560b8277adb716f5e5bb96d0ab022bf592552859071d64fd97b02"} Oct 30 20:08:07.294693 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:07.294667 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" event={"ID":"ab112498-5d12-44e0-85b3-0c9f5dc003d6","Type":"ContainerDied","Data":"4c1ce0dee5c3b0e7b2d4a265a2f55533135b4b148f6ae3808f6aa98729c11006"} Oct 30 20:08:07.294844 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:07.294678 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8" Oct 30 20:08:07.298510 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:07.298489 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" event={"ID":"b56fca64-1150-4198-945e-7e830300c11d","Type":"ContainerDied","Data":"241e1db8b27b2fc77a301bbac6d896975f8f1324b718a69b6736dcea185f8e19"} Oct 30 20:08:07.298655 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:07.298500 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m" Oct 30 20:08:07.300053 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:07.300033 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" Oct 30 20:08:07.300176 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:07.300050 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw" event={"ID":"baca0e89-a102-4354-af54-c14ccc418923","Type":"ContainerDied","Data":"f525517430be53ab0dd22245442935ae843a8330231eee8fabda7a6331ac36a7"} Oct 30 20:08:07.301893 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:07.301879 5088 scope.go:122] "RemoveContainer" containerID="15c727db7065a1567294b0f8b8556b22587ec02b41504624305b564a569fbb00" Oct 30 20:08:07.303232 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:07.303211 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" event={"ID":"05a46855-a143-4c6d-8c87-404dc271bdb3","Type":"ContainerDied","Data":"e98b120b8cfb5b7d9eb2682c24467d32b571678a51792f76b426412a5284c49f"} Oct 30 20:08:07.303343 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:07.303325 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s" Oct 30 20:08:07.312762 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:07.312744 5088 scope.go:122] "RemoveContainer" containerID="e68213262edc0b90dd827831d6b42294e4f6c2e96e19943a2dbf1a2894f0065b" Oct 30 20:08:07.323479 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:07.323464 5088 scope.go:122] "RemoveContainer" containerID="9eaf99979ceb984cf41aa039dda144ebbde0304a942ce7350ed64c04c8537d33" Oct 30 20:08:07.334909 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:07.334887 5088 scope.go:122] "RemoveContainer" containerID="945cf965ae8ee1f5738db7747631301dc305f411de087b51d4c826da927c53a1" Oct 30 20:08:07.345113 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:07.345088 5088 scope.go:122] "RemoveContainer" containerID="e2852ce9442a037ded3c3a1a2d61a71a7261186d4e0bd39c8dbc326bacd5a9b7" Oct 30 20:08:07.357980 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:07.357958 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s"] Oct 30 20:08:07.371251 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:07.371224 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-b5c6s"] Oct 30 20:08:07.402506 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:07.402479 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk"] Oct 30 20:08:07.410924 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:07.410901 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-jrxvk"] Oct 30 20:08:07.456103 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:07.456076 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw"] Oct 30 20:08:07.460481 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:07.460454 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-nsngw"] Oct 30 20:08:07.501867 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:07.501833 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7"] Oct 30 20:08:07.508999 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:07.508966 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f1f016-openshift-default-6c85499cc-d6tz7"] Oct 30 20:08:07.550707 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:07.550624 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8"] Oct 30 20:08:07.561472 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:07.561439 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-626870-openshift-default-7cd649d59c-blmn8"] Oct 30 20:08:07.637353 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:07.637318 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m"] Oct 30 20:08:07.675717 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:07.675174 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-912f24-openshift-default-5f56ff5f6c-wv49m"] Oct 30 20:08:08.466502 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:08.466466 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f10d2-58d98b4775-jq5gd" podUID="24f9b32b-8908-4adf-bda3-5cfc8cf34bc5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:08:08.466989 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:08.466564 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-62f4a1-645b9d8589-k6rx9" podUID="933b9fd2-86f8-4f62-a3e5-48636487d9b0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:08:08.740236 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:08.740201 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05a46855-a143-4c6d-8c87-404dc271bdb3" path="/var/lib/kubelet/pods/05a46855-a143-4c6d-8c87-404dc271bdb3/volumes" Oct 30 20:08:08.740755 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:08.740740 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b36276d-bc5b-4415-8cf9-461aee572f3a" path="/var/lib/kubelet/pods/0b36276d-bc5b-4415-8cf9-461aee572f3a/volumes" Oct 30 20:08:08.741195 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:08.741181 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14b2ad56-7181-420c-8ad1-bdc00f25e928" path="/var/lib/kubelet/pods/14b2ad56-7181-420c-8ad1-bdc00f25e928/volumes" Oct 30 20:08:08.741938 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:08.741923 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab112498-5d12-44e0-85b3-0c9f5dc003d6" path="/var/lib/kubelet/pods/ab112498-5d12-44e0-85b3-0c9f5dc003d6/volumes" Oct 30 20:08:08.742355 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:08.742343 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b56fca64-1150-4198-945e-7e830300c11d" path="/var/lib/kubelet/pods/b56fca64-1150-4198-945e-7e830300c11d/volumes" Oct 30 20:08:08.742778 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:08.742766 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baca0e89-a102-4354-af54-c14ccc418923" path="/var/lib/kubelet/pods/baca0e89-a102-4354-af54-c14ccc418923/volumes" Oct 30 20:08:08.743502 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:08.743480 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2757424-fb74-48ff-82c3-ccb79a27fd2a" path="/var/lib/kubelet/pods/f2757424-fb74-48ff-82c3-ccb79a27fd2a/volumes" Oct 30 20:08:08.743925 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:08.743912 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4b44c6e-891a-4073-9192-035aeaad10a4" path="/var/lib/kubelet/pods/f4b44c6e-891a-4073-9192-035aeaad10a4/volumes" Oct 30 20:08:08.772971 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:08.772936 5088 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","besteffort","pod3d117d60-d768-4d67-bb48-cd0522fe2a57"] err="unable to destroy cgroup paths for cgroup [kubepods besteffort pod3d117d60-d768-4d67-bb48-cd0522fe2a57] : Timed out while waiting for systemd to remove kubepods-besteffort-pod3d117d60_d768_4d67_bb48_cd0522fe2a57.slice" Oct 30 20:08:08.773132 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:08:08.772973 5088 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to delete cgroup paths for [kubepods besteffort pod3d117d60-d768-4d67-bb48-cd0522fe2a57] : unable to destroy cgroup paths for cgroup [kubepods besteffort pod3d117d60-d768-4d67-bb48-cd0522fe2a57] : Timed out while waiting for systemd to remove kubepods-besteffort-pod3d117d60_d768_4d67_bb48_cd0522fe2a57.slice" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" Oct 30 20:08:09.327688 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:09.327655 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj" Oct 30 20:08:09.388999 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:09.388964 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj"] Oct 30 20:08:09.402747 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:09.402711 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-626870/simulator-w5-626870-kserve-79f7696b7-gz9wj"] Oct 30 20:08:10.742972 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:10.742932 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d117d60-d768-4d67-bb48-cd0522fe2a57" path="/var/lib/kubelet/pods/3d117d60-d768-4d67-bb48-cd0522fe2a57/volumes" Oct 30 20:08:11.429577 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:11.428865 5088 generic.go:356] "Generic (PLEG): container finished" podID="24f9b32b-8908-4adf-bda3-5cfc8cf34bc5" containerID="489b0a7cc51dbb214bc5e596a066ac15f465bf372ee1bc62015e49f46dc7ae03" exitCode=0 Oct 30 20:08:11.429577 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:11.429058 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f10d2-58d98b4775-jq5gd" event={"ID":"24f9b32b-8908-4adf-bda3-5cfc8cf34bc5","Type":"ContainerDied","Data":"489b0a7cc51dbb214bc5e596a066ac15f465bf372ee1bc62015e49f46dc7ae03"} Oct 30 20:08:11.562018 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:11.561991 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f10d2-58d98b4775-jq5gd" Oct 30 20:08:11.615199 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:11.615168 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24f9b32b-8908-4adf-bda3-5cfc8cf34bc5-kube-api-access-rcv6d\" (UniqueName: \"kubernetes.io/projected/24f9b32b-8908-4adf-bda3-5cfc8cf34bc5-kube-api-access-rcv6d\") pod \"24f9b32b-8908-4adf-bda3-5cfc8cf34bc5\" (UID: \"24f9b32b-8908-4adf-bda3-5cfc8cf34bc5\") " Oct 30 20:08:11.616944 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:11.616915 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24f9b32b-8908-4adf-bda3-5cfc8cf34bc5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24f9b32b-8908-4adf-bda3-5cfc8cf34bc5-plugins-config-volume\") pod \"24f9b32b-8908-4adf-bda3-5cfc8cf34bc5\" (UID: \"24f9b32b-8908-4adf-bda3-5cfc8cf34bc5\") " Oct 30 20:08:11.618122 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:11.618092 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24f9b32b-8908-4adf-bda3-5cfc8cf34bc5-plugins-config-volume" pod "24f9b32b-8908-4adf-bda3-5cfc8cf34bc5" (UID: "24f9b32b-8908-4adf-bda3-5cfc8cf34bc5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:11.624551 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:11.624520 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24f9b32b-8908-4adf-bda3-5cfc8cf34bc5-kube-api-access-rcv6d" pod "24f9b32b-8908-4adf-bda3-5cfc8cf34bc5" (UID: "24f9b32b-8908-4adf-bda3-5cfc8cf34bc5"). InnerVolumeSpecName "kube-api-access-rcv6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:11.825092 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:11.807085 5088 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24f9b32b-8908-4adf-bda3-5cfc8cf34bc5-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:11.826986 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:11.826946 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcv6d\" (UniqueName: \"kubernetes.io/projected/24f9b32b-8908-4adf-bda3-5cfc8cf34bc5-kube-api-access-rcv6d\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:11.904446 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:11.904375 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-62f4a1-4vztc"] Oct 30 20:08:11.913493 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:11.913409 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-62f4a1-4vztc"] Oct 30 20:08:12.443916 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:12.443890 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f10d2-58d98b4775-jq5gd" Oct 30 20:08:12.443916 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:12.443899 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f10d2-58d98b4775-jq5gd" event={"ID":"24f9b32b-8908-4adf-bda3-5cfc8cf34bc5","Type":"ContainerDied","Data":"55df9f8943fcffc304a9b09fc65184eb43b4ec213e479a6728601437e115c92c"} Oct 30 20:08:12.444152 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:12.443962 5088 scope.go:122] "RemoveContainer" containerID="489b0a7cc51dbb214bc5e596a066ac15f465bf372ee1bc62015e49f46dc7ae03" Oct 30 20:08:12.447494 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:12.447470 5088 generic.go:356] "Generic (PLEG): container finished" podID="933b9fd2-86f8-4f62-a3e5-48636487d9b0" containerID="133ce1c35e98bbdfce32864378f02f84461a01e8d0bd8ae7160251bde03e9ece" exitCode=0 Oct 30 20:08:12.447606 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:12.447549 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-62f4a1-645b9d8589-k6rx9" event={"ID":"933b9fd2-86f8-4f62-a3e5-48636487d9b0","Type":"ContainerDied","Data":"133ce1c35e98bbdfce32864378f02f84461a01e8d0bd8ae7160251bde03e9ece"} Oct 30 20:08:12.534493 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:12.534466 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5f10d2-58d98b4775-jq5gd"] Oct 30 20:08:12.541579 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:12.541554 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5f10d2-58d98b4775-jq5gd"] Oct 30 20:08:12.564434 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:12.564398 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-62f4a1-645b9d8589-k6rx9" Oct 30 20:08:12.661267 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:12.661179 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/933b9fd2-86f8-4f62-a3e5-48636487d9b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/933b9fd2-86f8-4f62-a3e5-48636487d9b0-plugins-config-volume\") pod \"933b9fd2-86f8-4f62-a3e5-48636487d9b0\" (UID: \"933b9fd2-86f8-4f62-a3e5-48636487d9b0\") " Oct 30 20:08:12.666530 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:12.666501 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/933b9fd2-86f8-4f62-a3e5-48636487d9b0-kube-api-access-sk7th\" (UniqueName: \"kubernetes.io/projected/933b9fd2-86f8-4f62-a3e5-48636487d9b0-kube-api-access-sk7th\") pod \"933b9fd2-86f8-4f62-a3e5-48636487d9b0\" (UID: \"933b9fd2-86f8-4f62-a3e5-48636487d9b0\") " Oct 30 20:08:12.667686 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:12.662264 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/933b9fd2-86f8-4f62-a3e5-48636487d9b0-plugins-config-volume" pod "933b9fd2-86f8-4f62-a3e5-48636487d9b0" (UID: "933b9fd2-86f8-4f62-a3e5-48636487d9b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:12.676070 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:12.675986 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/933b9fd2-86f8-4f62-a3e5-48636487d9b0-kube-api-access-sk7th" pod "933b9fd2-86f8-4f62-a3e5-48636487d9b0" (UID: "933b9fd2-86f8-4f62-a3e5-48636487d9b0"). InnerVolumeSpecName "kube-api-access-sk7th". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:12.740775 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:12.740745 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24f9b32b-8908-4adf-bda3-5cfc8cf34bc5" path="/var/lib/kubelet/pods/24f9b32b-8908-4adf-bda3-5cfc8cf34bc5/volumes" Oct 30 20:08:12.741238 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:12.741223 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f60e22d-5bad-4327-b28c-4f618abce710" path="/var/lib/kubelet/pods/5f60e22d-5bad-4327-b28c-4f618abce710/volumes" Oct 30 20:08:12.768779 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:12.768742 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk7th\" (UniqueName: \"kubernetes.io/projected/933b9fd2-86f8-4f62-a3e5-48636487d9b0-kube-api-access-sk7th\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:12.768779 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:12.768775 5088 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/933b9fd2-86f8-4f62-a3e5-48636487d9b0-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:13.459443 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:13.459411 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-62f4a1-645b9d8589-k6rx9" Oct 30 20:08:13.459443 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:13.459426 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-62f4a1-645b9d8589-k6rx9" event={"ID":"933b9fd2-86f8-4f62-a3e5-48636487d9b0","Type":"ContainerDied","Data":"f24bdbc40c5a8bd771f53a456989c661966a5cc56077b276321f6c9cd4a9f6e8"} Oct 30 20:08:13.459946 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:13.459483 5088 scope.go:122] "RemoveContainer" containerID="133ce1c35e98bbdfce32864378f02f84461a01e8d0bd8ae7160251bde03e9ece" Oct 30 20:08:13.530581 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:13.530547 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-62f4a1-645b9d8589-k6rx9"] Oct 30 20:08:13.544043 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:13.544008 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-62f4a1-645b9d8589-k6rx9"] Oct 30 20:08:14.800634 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:14.797826 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="933b9fd2-86f8-4f62-a3e5-48636487d9b0" path="/var/lib/kubelet/pods/933b9fd2-86f8-4f62-a3e5-48636487d9b0/volumes" Oct 30 20:08:18.561516 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:18.561477 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:08:21.070554 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:21.070514 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2a06b2b-vvbdx"] Oct 30 20:08:21.167698 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:21.161418 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2a06b2b-vvbdx"] Oct 30 20:08:21.901988 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:21.901943 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" podUID="fec493c7-3f5a-459e-99e0-a4c816d5a907" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:22.079631 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:22.057195 5088 scope.go:122] "RemoveContainer" containerID="d1d18181ce66ed778e943e4fca72c0dd81f04ce988136fbea2a0e94269571404" Oct 30 20:08:22.378638 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:22.378344 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 30 20:08:22.665518 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:22.665470 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:22.895669 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:22.891927 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:22.998724 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:22.990135 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61dbdeab-407a-4902-aa37-28ec91a3770e" path="/var/lib/kubelet/pods/61dbdeab-407a-4902-aa37-28ec91a3770e/volumes" Oct 30 20:08:23.093644 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:08:23.093539 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:08:23.418700 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:23.403848 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 30 20:08:23.418700 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:23.404673 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 20:08:23.689834 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:23.689731 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 20:08:23.690778 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:23.690753 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:08:23.819786 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:23.819739 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:08:23.820183 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:23.820148 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" podUID="98275e84-2e20-46db-8264-27ec2ac2e2f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:23.820684 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:23.820659 5088 patch_prober.go:29] interesting pod/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:08:23.820854 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:23.820833 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" podUID="e402d580-e9a6-4e35-a2ff-e5ef2d78eb90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:23.888759 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:23.888720 5088 patch_prober.go:29] interesting pod/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:08:23.889055 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:23.889006 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" podUID="d44744f7-8f4c-4c79-978b-52649ce5a5ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:23.911249 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:23.911199 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:23.911950 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:23.911922 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" Oct 30 20:08:23.952875 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:23.952773 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" podUID="fec493c7-3f5a-459e-99e0-a4c816d5a907" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:24.505681 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:24.494779 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 30 20:08:24.709756 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:24.709701 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:24.933189 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:24.933132 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:24.993635 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:24.963127 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" podUID="fec493c7-3f5a-459e-99e0-a4c816d5a907" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:25.057745 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:25.038061 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2"] Oct 30 20:08:25.151636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:25.149073 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" podUID="a0c9594e-8534-4538-a4bc-2d40d918e362" containerName="istio-proxy" containerID="cri-o://e2b0d690456a79235aab26cb2f341ef4af7ae03bad18f9d8222163d3b012ead0" gracePeriod=30 Oct 30 20:08:25.584694 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:25.579130 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:25.855700 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:25.824826 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 20:08:25.978836 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:25.926743 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" Oct 30 20:08:25.978836 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:25.971075 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg"] Oct 30 20:08:25.978836 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:25.971132 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r"] Oct 30 20:08:25.978836 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:25.971365 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" podUID="98275e84-2e20-46db-8264-27ec2ac2e2f2" containerName="istio-proxy" containerID="cri-o://c3fb1272968557cba700fced30ca4bd585345d53ef20b3c596965d955230c051" gracePeriod=30 Oct 30 20:08:25.978836 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:25.967706 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" podUID="fec493c7-3f5a-459e-99e0-a4c816d5a907" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": context deadline exceeded" Oct 30 20:08:25.978836 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:25.967754 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:25.987545 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:25.984887 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" podUID="419aecf7-a250-42ad-995b-e7ebd9630a76" containerName="istio-proxy" containerID="cri-o://b05e9b0d77de96b9f1159bdf32750a69a00a0f90e63cece81107203cb2040bc2" gracePeriod=30 Oct 30 20:08:26.011069 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:26.011029 5088 patch_prober.go:29] interesting pod/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:08:26.011325 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:26.011301 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" podUID="a0c9594e-8534-4538-a4bc-2d40d918e362" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:08:26.242235 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:26.242202 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj"] Oct 30 20:08:26.243159 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:26.243125 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" podUID="b55ec6e2-9392-4d8d-8340-ef12f674970c" containerName="istio-proxy" containerID="cri-o://5a1758a4fb019200747822b65a3e9e033faf2d04d992fa5b6cb39c925af93acd" gracePeriod=30 Oct 30 20:08:26.280657 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:26.243767 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d"] Oct 30 20:08:26.317510 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:26.317466 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" podUID="d44744f7-8f4c-4c79-978b-52649ce5a5ed" containerName="istio-proxy" containerID="cri-o://314ac6568ace40ce03f7bf80c6852789d144db290701be78a5ba82f0f10dc195" gracePeriod=30 Oct 30 20:08:26.333329 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:26.333283 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" podUID="fec493c7-3f5a-459e-99e0-a4c816d5a907" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/health\": context deadline exceeded" Oct 30 20:08:26.437848 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:26.437758 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn"] Oct 30 20:08:26.438351 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:26.438317 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" podUID="fc3ed9fa-b7f0-418f-8ac9-422f9e765027" containerName="istio-proxy" containerID="cri-o://ec0589382cf960ccf68133b378f1efc8178b0d114b31e0ab49aabf9a1e03f11a" gracePeriod=30 Oct 30 20:08:26.531215 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:26.531150 5088 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:26.590032 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:26.589997 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" Oct 30 20:08:26.593286 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:26.593229 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" podUID="ee73a969-6d25-490c-b943-2c9771cfa738" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 30 20:08:26.677242 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:26.677189 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s"] Oct 30 20:08:26.688820 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:26.688714 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" podUID="ff410c2e-9048-4e5c-965b-1261336561bb" containerName="istio-proxy" containerID="cri-o://6cfacb45b1d70e6292533f4fefcaba52f16eaac85d2922a2b8acf5e5fb5791d1" gracePeriod=30 Oct 30 20:08:26.689863 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:26.689674 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp"] Oct 30 20:08:26.691241 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:26.691211 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" podUID="d209c20c-000f-4cc4-9792-5a71eeb0a773" containerName="istio-proxy" containerID="cri-o://1619335ad2605965c2c5a4f2c72a93cf39d2404688f2c83428c6b8155e763dc7" gracePeriod=30 Oct 30 20:08:26.815442 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:26.810325 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2" Oct 30 20:08:26.815442 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:26.810388 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:08:26.815442 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:26.810443 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr"] Oct 30 20:08:26.815442 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:26.810484 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s"] Oct 30 20:08:26.816435 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:26.816397 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" podUID="00264202-1595-4616-8796-2ecd7415c577" containerName="istio-proxy" containerID="cri-o://5e32dd83a9a502d3feb8fbe6ae10467c0acbc30fa91cd27137549b4abbe249df" gracePeriod=30 Oct 30 20:08:26.816715 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:26.816606 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" podUID="ba76bfa7-6ba7-4862-8348-ca7c17487b9f" containerName="istio-proxy" containerID="cri-o://f54a4ec28ad66ba296a4360924c66536ec38efc33f4ff1e36c3bb003b0ba44ed" gracePeriod=30 Oct 30 20:08:26.834375 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:26.834351 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" Oct 30 20:08:28.730049 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.730014 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcafb519-gmmj2"] Oct 30 20:08:28.730688 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.730670 5088 cpu_manager.go:414] "containerMap: removing container" podUID="f2757424-fb74-48ff-82c3-ccb79a27fd2a" containerName="istio-proxy" Oct 30 20:08:28.730754 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.730689 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2757424-fb74-48ff-82c3-ccb79a27fd2a" containerName="istio-proxy" Oct 30 20:08:28.730754 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.730703 5088 cpu_manager.go:414] "containerMap: removing container" podUID="baca0e89-a102-4354-af54-c14ccc418923" containerName="istio-proxy" Oct 30 20:08:28.730754 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.730710 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baca0e89-a102-4354-af54-c14ccc418923" containerName="istio-proxy" Oct 30 20:08:28.730754 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.730722 5088 cpu_manager.go:414] "containerMap: removing container" podUID="933b9fd2-86f8-4f62-a3e5-48636487d9b0" containerName="payload-processing" Oct 30 20:08:28.730754 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.730727 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="933b9fd2-86f8-4f62-a3e5-48636487d9b0" containerName="payload-processing" Oct 30 20:08:28.730754 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.730734 5088 cpu_manager.go:414] "containerMap: removing container" podUID="14b2ad56-7181-420c-8ad1-bdc00f25e928" containerName="istio-proxy" Oct 30 20:08:28.730754 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.730739 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14b2ad56-7181-420c-8ad1-bdc00f25e928" containerName="istio-proxy" Oct 30 20:08:28.730754 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.730746 5088 cpu_manager.go:414] "containerMap: removing container" podUID="ab112498-5d12-44e0-85b3-0c9f5dc003d6" containerName="istio-proxy" Oct 30 20:08:28.730754 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.730750 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab112498-5d12-44e0-85b3-0c9f5dc003d6" containerName="istio-proxy" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.730766 5088 cpu_manager.go:414] "containerMap: removing container" podUID="a4af0759-e61a-48f6-821a-e1ad34261f11" containerName="insights-gathering" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.730770 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4af0759-e61a-48f6-821a-e1ad34261f11" containerName="insights-gathering" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.730780 5088 cpu_manager.go:414] "containerMap: removing container" podUID="05a46855-a143-4c6d-8c87-404dc271bdb3" containerName="istio-proxy" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.730784 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05a46855-a143-4c6d-8c87-404dc271bdb3" containerName="istio-proxy" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.730790 5088 cpu_manager.go:414] "containerMap: removing container" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.730794 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.730803 5088 cpu_manager.go:414] "containerMap: removing container" podUID="f4b44c6e-891a-4073-9192-035aeaad10a4" containerName="istio-proxy" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.730807 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4b44c6e-891a-4073-9192-035aeaad10a4" containerName="istio-proxy" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.730813 5088 cpu_manager.go:414] "containerMap: removing container" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.730817 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.730826 5088 cpu_manager.go:414] "containerMap: removing container" podUID="b56fca64-1150-4198-945e-7e830300c11d" containerName="istio-proxy" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.730831 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b56fca64-1150-4198-945e-7e830300c11d" containerName="istio-proxy" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.730839 5088 cpu_manager.go:414] "containerMap: removing container" podUID="a255c4a5-c090-49d2-84ba-536dfb9b8a69" containerName="istio-proxy" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.730843 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a255c4a5-c090-49d2-84ba-536dfb9b8a69" containerName="istio-proxy" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.730858 5088 cpu_manager.go:414] "containerMap: removing container" podUID="cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980" containerName="istio-proxy" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.730862 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980" containerName="istio-proxy" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.730873 5088 cpu_manager.go:414] "containerMap: removing container" podUID="4789c6d9-e328-4890-a2b3-a2fdbfdb04b9" containerName="istio-proxy" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.730877 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4789c6d9-e328-4890-a2b3-a2fdbfdb04b9" containerName="istio-proxy" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.730883 5088 cpu_manager.go:414] "containerMap: removing container" podUID="678518a9-8731-4f6f-a7a1-4b0e64070e92" containerName="istio-proxy" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.730887 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="678518a9-8731-4f6f-a7a1-4b0e64070e92" containerName="istio-proxy" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.730893 5088 cpu_manager.go:414] "containerMap: removing container" podUID="0b36276d-bc5b-4415-8cf9-461aee572f3a" containerName="istio-proxy" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.730897 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b36276d-bc5b-4415-8cf9-461aee572f3a" containerName="istio-proxy" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.730904 5088 cpu_manager.go:414] "containerMap: removing container" podUID="24f9b32b-8908-4adf-bda3-5cfc8cf34bc5" containerName="payload-pre-processing" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.730908 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24f9b32b-8908-4adf-bda3-5cfc8cf34bc5" containerName="payload-pre-processing" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.730916 5088 cpu_manager.go:414] "containerMap: removing container" podUID="f4b44c6e-891a-4073-9192-035aeaad10a4" containerName="istio-proxy" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.730921 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4b44c6e-891a-4073-9192-035aeaad10a4" containerName="istio-proxy" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.730926 5088 cpu_manager.go:414] "containerMap: removing container" podUID="05a46855-a143-4c6d-8c87-404dc271bdb3" containerName="istio-proxy" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.730932 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05a46855-a143-4c6d-8c87-404dc271bdb3" containerName="istio-proxy" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.730942 5088 cpu_manager.go:414] "containerMap: removing container" podUID="ad2f9107-20fe-4ae3-ae9a-e92c315e5a45" containerName="revoke-keys" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.730947 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad2f9107-20fe-4ae3-ae9a-e92c315e5a45" containerName="revoke-keys" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.731046 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="baca0e89-a102-4354-af54-c14ccc418923" containerName="istio-proxy" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.731055 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="933b9fd2-86f8-4f62-a3e5-48636487d9b0" containerName="payload-processing" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.731062 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="14b2ad56-7181-420c-8ad1-bdc00f25e928" containerName="istio-proxy" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.731070 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab112498-5d12-44e0-85b3-0c9f5dc003d6" containerName="istio-proxy" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.731076 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4af0759-e61a-48f6-821a-e1ad34261f11" containerName="insights-gathering" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.731081 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="05a46855-a143-4c6d-8c87-404dc271bdb3" containerName="istio-proxy" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.731087 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="682a764a-19ca-49e8-a1d4-1258b8d74bf4" containerName="main" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.731092 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4b44c6e-891a-4073-9192-035aeaad10a4" containerName="istio-proxy" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.731097 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6cf0cd1-3f5f-4e72-b1cd-f9194e7dac21" containerName="main" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.731104 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="b56fca64-1150-4198-945e-7e830300c11d" containerName="istio-proxy" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.731110 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="a255c4a5-c090-49d2-84ba-536dfb9b8a69" containerName="istio-proxy" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.731117 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdf81fa2-7f24-4dd4-bff8-0ff7f4f64980" containerName="istio-proxy" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.731123 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="4789c6d9-e328-4890-a2b3-a2fdbfdb04b9" containerName="istio-proxy" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.731130 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="678518a9-8731-4f6f-a7a1-4b0e64070e92" containerName="istio-proxy" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.731135 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b36276d-bc5b-4415-8cf9-461aee572f3a" containerName="istio-proxy" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.731140 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="24f9b32b-8908-4adf-bda3-5cfc8cf34bc5" containerName="payload-pre-processing" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.731146 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4b44c6e-891a-4073-9192-035aeaad10a4" containerName="istio-proxy" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.731152 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad2f9107-20fe-4ae3-ae9a-e92c315e5a45" containerName="revoke-keys" Oct 30 20:08:28.731313 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.731158 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2757424-fb74-48ff-82c3-ccb79a27fd2a" containerName="istio-proxy" Oct 30 20:08:28.738569 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.738542 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcafb519-gmmj2" Oct 30 20:08:28.761274 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.761242 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcafb519-gmmj2"] Oct 30 20:08:28.769230 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.769202 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:08:28.769230 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.769206 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:08:28.863402 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.863364 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/442a352a-40cd-4152-97a5-88a598908ac1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcafb519-gmmj2\" (UID: \"442a352a-40cd-4152-97a5-88a598908ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcafb519-gmmj2" Oct 30 20:08:28.863603 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.863581 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/442a352a-40cd-4152-97a5-88a598908ac1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcafb519-gmmj2\" (UID: \"442a352a-40cd-4152-97a5-88a598908ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcafb519-gmmj2" Oct 30 20:08:28.863744 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.863726 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g79vg\" (UniqueName: \"kubernetes.io/projected/442a352a-40cd-4152-97a5-88a598908ac1-kube-api-access-g79vg\") pod \"maas-api-revoke-sub-bcafb519-gmmj2\" (UID: \"442a352a-40cd-4152-97a5-88a598908ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcafb519-gmmj2" Oct 30 20:08:28.965444 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.965400 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g79vg\" (UniqueName: \"kubernetes.io/projected/442a352a-40cd-4152-97a5-88a598908ac1-kube-api-access-g79vg\") pod \"maas-api-revoke-sub-bcafb519-gmmj2\" (UID: \"442a352a-40cd-4152-97a5-88a598908ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcafb519-gmmj2" Oct 30 20:08:28.965751 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.965546 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/442a352a-40cd-4152-97a5-88a598908ac1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcafb519-gmmj2\" (UID: \"442a352a-40cd-4152-97a5-88a598908ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcafb519-gmmj2" Oct 30 20:08:28.965829 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.965747 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/442a352a-40cd-4152-97a5-88a598908ac1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcafb519-gmmj2\" (UID: \"442a352a-40cd-4152-97a5-88a598908ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcafb519-gmmj2" Oct 30 20:08:28.966269 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.966245 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/442a352a-40cd-4152-97a5-88a598908ac1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcafb519-gmmj2\" (UID: \"442a352a-40cd-4152-97a5-88a598908ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcafb519-gmmj2" Oct 30 20:08:28.968179 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.968158 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/442a352a-40cd-4152-97a5-88a598908ac1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcafb519-gmmj2\" (UID: \"442a352a-40cd-4152-97a5-88a598908ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcafb519-gmmj2" Oct 30 20:08:28.996520 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:28.996491 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g79vg\" (UniqueName: \"kubernetes.io/projected/442a352a-40cd-4152-97a5-88a598908ac1-kube-api-access-g79vg\") pod \"maas-api-revoke-sub-bcafb519-gmmj2\" (UID: \"442a352a-40cd-4152-97a5-88a598908ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcafb519-gmmj2" Oct 30 20:08:29.053425 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:29.053395 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcafb519-gmmj2" Oct 30 20:08:29.367474 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:08:29.367437 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod442a352a_40cd_4152_97a5_88a598908ac1.slice/crio-3527ad9fad34b82c55d3221cb5d8d3e13594207b8578be727e728c849ff85d70 WatchSource:0}: Error finding container 3527ad9fad34b82c55d3221cb5d8d3e13594207b8578be727e728c849ff85d70: Status 404 returned error can't find the container with id 3527ad9fad34b82c55d3221cb5d8d3e13594207b8578be727e728c849ff85d70 Oct 30 20:08:29.367782 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:29.367758 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcafb519-gmmj2"] Oct 30 20:08:29.827591 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:29.827547 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcafb519-gmmj2" event={"ID":"442a352a-40cd-4152-97a5-88a598908ac1","Type":"ContainerStarted","Data":"f98887d568494e01a9837fcaa2ce0d347dbc14bbea23527a601f5d8f8403c522"} Oct 30 20:08:29.828057 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:29.827628 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcafb519-gmmj2" event={"ID":"442a352a-40cd-4152-97a5-88a598908ac1","Type":"ContainerStarted","Data":"3527ad9fad34b82c55d3221cb5d8d3e13594207b8578be727e728c849ff85d70"} Oct 30 20:08:29.895966 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:29.895914 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcafb519-gmmj2" podStartSLOduration=1.895899706 podStartE2EDuration="1.895899706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:29.881746639 +0000 UTC m=+1271.294460449" watchObservedRunningTime="2026-10-30 20:08:29.895899706 +0000 UTC m=+1271.308613484" Oct 30 20:08:30.844525 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:30.844493 5088 generic.go:356] "Generic (PLEG): container finished" podID="442a352a-40cd-4152-97a5-88a598908ac1" containerID="f98887d568494e01a9837fcaa2ce0d347dbc14bbea23527a601f5d8f8403c522" exitCode=0 Oct 30 20:08:30.844967 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:30.844580 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcafb519-gmmj2" event={"ID":"442a352a-40cd-4152-97a5-88a598908ac1","Type":"ContainerDied","Data":"f98887d568494e01a9837fcaa2ce0d347dbc14bbea23527a601f5d8f8403c522"} Oct 30 20:08:31.773056 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:08:31.773012 5088 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd209c20c_000f_4cc4_9792_5a71eeb0a773.slice/crio-conmon-1619335ad2605965c2c5a4f2c72a93cf39d2404688f2c83428c6b8155e763dc7.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb55ec6e2_9392_4d8d_8340_ef12f674970c.slice/crio-conmon-5a1758a4fb019200747822b65a3e9e033faf2d04d992fa5b6cb39c925af93acd.scope\": RecentStats: unable to find data in memory cache]" Oct 30 20:08:31.858353 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:31.858320 5088 generic.go:356] "Generic (PLEG): container finished" podID="d44744f7-8f4c-4c79-978b-52649ce5a5ed" containerID="314ac6568ace40ce03f7bf80c6852789d144db290701be78a5ba82f0f10dc195" exitCode=0 Oct 30 20:08:31.858865 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:31.858401 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" event={"ID":"d44744f7-8f4c-4c79-978b-52649ce5a5ed","Type":"ContainerDied","Data":"314ac6568ace40ce03f7bf80c6852789d144db290701be78a5ba82f0f10dc195"} Oct 30 20:08:31.859885 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:31.859861 5088 generic.go:356] "Generic (PLEG): container finished" podID="a0c9594e-8534-4538-a4bc-2d40d918e362" containerID="e2b0d690456a79235aab26cb2f341ef4af7ae03bad18f9d8222163d3b012ead0" exitCode=0 Oct 30 20:08:31.860025 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:31.860009 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" event={"ID":"a0c9594e-8534-4538-a4bc-2d40d918e362","Type":"ContainerDied","Data":"e2b0d690456a79235aab26cb2f341ef4af7ae03bad18f9d8222163d3b012ead0"} Oct 30 20:08:31.860073 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:31.860050 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" event={"ID":"a0c9594e-8534-4538-a4bc-2d40d918e362","Type":"ContainerDied","Data":"a60e757d74802457065a4422e0679f5350f09a38c4a0a49d045e980d88d8514b"} Oct 30 20:08:31.860073 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:31.860065 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a60e757d74802457065a4422e0679f5350f09a38c4a0a49d045e980d88d8514b" Oct 30 20:08:31.861755 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:31.861736 5088 generic.go:356] "Generic (PLEG): container finished" podID="ff410c2e-9048-4e5c-965b-1261336561bb" containerID="6cfacb45b1d70e6292533f4fefcaba52f16eaac85d2922a2b8acf5e5fb5791d1" exitCode=0 Oct 30 20:08:31.861813 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:31.861803 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" event={"ID":"ff410c2e-9048-4e5c-965b-1261336561bb","Type":"ContainerDied","Data":"6cfacb45b1d70e6292533f4fefcaba52f16eaac85d2922a2b8acf5e5fb5791d1"} Oct 30 20:08:31.863453 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:31.863431 5088 generic.go:356] "Generic (PLEG): container finished" podID="d209c20c-000f-4cc4-9792-5a71eeb0a773" containerID="1619335ad2605965c2c5a4f2c72a93cf39d2404688f2c83428c6b8155e763dc7" exitCode=0 Oct 30 20:08:31.863556 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:31.863501 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" event={"ID":"d209c20c-000f-4cc4-9792-5a71eeb0a773","Type":"ContainerDied","Data":"1619335ad2605965c2c5a4f2c72a93cf39d2404688f2c83428c6b8155e763dc7"} Oct 30 20:08:31.865561 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:31.865535 5088 generic.go:356] "Generic (PLEG): container finished" podID="98275e84-2e20-46db-8264-27ec2ac2e2f2" containerID="c3fb1272968557cba700fced30ca4bd585345d53ef20b3c596965d955230c051" exitCode=0 Oct 30 20:08:31.865674 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:31.865594 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" event={"ID":"98275e84-2e20-46db-8264-27ec2ac2e2f2","Type":"ContainerDied","Data":"c3fb1272968557cba700fced30ca4bd585345d53ef20b3c596965d955230c051"} Oct 30 20:08:31.865674 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:31.865660 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" event={"ID":"98275e84-2e20-46db-8264-27ec2ac2e2f2","Type":"ContainerDied","Data":"55f92264bb19aec87a9df7cbc8b68908112c59f40da1db7e9bfd362207a8e997"} Oct 30 20:08:31.865759 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:31.865686 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55f92264bb19aec87a9df7cbc8b68908112c59f40da1db7e9bfd362207a8e997" Oct 30 20:08:31.867379 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:31.867356 5088 generic.go:356] "Generic (PLEG): container finished" podID="fc3ed9fa-b7f0-418f-8ac9-422f9e765027" containerID="ec0589382cf960ccf68133b378f1efc8178b0d114b31e0ab49aabf9a1e03f11a" exitCode=0 Oct 30 20:08:31.867471 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:31.867434 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" event={"ID":"fc3ed9fa-b7f0-418f-8ac9-422f9e765027","Type":"ContainerDied","Data":"ec0589382cf960ccf68133b378f1efc8178b0d114b31e0ab49aabf9a1e03f11a"} Oct 30 20:08:31.867520 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:31.867472 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" event={"ID":"fc3ed9fa-b7f0-418f-8ac9-422f9e765027","Type":"ContainerDied","Data":"9c9e83ff2799a95ce0050664bd68844e66ec5196f8b92985e703b89ffa636026"} Oct 30 20:08:31.867520 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:31.867504 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c9e83ff2799a95ce0050664bd68844e66ec5196f8b92985e703b89ffa636026" Oct 30 20:08:31.869545 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:31.869508 5088 generic.go:356] "Generic (PLEG): container finished" podID="b55ec6e2-9392-4d8d-8340-ef12f674970c" containerID="5a1758a4fb019200747822b65a3e9e033faf2d04d992fa5b6cb39c925af93acd" exitCode=0 Oct 30 20:08:31.869715 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:31.869697 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" event={"ID":"b55ec6e2-9392-4d8d-8340-ef12f674970c","Type":"ContainerDied","Data":"5a1758a4fb019200747822b65a3e9e033faf2d04d992fa5b6cb39c925af93acd"} Oct 30 20:08:31.871007 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:31.870987 5088 generic.go:356] "Generic (PLEG): container finished" podID="419aecf7-a250-42ad-995b-e7ebd9630a76" containerID="b05e9b0d77de96b9f1159bdf32750a69a00a0f90e63cece81107203cb2040bc2" exitCode=0 Oct 30 20:08:31.871211 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:31.871193 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" event={"ID":"419aecf7-a250-42ad-995b-e7ebd9630a76","Type":"ContainerDied","Data":"b05e9b0d77de96b9f1159bdf32750a69a00a0f90e63cece81107203cb2040bc2"} Oct 30 20:08:31.871321 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:31.871233 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" event={"ID":"419aecf7-a250-42ad-995b-e7ebd9630a76","Type":"ContainerDied","Data":"a4a188e6739c6e39675d0b3e5a71629da637b3589df8952dfe382ae853ec085c"} Oct 30 20:08:31.871321 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:31.871298 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4a188e6739c6e39675d0b3e5a71629da637b3589df8952dfe382ae853ec085c" Oct 30 20:08:31.951021 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:31.950584 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" Oct 30 20:08:31.966246 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:31.966145 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" Oct 30 20:08:31.972842 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:31.972742 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" Oct 30 20:08:31.981889 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:31.981795 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" Oct 30 20:08:32.000680 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:31.998441 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" Oct 30 20:08:32.020726 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.020694 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-workload-socket\") pod \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\" (UID: \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\") " Oct 30 20:08:32.020876 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.020757 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0c9594e-8534-4538-a4bc-2d40d918e362-kube-api-access-mqb82\" (UniqueName: \"kubernetes.io/projected/a0c9594e-8534-4538-a4bc-2d40d918e362-kube-api-access-mqb82\") pod \"a0c9594e-8534-4538-a4bc-2d40d918e362\" (UID: \"a0c9594e-8534-4538-a4bc-2d40d918e362\") " Oct 30 20:08:32.020876 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.020790 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/419aecf7-a250-42ad-995b-e7ebd9630a76-istio-token\" (UniqueName: \"kubernetes.io/projected/419aecf7-a250-42ad-995b-e7ebd9630a76-istio-token\") pod \"419aecf7-a250-42ad-995b-e7ebd9630a76\" (UID: \"419aecf7-a250-42ad-995b-e7ebd9630a76\") " Oct 30 20:08:32.020876 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.020817 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0c9594e-8534-4538-a4bc-2d40d918e362-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0c9594e-8534-4538-a4bc-2d40d918e362-workload-socket\") pod \"a0c9594e-8534-4538-a4bc-2d40d918e362\" (UID: \"a0c9594e-8534-4538-a4bc-2d40d918e362\") " Oct 30 20:08:32.020876 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.020848 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0c9594e-8534-4538-a4bc-2d40d918e362-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0c9594e-8534-4538-a4bc-2d40d918e362-istiod-ca-cert\") pod \"a0c9594e-8534-4538-a4bc-2d40d918e362\" (UID: \"a0c9594e-8534-4538-a4bc-2d40d918e362\") " Oct 30 20:08:32.021117 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.020894 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b55ec6e2-9392-4d8d-8340-ef12f674970c-istio-token\" (UniqueName: \"kubernetes.io/projected/b55ec6e2-9392-4d8d-8340-ef12f674970c-istio-token\") pod \"b55ec6e2-9392-4d8d-8340-ef12f674970c\" (UID: \"b55ec6e2-9392-4d8d-8340-ef12f674970c\") " Oct 30 20:08:32.021117 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.020925 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0c9594e-8534-4538-a4bc-2d40d918e362-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0c9594e-8534-4538-a4bc-2d40d918e362-credential-socket\") pod \"a0c9594e-8534-4538-a4bc-2d40d918e362\" (UID: \"a0c9594e-8534-4538-a4bc-2d40d918e362\") " Oct 30 20:08:32.021117 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.020954 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/419aecf7-a250-42ad-995b-e7ebd9630a76-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/419aecf7-a250-42ad-995b-e7ebd9630a76-workload-certs\") pod \"419aecf7-a250-42ad-995b-e7ebd9630a76\" (UID: \"419aecf7-a250-42ad-995b-e7ebd9630a76\") " Oct 30 20:08:32.021117 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.020983 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-istio-podinfo\") pod \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\" (UID: \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\") " Oct 30 20:08:32.021117 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.021018 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98275e84-2e20-46db-8264-27ec2ac2e2f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98275e84-2e20-46db-8264-27ec2ac2e2f2-istio-data\") pod \"98275e84-2e20-46db-8264-27ec2ac2e2f2\" (UID: \"98275e84-2e20-46db-8264-27ec2ac2e2f2\") " Oct 30 20:08:32.021117 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.021054 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-istiod-ca-cert\") pod \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\" (UID: \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\") " Oct 30 20:08:32.021117 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.021090 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/419aecf7-a250-42ad-995b-e7ebd9630a76-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/419aecf7-a250-42ad-995b-e7ebd9630a76-credential-socket\") pod \"419aecf7-a250-42ad-995b-e7ebd9630a76\" (UID: \"419aecf7-a250-42ad-995b-e7ebd9630a76\") " Oct 30 20:08:32.021117 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.021116 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0c9594e-8534-4538-a4bc-2d40d918e362-istio-token\" (UniqueName: \"kubernetes.io/projected/a0c9594e-8534-4538-a4bc-2d40d918e362-istio-token\") pod \"a0c9594e-8534-4538-a4bc-2d40d918e362\" (UID: \"a0c9594e-8534-4538-a4bc-2d40d918e362\") " Oct 30 20:08:32.021532 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.021154 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-kube-api-access-mp5fz\" (UniqueName: \"kubernetes.io/projected/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-kube-api-access-mp5fz\") pod \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\" (UID: \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\") " Oct 30 20:08:32.021532 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.021211 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-istio-token\" (UniqueName: \"kubernetes.io/projected/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-istio-token\") pod \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\" (UID: \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\") " Oct 30 20:08:32.021532 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.021247 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b55ec6e2-9392-4d8d-8340-ef12f674970c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b55ec6e2-9392-4d8d-8340-ef12f674970c-istio-data\") pod \"b55ec6e2-9392-4d8d-8340-ef12f674970c\" (UID: \"b55ec6e2-9392-4d8d-8340-ef12f674970c\") " Oct 30 20:08:32.021532 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.021285 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98275e84-2e20-46db-8264-27ec2ac2e2f2-istio-token\" (UniqueName: \"kubernetes.io/projected/98275e84-2e20-46db-8264-27ec2ac2e2f2-istio-token\") pod \"98275e84-2e20-46db-8264-27ec2ac2e2f2\" (UID: \"98275e84-2e20-46db-8264-27ec2ac2e2f2\") " Oct 30 20:08:32.021532 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.021352 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98275e84-2e20-46db-8264-27ec2ac2e2f2-kube-api-access-86cmb\" (UniqueName: \"kubernetes.io/projected/98275e84-2e20-46db-8264-27ec2ac2e2f2-kube-api-access-86cmb\") pod \"98275e84-2e20-46db-8264-27ec2ac2e2f2\" (UID: \"98275e84-2e20-46db-8264-27ec2ac2e2f2\") " Oct 30 20:08:32.021532 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.021382 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b55ec6e2-9392-4d8d-8340-ef12f674970c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b55ec6e2-9392-4d8d-8340-ef12f674970c-istiod-ca-cert\") pod \"b55ec6e2-9392-4d8d-8340-ef12f674970c\" (UID: \"b55ec6e2-9392-4d8d-8340-ef12f674970c\") " Oct 30 20:08:32.021532 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.021426 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98275e84-2e20-46db-8264-27ec2ac2e2f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98275e84-2e20-46db-8264-27ec2ac2e2f2-credential-socket\") pod \"98275e84-2e20-46db-8264-27ec2ac2e2f2\" (UID: \"98275e84-2e20-46db-8264-27ec2ac2e2f2\") " Oct 30 20:08:32.021532 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.021457 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0c9594e-8534-4538-a4bc-2d40d918e362-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0c9594e-8534-4538-a4bc-2d40d918e362-istio-podinfo\") pod \"a0c9594e-8534-4538-a4bc-2d40d918e362\" (UID: \"a0c9594e-8534-4538-a4bc-2d40d918e362\") " Oct 30 20:08:32.021532 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.021487 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-istio-envoy\") pod \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\" (UID: \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\") " Oct 30 20:08:32.021532 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.021488 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-workload-socket" pod "fc3ed9fa-b7f0-418f-8ac9-422f9e765027" (UID: "fc3ed9fa-b7f0-418f-8ac9-422f9e765027"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.021532 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.021522 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98275e84-2e20-46db-8264-27ec2ac2e2f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98275e84-2e20-46db-8264-27ec2ac2e2f2-workload-socket\") pod \"98275e84-2e20-46db-8264-27ec2ac2e2f2\" (UID: \"98275e84-2e20-46db-8264-27ec2ac2e2f2\") " Oct 30 20:08:32.022109 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.021551 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0c9594e-8534-4538-a4bc-2d40d918e362-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0c9594e-8534-4538-a4bc-2d40d918e362-workload-certs\") pod \"a0c9594e-8534-4538-a4bc-2d40d918e362\" (UID: \"a0c9594e-8534-4538-a4bc-2d40d918e362\") " Oct 30 20:08:32.022109 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.021595 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/419aecf7-a250-42ad-995b-e7ebd9630a76-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/419aecf7-a250-42ad-995b-e7ebd9630a76-istio-data\") pod \"419aecf7-a250-42ad-995b-e7ebd9630a76\" (UID: \"419aecf7-a250-42ad-995b-e7ebd9630a76\") " Oct 30 20:08:32.022109 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.021653 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98275e84-2e20-46db-8264-27ec2ac2e2f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98275e84-2e20-46db-8264-27ec2ac2e2f2-istiod-ca-cert\") pod \"98275e84-2e20-46db-8264-27ec2ac2e2f2\" (UID: \"98275e84-2e20-46db-8264-27ec2ac2e2f2\") " Oct 30 20:08:32.022109 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.021686 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-istio-data\") pod \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\" (UID: \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\") " Oct 30 20:08:32.022109 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.021751 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b55ec6e2-9392-4d8d-8340-ef12f674970c-kube-api-access-5xtt5\" (UniqueName: \"kubernetes.io/projected/b55ec6e2-9392-4d8d-8340-ef12f674970c-kube-api-access-5xtt5\") pod \"b55ec6e2-9392-4d8d-8340-ef12f674970c\" (UID: \"b55ec6e2-9392-4d8d-8340-ef12f674970c\") " Oct 30 20:08:32.022109 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.021786 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b55ec6e2-9392-4d8d-8340-ef12f674970c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b55ec6e2-9392-4d8d-8340-ef12f674970c-credential-socket\") pod \"b55ec6e2-9392-4d8d-8340-ef12f674970c\" (UID: \"b55ec6e2-9392-4d8d-8340-ef12f674970c\") " Oct 30 20:08:32.022109 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.021797 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98275e84-2e20-46db-8264-27ec2ac2e2f2-istio-data" pod "98275e84-2e20-46db-8264-27ec2ac2e2f2" (UID: "98275e84-2e20-46db-8264-27ec2ac2e2f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.022109 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.021821 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0c9594e-8534-4538-a4bc-2d40d918e362-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0c9594e-8534-4538-a4bc-2d40d918e362-istio-envoy\") pod \"a0c9594e-8534-4538-a4bc-2d40d918e362\" (UID: \"a0c9594e-8534-4538-a4bc-2d40d918e362\") " Oct 30 20:08:32.022109 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.021860 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0c9594e-8534-4538-a4bc-2d40d918e362-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0c9594e-8534-4538-a4bc-2d40d918e362-istio-data\") pod \"a0c9594e-8534-4538-a4bc-2d40d918e362\" (UID: \"a0c9594e-8534-4538-a4bc-2d40d918e362\") " Oct 30 20:08:32.022565 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.022142 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0c9594e-8534-4538-a4bc-2d40d918e362-istio-data" pod "a0c9594e-8534-4538-a4bc-2d40d918e362" (UID: "a0c9594e-8534-4538-a4bc-2d40d918e362"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.022565 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.022426 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b55ec6e2-9392-4d8d-8340-ef12f674970c-istiod-ca-cert" pod "b55ec6e2-9392-4d8d-8340-ef12f674970c" (UID: "b55ec6e2-9392-4d8d-8340-ef12f674970c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:32.022737 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.022533 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-istiod-ca-cert" pod "fc3ed9fa-b7f0-418f-8ac9-422f9e765027" (UID: "fc3ed9fa-b7f0-418f-8ac9-422f9e765027"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:32.022737 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.022685 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98275e84-2e20-46db-8264-27ec2ac2e2f2-credential-socket" pod "98275e84-2e20-46db-8264-27ec2ac2e2f2" (UID: "98275e84-2e20-46db-8264-27ec2ac2e2f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.022894 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.022862 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/419aecf7-a250-42ad-995b-e7ebd9630a76-credential-socket" pod "419aecf7-a250-42ad-995b-e7ebd9630a76" (UID: "419aecf7-a250-42ad-995b-e7ebd9630a76"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.023427 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.023394 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0c9594e-8534-4538-a4bc-2d40d918e362-workload-socket" pod "a0c9594e-8534-4538-a4bc-2d40d918e362" (UID: "a0c9594e-8534-4538-a4bc-2d40d918e362"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.023977 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.023699 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0c9594e-8534-4538-a4bc-2d40d918e362-istiod-ca-cert" pod "a0c9594e-8534-4538-a4bc-2d40d918e362" (UID: "a0c9594e-8534-4538-a4bc-2d40d918e362"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:32.027320 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.027293 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/419aecf7-a250-42ad-995b-e7ebd9630a76-workload-certs" pod "419aecf7-a250-42ad-995b-e7ebd9630a76" (UID: "419aecf7-a250-42ad-995b-e7ebd9630a76"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.028406 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.028377 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98275e84-2e20-46db-8264-27ec2ac2e2f2-istiod-ca-cert" pod "98275e84-2e20-46db-8264-27ec2ac2e2f2" (UID: "98275e84-2e20-46db-8264-27ec2ac2e2f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:32.028853 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.028826 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b55ec6e2-9392-4d8d-8340-ef12f674970c-istio-data" pod "b55ec6e2-9392-4d8d-8340-ef12f674970c" (UID: "b55ec6e2-9392-4d8d-8340-ef12f674970c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.034667 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.030671 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b55ec6e2-9392-4d8d-8340-ef12f674970c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b55ec6e2-9392-4d8d-8340-ef12f674970c-istio-envoy\") pod \"b55ec6e2-9392-4d8d-8340-ef12f674970c\" (UID: \"b55ec6e2-9392-4d8d-8340-ef12f674970c\") " Oct 30 20:08:32.034667 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.030745 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/419aecf7-a250-42ad-995b-e7ebd9630a76-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/419aecf7-a250-42ad-995b-e7ebd9630a76-istio-podinfo\") pod \"419aecf7-a250-42ad-995b-e7ebd9630a76\" (UID: \"419aecf7-a250-42ad-995b-e7ebd9630a76\") " Oct 30 20:08:32.034667 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.030792 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/419aecf7-a250-42ad-995b-e7ebd9630a76-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/419aecf7-a250-42ad-995b-e7ebd9630a76-workload-socket\") pod \"419aecf7-a250-42ad-995b-e7ebd9630a76\" (UID: \"419aecf7-a250-42ad-995b-e7ebd9630a76\") " Oct 30 20:08:32.034667 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.030820 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b55ec6e2-9392-4d8d-8340-ef12f674970c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b55ec6e2-9392-4d8d-8340-ef12f674970c-workload-socket\") pod \"b55ec6e2-9392-4d8d-8340-ef12f674970c\" (UID: \"b55ec6e2-9392-4d8d-8340-ef12f674970c\") " Oct 30 20:08:32.034667 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.030855 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-credential-socket\") pod \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\" (UID: \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\") " Oct 30 20:08:32.034667 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.030891 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b55ec6e2-9392-4d8d-8340-ef12f674970c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b55ec6e2-9392-4d8d-8340-ef12f674970c-istio-podinfo\") pod \"b55ec6e2-9392-4d8d-8340-ef12f674970c\" (UID: \"b55ec6e2-9392-4d8d-8340-ef12f674970c\") " Oct 30 20:08:32.034667 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.031013 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-workload-certs\") pod \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\" (UID: \"fc3ed9fa-b7f0-418f-8ac9-422f9e765027\") " Oct 30 20:08:32.034667 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.031054 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98275e84-2e20-46db-8264-27ec2ac2e2f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98275e84-2e20-46db-8264-27ec2ac2e2f2-istio-podinfo\") pod \"98275e84-2e20-46db-8264-27ec2ac2e2f2\" (UID: \"98275e84-2e20-46db-8264-27ec2ac2e2f2\") " Oct 30 20:08:32.034667 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.031083 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98275e84-2e20-46db-8264-27ec2ac2e2f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98275e84-2e20-46db-8264-27ec2ac2e2f2-workload-certs\") pod \"98275e84-2e20-46db-8264-27ec2ac2e2f2\" (UID: \"98275e84-2e20-46db-8264-27ec2ac2e2f2\") " Oct 30 20:08:32.034667 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.031112 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/419aecf7-a250-42ad-995b-e7ebd9630a76-kube-api-access-xj2vj\" (UniqueName: \"kubernetes.io/projected/419aecf7-a250-42ad-995b-e7ebd9630a76-kube-api-access-xj2vj\") pod \"419aecf7-a250-42ad-995b-e7ebd9630a76\" (UID: \"419aecf7-a250-42ad-995b-e7ebd9630a76\") " Oct 30 20:08:32.034667 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.031175 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/419aecf7-a250-42ad-995b-e7ebd9630a76-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/419aecf7-a250-42ad-995b-e7ebd9630a76-istio-envoy\") pod \"419aecf7-a250-42ad-995b-e7ebd9630a76\" (UID: \"419aecf7-a250-42ad-995b-e7ebd9630a76\") " Oct 30 20:08:32.034667 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.031215 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/419aecf7-a250-42ad-995b-e7ebd9630a76-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/419aecf7-a250-42ad-995b-e7ebd9630a76-istiod-ca-cert\") pod \"419aecf7-a250-42ad-995b-e7ebd9630a76\" (UID: \"419aecf7-a250-42ad-995b-e7ebd9630a76\") " Oct 30 20:08:32.034667 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.031248 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b55ec6e2-9392-4d8d-8340-ef12f674970c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b55ec6e2-9392-4d8d-8340-ef12f674970c-workload-certs\") pod \"b55ec6e2-9392-4d8d-8340-ef12f674970c\" (UID: \"b55ec6e2-9392-4d8d-8340-ef12f674970c\") " Oct 30 20:08:32.034667 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.031315 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98275e84-2e20-46db-8264-27ec2ac2e2f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98275e84-2e20-46db-8264-27ec2ac2e2f2-istio-envoy\") pod \"98275e84-2e20-46db-8264-27ec2ac2e2f2\" (UID: \"98275e84-2e20-46db-8264-27ec2ac2e2f2\") " Oct 30 20:08:32.034667 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.033999 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0c9594e-8534-4538-a4bc-2d40d918e362-credential-socket" pod "a0c9594e-8534-4538-a4bc-2d40d918e362" (UID: "a0c9594e-8534-4538-a4bc-2d40d918e362"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.035539 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.035461 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-istio-data" pod "fc3ed9fa-b7f0-418f-8ac9-422f9e765027" (UID: "fc3ed9fa-b7f0-418f-8ac9-422f9e765027"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.035539 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.035524 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-istio-podinfo" pod "fc3ed9fa-b7f0-418f-8ac9-422f9e765027" (UID: "fc3ed9fa-b7f0-418f-8ac9-422f9e765027"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:08:32.035689 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.035585 5088 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.035689 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.035605 5088 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0c9594e-8534-4538-a4bc-2d40d918e362-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.035689 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.035639 5088 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0c9594e-8534-4538-a4bc-2d40d918e362-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.035689 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.035653 5088 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/419aecf7-a250-42ad-995b-e7ebd9630a76-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.035689 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.035667 5088 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98275e84-2e20-46db-8264-27ec2ac2e2f2-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.035689 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.035682 5088 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.035945 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.035696 5088 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/419aecf7-a250-42ad-995b-e7ebd9630a76-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.035945 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.035713 5088 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b55ec6e2-9392-4d8d-8340-ef12f674970c-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.035945 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.035730 5088 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b55ec6e2-9392-4d8d-8340-ef12f674970c-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.035945 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.035744 5088 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98275e84-2e20-46db-8264-27ec2ac2e2f2-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.035945 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.035758 5088 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98275e84-2e20-46db-8264-27ec2ac2e2f2-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.035945 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.035773 5088 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0c9594e-8534-4538-a4bc-2d40d918e362-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.036545 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.036484 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/419aecf7-a250-42ad-995b-e7ebd9630a76-workload-socket" pod "419aecf7-a250-42ad-995b-e7ebd9630a76" (UID: "419aecf7-a250-42ad-995b-e7ebd9630a76"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.036743 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.036570 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b55ec6e2-9392-4d8d-8340-ef12f674970c-credential-socket" pod "b55ec6e2-9392-4d8d-8340-ef12f674970c" (UID: "b55ec6e2-9392-4d8d-8340-ef12f674970c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.036743 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.036703 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b55ec6e2-9392-4d8d-8340-ef12f674970c-workload-socket" pod "b55ec6e2-9392-4d8d-8340-ef12f674970c" (UID: "b55ec6e2-9392-4d8d-8340-ef12f674970c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.036893 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.036780 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98275e84-2e20-46db-8264-27ec2ac2e2f2-workload-certs" pod "98275e84-2e20-46db-8264-27ec2ac2e2f2" (UID: "98275e84-2e20-46db-8264-27ec2ac2e2f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.036986 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.036966 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-credential-socket" pod "fc3ed9fa-b7f0-418f-8ac9-422f9e765027" (UID: "fc3ed9fa-b7f0-418f-8ac9-422f9e765027"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.037786 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.037755 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/419aecf7-a250-42ad-995b-e7ebd9630a76-istiod-ca-cert" pod "419aecf7-a250-42ad-995b-e7ebd9630a76" (UID: "419aecf7-a250-42ad-995b-e7ebd9630a76"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:32.039545 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.039198 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-workload-certs" pod "fc3ed9fa-b7f0-418f-8ac9-422f9e765027" (UID: "fc3ed9fa-b7f0-418f-8ac9-422f9e765027"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.040716 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.040690 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b55ec6e2-9392-4d8d-8340-ef12f674970c-workload-certs" pod "b55ec6e2-9392-4d8d-8340-ef12f674970c" (UID: "b55ec6e2-9392-4d8d-8340-ef12f674970c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.041106 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.041084 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0c9594e-8534-4538-a4bc-2d40d918e362-workload-certs" pod "a0c9594e-8534-4538-a4bc-2d40d918e362" (UID: "a0c9594e-8534-4538-a4bc-2d40d918e362"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.043775 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.042600 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b55ec6e2-9392-4d8d-8340-ef12f674970c-kube-api-access-5xtt5" pod "b55ec6e2-9392-4d8d-8340-ef12f674970c" (UID: "b55ec6e2-9392-4d8d-8340-ef12f674970c"). InnerVolumeSpecName "kube-api-access-5xtt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:32.043775 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.042721 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/419aecf7-a250-42ad-995b-e7ebd9630a76-istio-envoy" pod "419aecf7-a250-42ad-995b-e7ebd9630a76" (UID: "419aecf7-a250-42ad-995b-e7ebd9630a76"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.043775 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.042787 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98275e84-2e20-46db-8264-27ec2ac2e2f2-istio-podinfo" pod "98275e84-2e20-46db-8264-27ec2ac2e2f2" (UID: "98275e84-2e20-46db-8264-27ec2ac2e2f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:08:32.043775 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.042831 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b55ec6e2-9392-4d8d-8340-ef12f674970c-istio-envoy" pod "b55ec6e2-9392-4d8d-8340-ef12f674970c" (UID: "b55ec6e2-9392-4d8d-8340-ef12f674970c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.043775 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.043062 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/419aecf7-a250-42ad-995b-e7ebd9630a76-istio-data" pod "419aecf7-a250-42ad-995b-e7ebd9630a76" (UID: "419aecf7-a250-42ad-995b-e7ebd9630a76"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.043775 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.043673 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98275e84-2e20-46db-8264-27ec2ac2e2f2-istio-token" pod "98275e84-2e20-46db-8264-27ec2ac2e2f2" (UID: "98275e84-2e20-46db-8264-27ec2ac2e2f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:32.047187 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.044249 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98275e84-2e20-46db-8264-27ec2ac2e2f2-workload-socket" pod "98275e84-2e20-46db-8264-27ec2ac2e2f2" (UID: "98275e84-2e20-46db-8264-27ec2ac2e2f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.047187 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.045045 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b55ec6e2-9392-4d8d-8340-ef12f674970c-istio-podinfo" pod "b55ec6e2-9392-4d8d-8340-ef12f674970c" (UID: "b55ec6e2-9392-4d8d-8340-ef12f674970c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:08:32.047187 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.045299 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/419aecf7-a250-42ad-995b-e7ebd9630a76-istio-podinfo" pod "419aecf7-a250-42ad-995b-e7ebd9630a76" (UID: "419aecf7-a250-42ad-995b-e7ebd9630a76"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:08:32.047187 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.046387 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98275e84-2e20-46db-8264-27ec2ac2e2f2-istio-envoy" pod "98275e84-2e20-46db-8264-27ec2ac2e2f2" (UID: "98275e84-2e20-46db-8264-27ec2ac2e2f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.047187 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.046557 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-istio-envoy" pod "fc3ed9fa-b7f0-418f-8ac9-422f9e765027" (UID: "fc3ed9fa-b7f0-418f-8ac9-422f9e765027"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.047187 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.046572 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-kube-api-access-mp5fz" pod "fc3ed9fa-b7f0-418f-8ac9-422f9e765027" (UID: "fc3ed9fa-b7f0-418f-8ac9-422f9e765027"). InnerVolumeSpecName "kube-api-access-mp5fz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:32.048450 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.048426 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/419aecf7-a250-42ad-995b-e7ebd9630a76-kube-api-access-xj2vj" pod "419aecf7-a250-42ad-995b-e7ebd9630a76" (UID: "419aecf7-a250-42ad-995b-e7ebd9630a76"). InnerVolumeSpecName "kube-api-access-xj2vj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:32.050236 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.050216 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0c9594e-8534-4538-a4bc-2d40d918e362-istio-envoy" pod "a0c9594e-8534-4538-a4bc-2d40d918e362" (UID: "a0c9594e-8534-4538-a4bc-2d40d918e362"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.050700 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.050678 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0c9594e-8534-4538-a4bc-2d40d918e362-istio-podinfo" pod "a0c9594e-8534-4538-a4bc-2d40d918e362" (UID: "a0c9594e-8534-4538-a4bc-2d40d918e362"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:08:32.050830 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.050811 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98275e84-2e20-46db-8264-27ec2ac2e2f2-kube-api-access-86cmb" pod "98275e84-2e20-46db-8264-27ec2ac2e2f2" (UID: "98275e84-2e20-46db-8264-27ec2ac2e2f2"). InnerVolumeSpecName "kube-api-access-86cmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:32.050961 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.050943 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-istio-token" pod "fc3ed9fa-b7f0-418f-8ac9-422f9e765027" (UID: "fc3ed9fa-b7f0-418f-8ac9-422f9e765027"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:32.051088 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.051068 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/419aecf7-a250-42ad-995b-e7ebd9630a76-istio-token" pod "419aecf7-a250-42ad-995b-e7ebd9630a76" (UID: "419aecf7-a250-42ad-995b-e7ebd9630a76"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:32.051159 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.051116 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0c9594e-8534-4538-a4bc-2d40d918e362-kube-api-access-mqb82" pod "a0c9594e-8534-4538-a4bc-2d40d918e362" (UID: "a0c9594e-8534-4538-a4bc-2d40d918e362"). InnerVolumeSpecName "kube-api-access-mqb82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:32.060564 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.060530 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0c9594e-8534-4538-a4bc-2d40d918e362-istio-token" pod "a0c9594e-8534-4538-a4bc-2d40d918e362" (UID: "a0c9594e-8534-4538-a4bc-2d40d918e362"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:32.064721 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.064632 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" Oct 30 20:08:32.067254 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.067224 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b55ec6e2-9392-4d8d-8340-ef12f674970c-istio-token" pod "b55ec6e2-9392-4d8d-8340-ef12f674970c" (UID: "b55ec6e2-9392-4d8d-8340-ef12f674970c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:32.137180 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.137139 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff410c2e-9048-4e5c-965b-1261336561bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff410c2e-9048-4e5c-965b-1261336561bb-workload-certs\") pod \"ff410c2e-9048-4e5c-965b-1261336561bb\" (UID: \"ff410c2e-9048-4e5c-965b-1261336561bb\") " Oct 30 20:08:32.137293 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.137204 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff410c2e-9048-4e5c-965b-1261336561bb-istio-token\" (UniqueName: \"kubernetes.io/projected/ff410c2e-9048-4e5c-965b-1261336561bb-istio-token\") pod \"ff410c2e-9048-4e5c-965b-1261336561bb\" (UID: \"ff410c2e-9048-4e5c-965b-1261336561bb\") " Oct 30 20:08:32.137293 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.137235 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff410c2e-9048-4e5c-965b-1261336561bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff410c2e-9048-4e5c-965b-1261336561bb-istio-data\") pod \"ff410c2e-9048-4e5c-965b-1261336561bb\" (UID: \"ff410c2e-9048-4e5c-965b-1261336561bb\") " Oct 30 20:08:32.137416 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.137298 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff410c2e-9048-4e5c-965b-1261336561bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff410c2e-9048-4e5c-965b-1261336561bb-istiod-ca-cert\") pod \"ff410c2e-9048-4e5c-965b-1261336561bb\" (UID: \"ff410c2e-9048-4e5c-965b-1261336561bb\") " Oct 30 20:08:32.137416 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.137345 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff410c2e-9048-4e5c-965b-1261336561bb-kube-api-access-qckvd\" (UniqueName: \"kubernetes.io/projected/ff410c2e-9048-4e5c-965b-1261336561bb-kube-api-access-qckvd\") pod \"ff410c2e-9048-4e5c-965b-1261336561bb\" (UID: \"ff410c2e-9048-4e5c-965b-1261336561bb\") " Oct 30 20:08:32.137416 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.137383 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff410c2e-9048-4e5c-965b-1261336561bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff410c2e-9048-4e5c-965b-1261336561bb-istio-envoy\") pod \"ff410c2e-9048-4e5c-965b-1261336561bb\" (UID: \"ff410c2e-9048-4e5c-965b-1261336561bb\") " Oct 30 20:08:32.138443 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.137477 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff410c2e-9048-4e5c-965b-1261336561bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff410c2e-9048-4e5c-965b-1261336561bb-credential-socket\") pod \"ff410c2e-9048-4e5c-965b-1261336561bb\" (UID: \"ff410c2e-9048-4e5c-965b-1261336561bb\") " Oct 30 20:08:32.138443 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.137652 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff410c2e-9048-4e5c-965b-1261336561bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff410c2e-9048-4e5c-965b-1261336561bb-istio-podinfo\") pod \"ff410c2e-9048-4e5c-965b-1261336561bb\" (UID: \"ff410c2e-9048-4e5c-965b-1261336561bb\") " Oct 30 20:08:32.138443 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.137733 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff410c2e-9048-4e5c-965b-1261336561bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff410c2e-9048-4e5c-965b-1261336561bb-workload-socket\") pod \"ff410c2e-9048-4e5c-965b-1261336561bb\" (UID: \"ff410c2e-9048-4e5c-965b-1261336561bb\") " Oct 30 20:08:32.138443 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.138366 5088 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98275e84-2e20-46db-8264-27ec2ac2e2f2-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.138443 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.138389 5088 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98275e84-2e20-46db-8264-27ec2ac2e2f2-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.138443 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.138404 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj2vj\" (UniqueName: \"kubernetes.io/projected/419aecf7-a250-42ad-995b-e7ebd9630a76-kube-api-access-xj2vj\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.138443 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.138417 5088 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/419aecf7-a250-42ad-995b-e7ebd9630a76-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.138443 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.138430 5088 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/419aecf7-a250-42ad-995b-e7ebd9630a76-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.138443 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.138442 5088 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b55ec6e2-9392-4d8d-8340-ef12f674970c-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.138981 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.138456 5088 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98275e84-2e20-46db-8264-27ec2ac2e2f2-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.138981 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.138469 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqb82\" (UniqueName: \"kubernetes.io/projected/a0c9594e-8534-4538-a4bc-2d40d918e362-kube-api-access-mqb82\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.138981 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.138482 5088 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/419aecf7-a250-42ad-995b-e7ebd9630a76-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.138981 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.138496 5088 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b55ec6e2-9392-4d8d-8340-ef12f674970c-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.138981 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.138509 5088 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0c9594e-8534-4538-a4bc-2d40d918e362-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.138981 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.138525 5088 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.138981 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.138540 5088 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0c9594e-8534-4538-a4bc-2d40d918e362-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.138981 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.138554 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp5fz\" (UniqueName: \"kubernetes.io/projected/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-kube-api-access-mp5fz\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.138981 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.138567 5088 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.138981 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.138580 5088 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98275e84-2e20-46db-8264-27ec2ac2e2f2-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.138981 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.138593 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86cmb\" (UniqueName: \"kubernetes.io/projected/98275e84-2e20-46db-8264-27ec2ac2e2f2-kube-api-access-86cmb\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.138981 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.138606 5088 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0c9594e-8534-4538-a4bc-2d40d918e362-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.139569 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.139000 5088 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.139569 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.139017 5088 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98275e84-2e20-46db-8264-27ec2ac2e2f2-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.139569 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.139032 5088 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0c9594e-8534-4538-a4bc-2d40d918e362-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.139569 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.139047 5088 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/419aecf7-a250-42ad-995b-e7ebd9630a76-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.139569 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.139059 5088 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.139569 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.139072 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xtt5\" (UniqueName: \"kubernetes.io/projected/b55ec6e2-9392-4d8d-8340-ef12f674970c-kube-api-access-5xtt5\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.139569 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.139086 5088 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b55ec6e2-9392-4d8d-8340-ef12f674970c-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.139569 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.139100 5088 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0c9594e-8534-4538-a4bc-2d40d918e362-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.139569 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.139114 5088 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b55ec6e2-9392-4d8d-8340-ef12f674970c-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.139569 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.139127 5088 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/419aecf7-a250-42ad-995b-e7ebd9630a76-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.139569 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.139381 5088 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/419aecf7-a250-42ad-995b-e7ebd9630a76-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.139569 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.139403 5088 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b55ec6e2-9392-4d8d-8340-ef12f674970c-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.139569 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.139403 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff410c2e-9048-4e5c-965b-1261336561bb-workload-certs" pod "ff410c2e-9048-4e5c-965b-1261336561bb" (UID: "ff410c2e-9048-4e5c-965b-1261336561bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.139569 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.139417 5088 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.139569 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.139435 5088 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b55ec6e2-9392-4d8d-8340-ef12f674970c-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.139569 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.139459 5088 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc3ed9fa-b7f0-418f-8ac9-422f9e765027-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.140265 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.140016 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff410c2e-9048-4e5c-965b-1261336561bb-credential-socket" pod "ff410c2e-9048-4e5c-965b-1261336561bb" (UID: "ff410c2e-9048-4e5c-965b-1261336561bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.140265 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.140202 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff410c2e-9048-4e5c-965b-1261336561bb-istio-data" pod "ff410c2e-9048-4e5c-965b-1261336561bb" (UID: "ff410c2e-9048-4e5c-965b-1261336561bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.140265 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.140251 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff410c2e-9048-4e5c-965b-1261336561bb-workload-socket" pod "ff410c2e-9048-4e5c-965b-1261336561bb" (UID: "ff410c2e-9048-4e5c-965b-1261336561bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.140506 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.140481 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff410c2e-9048-4e5c-965b-1261336561bb-istiod-ca-cert" pod "ff410c2e-9048-4e5c-965b-1261336561bb" (UID: "ff410c2e-9048-4e5c-965b-1261336561bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:32.149446 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.149412 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff410c2e-9048-4e5c-965b-1261336561bb-kube-api-access-qckvd" pod "ff410c2e-9048-4e5c-965b-1261336561bb" (UID: "ff410c2e-9048-4e5c-965b-1261336561bb"). InnerVolumeSpecName "kube-api-access-qckvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:32.153512 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.153426 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcafb519-gmmj2" Oct 30 20:08:32.154112 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.154057 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff410c2e-9048-4e5c-965b-1261336561bb-istio-envoy" pod "ff410c2e-9048-4e5c-965b-1261336561bb" (UID: "ff410c2e-9048-4e5c-965b-1261336561bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.155188 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.155129 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff410c2e-9048-4e5c-965b-1261336561bb-istio-podinfo" pod "ff410c2e-9048-4e5c-965b-1261336561bb" (UID: "ff410c2e-9048-4e5c-965b-1261336561bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:08:32.156171 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.156141 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff410c2e-9048-4e5c-965b-1261336561bb-istio-token" pod "ff410c2e-9048-4e5c-965b-1261336561bb" (UID: "ff410c2e-9048-4e5c-965b-1261336561bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:32.157500 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.157472 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" Oct 30 20:08:32.179301 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.179257 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" Oct 30 20:08:32.240953 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.240918 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d209c20c-000f-4cc4-9792-5a71eeb0a773-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d209c20c-000f-4cc4-9792-5a71eeb0a773-istio-envoy\") pod \"d209c20c-000f-4cc4-9792-5a71eeb0a773\" (UID: \"d209c20c-000f-4cc4-9792-5a71eeb0a773\") " Oct 30 20:08:32.241162 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.240969 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/442a352a-40cd-4152-97a5-88a598908ac1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/442a352a-40cd-4152-97a5-88a598908ac1-maas-api-xks-ca\") pod \"442a352a-40cd-4152-97a5-88a598908ac1\" (UID: \"442a352a-40cd-4152-97a5-88a598908ac1\") " Oct 30 20:08:32.241162 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.241047 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-istiod-ca-cert\") pod \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\" (UID: \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\") " Oct 30 20:08:32.241162 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.241078 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-workload-socket\") pod \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\" (UID: \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\") " Oct 30 20:08:32.241162 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.241111 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d209c20c-000f-4cc4-9792-5a71eeb0a773-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d209c20c-000f-4cc4-9792-5a71eeb0a773-istio-podinfo\") pod \"d209c20c-000f-4cc4-9792-5a71eeb0a773\" (UID: \"d209c20c-000f-4cc4-9792-5a71eeb0a773\") " Oct 30 20:08:32.241405 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.241178 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d209c20c-000f-4cc4-9792-5a71eeb0a773-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d209c20c-000f-4cc4-9792-5a71eeb0a773-workload-certs\") pod \"d209c20c-000f-4cc4-9792-5a71eeb0a773\" (UID: \"d209c20c-000f-4cc4-9792-5a71eeb0a773\") " Oct 30 20:08:32.241405 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.241219 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d209c20c-000f-4cc4-9792-5a71eeb0a773-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d209c20c-000f-4cc4-9792-5a71eeb0a773-credential-socket\") pod \"d209c20c-000f-4cc4-9792-5a71eeb0a773\" (UID: \"d209c20c-000f-4cc4-9792-5a71eeb0a773\") " Oct 30 20:08:32.241405 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.241248 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d209c20c-000f-4cc4-9792-5a71eeb0a773-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d209c20c-000f-4cc4-9792-5a71eeb0a773-istiod-ca-cert\") pod \"d209c20c-000f-4cc4-9792-5a71eeb0a773\" (UID: \"d209c20c-000f-4cc4-9792-5a71eeb0a773\") " Oct 30 20:08:32.241405 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.241280 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d209c20c-000f-4cc4-9792-5a71eeb0a773-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d209c20c-000f-4cc4-9792-5a71eeb0a773-istio-data\") pod \"d209c20c-000f-4cc4-9792-5a71eeb0a773\" (UID: \"d209c20c-000f-4cc4-9792-5a71eeb0a773\") " Oct 30 20:08:32.241405 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.241310 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/442a352a-40cd-4152-97a5-88a598908ac1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/442a352a-40cd-4152-97a5-88a598908ac1-maas-api-openshift-ca\") pod \"442a352a-40cd-4152-97a5-88a598908ac1\" (UID: \"442a352a-40cd-4152-97a5-88a598908ac1\") " Oct 30 20:08:32.241405 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.241367 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-istio-podinfo\") pod \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\" (UID: \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\") " Oct 30 20:08:32.241715 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.241462 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-credential-socket\") pod \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\" (UID: \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\") " Oct 30 20:08:32.241715 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.241498 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d209c20c-000f-4cc4-9792-5a71eeb0a773-istio-token\" (UniqueName: \"kubernetes.io/projected/d209c20c-000f-4cc4-9792-5a71eeb0a773-istio-token\") pod \"d209c20c-000f-4cc4-9792-5a71eeb0a773\" (UID: \"d209c20c-000f-4cc4-9792-5a71eeb0a773\") " Oct 30 20:08:32.241715 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.241567 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-workload-certs\") pod \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\" (UID: \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\") " Oct 30 20:08:32.241715 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.241667 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/442a352a-40cd-4152-97a5-88a598908ac1-kube-api-access-g79vg\" (UniqueName: \"kubernetes.io/projected/442a352a-40cd-4152-97a5-88a598908ac1-kube-api-access-g79vg\") pod \"442a352a-40cd-4152-97a5-88a598908ac1\" (UID: \"442a352a-40cd-4152-97a5-88a598908ac1\") " Oct 30 20:08:32.241925 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.241733 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d209c20c-000f-4cc4-9792-5a71eeb0a773-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d209c20c-000f-4cc4-9792-5a71eeb0a773-workload-socket\") pod \"d209c20c-000f-4cc4-9792-5a71eeb0a773\" (UID: \"d209c20c-000f-4cc4-9792-5a71eeb0a773\") " Oct 30 20:08:32.241925 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.241844 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-istio-token\" (UniqueName: \"kubernetes.io/projected/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-istio-token\") pod \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\" (UID: \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\") " Oct 30 20:08:32.241925 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.241880 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-istio-envoy\") pod \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\" (UID: \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\") " Oct 30 20:08:32.242083 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.241940 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-istio-data\") pod \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\" (UID: \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\") " Oct 30 20:08:32.242083 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.242051 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-kube-api-access-8zp8v\" (UniqueName: \"kubernetes.io/projected/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-kube-api-access-8zp8v\") pod \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\" (UID: \"ba76bfa7-6ba7-4862-8348-ca7c17487b9f\") " Oct 30 20:08:32.242195 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.242171 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d209c20c-000f-4cc4-9792-5a71eeb0a773-kube-api-access-mrm5m\" (UniqueName: \"kubernetes.io/projected/d209c20c-000f-4cc4-9792-5a71eeb0a773-kube-api-access-mrm5m\") pod \"d209c20c-000f-4cc4-9792-5a71eeb0a773\" (UID: \"d209c20c-000f-4cc4-9792-5a71eeb0a773\") " Oct 30 20:08:32.243212 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.242858 5088 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff410c2e-9048-4e5c-965b-1261336561bb-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.243212 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.242886 5088 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff410c2e-9048-4e5c-965b-1261336561bb-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.243212 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.242902 5088 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff410c2e-9048-4e5c-965b-1261336561bb-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.243212 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.242916 5088 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff410c2e-9048-4e5c-965b-1261336561bb-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.243212 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.242928 5088 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff410c2e-9048-4e5c-965b-1261336561bb-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.243212 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.242940 5088 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff410c2e-9048-4e5c-965b-1261336561bb-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.243212 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.242933 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d209c20c-000f-4cc4-9792-5a71eeb0a773-workload-socket" pod "d209c20c-000f-4cc4-9792-5a71eeb0a773" (UID: "d209c20c-000f-4cc4-9792-5a71eeb0a773"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.243212 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.242956 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qckvd\" (UniqueName: \"kubernetes.io/projected/ff410c2e-9048-4e5c-965b-1261336561bb-kube-api-access-qckvd\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.243212 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.242970 5088 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff410c2e-9048-4e5c-965b-1261336561bb-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.243212 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.242984 5088 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff410c2e-9048-4e5c-965b-1261336561bb-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.243799 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.243270 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-credential-socket" pod "ba76bfa7-6ba7-4862-8348-ca7c17487b9f" (UID: "ba76bfa7-6ba7-4862-8348-ca7c17487b9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.244442 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.244339 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d209c20c-000f-4cc4-9792-5a71eeb0a773-workload-certs" pod "d209c20c-000f-4cc4-9792-5a71eeb0a773" (UID: "d209c20c-000f-4cc4-9792-5a71eeb0a773"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.244827 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.244727 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-workload-socket" pod "ba76bfa7-6ba7-4862-8348-ca7c17487b9f" (UID: "ba76bfa7-6ba7-4862-8348-ca7c17487b9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.245149 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.244995 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-workload-certs" pod "ba76bfa7-6ba7-4862-8348-ca7c17487b9f" (UID: "ba76bfa7-6ba7-4862-8348-ca7c17487b9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.245920 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.245604 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d209c20c-000f-4cc4-9792-5a71eeb0a773-credential-socket" pod "d209c20c-000f-4cc4-9792-5a71eeb0a773" (UID: "d209c20c-000f-4cc4-9792-5a71eeb0a773"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.245920 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.245648 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d209c20c-000f-4cc4-9792-5a71eeb0a773-istio-data" pod "d209c20c-000f-4cc4-9792-5a71eeb0a773" (UID: "d209c20c-000f-4cc4-9792-5a71eeb0a773"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.245920 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.245812 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-istiod-ca-cert" pod "ba76bfa7-6ba7-4862-8348-ca7c17487b9f" (UID: "ba76bfa7-6ba7-4862-8348-ca7c17487b9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:32.245920 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.245895 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d209c20c-000f-4cc4-9792-5a71eeb0a773-istiod-ca-cert" pod "d209c20c-000f-4cc4-9792-5a71eeb0a773" (UID: "d209c20c-000f-4cc4-9792-5a71eeb0a773"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:32.246850 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.246275 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-istio-data" pod "ba76bfa7-6ba7-4862-8348-ca7c17487b9f" (UID: "ba76bfa7-6ba7-4862-8348-ca7c17487b9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.246850 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.246433 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/442a352a-40cd-4152-97a5-88a598908ac1-maas-api-openshift-ca" pod "442a352a-40cd-4152-97a5-88a598908ac1" (UID: "442a352a-40cd-4152-97a5-88a598908ac1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:32.246850 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.246723 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d209c20c-000f-4cc4-9792-5a71eeb0a773-istio-envoy" pod "d209c20c-000f-4cc4-9792-5a71eeb0a773" (UID: "d209c20c-000f-4cc4-9792-5a71eeb0a773"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.251086 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.250989 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-istio-token" pod "ba76bfa7-6ba7-4862-8348-ca7c17487b9f" (UID: "ba76bfa7-6ba7-4862-8348-ca7c17487b9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:32.251086 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.251051 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d209c20c-000f-4cc4-9792-5a71eeb0a773-istio-podinfo" pod "d209c20c-000f-4cc4-9792-5a71eeb0a773" (UID: "d209c20c-000f-4cc4-9792-5a71eeb0a773"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:08:32.251359 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.251335 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/442a352a-40cd-4152-97a5-88a598908ac1-kube-api-access-g79vg" pod "442a352a-40cd-4152-97a5-88a598908ac1" (UID: "442a352a-40cd-4152-97a5-88a598908ac1"). InnerVolumeSpecName "kube-api-access-g79vg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:32.251448 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.251382 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d209c20c-000f-4cc4-9792-5a71eeb0a773-kube-api-access-mrm5m" pod "d209c20c-000f-4cc4-9792-5a71eeb0a773" (UID: "d209c20c-000f-4cc4-9792-5a71eeb0a773"). InnerVolumeSpecName "kube-api-access-mrm5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:32.251448 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.251420 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d209c20c-000f-4cc4-9792-5a71eeb0a773-istio-token" pod "d209c20c-000f-4cc4-9792-5a71eeb0a773" (UID: "d209c20c-000f-4cc4-9792-5a71eeb0a773"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:32.251556 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.251513 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-istio-podinfo" pod "ba76bfa7-6ba7-4862-8348-ca7c17487b9f" (UID: "ba76bfa7-6ba7-4862-8348-ca7c17487b9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:08:32.251830 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.251782 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-istio-envoy" pod "ba76bfa7-6ba7-4862-8348-ca7c17487b9f" (UID: "ba76bfa7-6ba7-4862-8348-ca7c17487b9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.252054 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.252033 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/442a352a-40cd-4152-97a5-88a598908ac1-maas-api-xks-ca" pod "442a352a-40cd-4152-97a5-88a598908ac1" (UID: "442a352a-40cd-4152-97a5-88a598908ac1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:08:32.253484 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.253455 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-kube-api-access-8zp8v" pod "ba76bfa7-6ba7-4862-8348-ca7c17487b9f" (UID: "ba76bfa7-6ba7-4862-8348-ca7c17487b9f"). InnerVolumeSpecName "kube-api-access-8zp8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:32.344859 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.344803 5088 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.344859 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.344834 5088 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.344859 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.344849 5088 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d209c20c-000f-4cc4-9792-5a71eeb0a773-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.345048 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.344863 5088 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d209c20c-000f-4cc4-9792-5a71eeb0a773-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.345048 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.344879 5088 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d209c20c-000f-4cc4-9792-5a71eeb0a773-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.345048 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.344893 5088 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d209c20c-000f-4cc4-9792-5a71eeb0a773-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.345048 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.344906 5088 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d209c20c-000f-4cc4-9792-5a71eeb0a773-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.345048 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.344920 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/442a352a-40cd-4152-97a5-88a598908ac1-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.345048 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.344934 5088 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.345048 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.344948 5088 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.345048 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.344961 5088 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d209c20c-000f-4cc4-9792-5a71eeb0a773-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.345048 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.344975 5088 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.345048 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.344987 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g79vg\" (UniqueName: \"kubernetes.io/projected/442a352a-40cd-4152-97a5-88a598908ac1-kube-api-access-g79vg\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.345048 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.345001 5088 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d209c20c-000f-4cc4-9792-5a71eeb0a773-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.345048 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.345014 5088 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.345048 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.345035 5088 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.345048 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.345047 5088 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.345483 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.345059 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zp8v\" (UniqueName: \"kubernetes.io/projected/ba76bfa7-6ba7-4862-8348-ca7c17487b9f-kube-api-access-8zp8v\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.345483 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.345072 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrm5m\" (UniqueName: \"kubernetes.io/projected/d209c20c-000f-4cc4-9792-5a71eeb0a773-kube-api-access-mrm5m\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.345483 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.345087 5088 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d209c20c-000f-4cc4-9792-5a71eeb0a773-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.345483 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.345099 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/442a352a-40cd-4152-97a5-88a598908ac1-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.414668 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.414640 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" Oct 30 20:08:32.547671 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.547635 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d44744f7-8f4c-4c79-978b-52649ce5a5ed-istio-token\" (UniqueName: \"kubernetes.io/projected/d44744f7-8f4c-4c79-978b-52649ce5a5ed-istio-token\") pod \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\" (UID: \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\") " Oct 30 20:08:32.547828 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.547748 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d44744f7-8f4c-4c79-978b-52649ce5a5ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d44744f7-8f4c-4c79-978b-52649ce5a5ed-workload-certs\") pod \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\" (UID: \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\") " Oct 30 20:08:32.547828 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.547785 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d44744f7-8f4c-4c79-978b-52649ce5a5ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d44744f7-8f4c-4c79-978b-52649ce5a5ed-istio-envoy\") pod \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\" (UID: \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\") " Oct 30 20:08:32.547949 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.547829 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d44744f7-8f4c-4c79-978b-52649ce5a5ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d44744f7-8f4c-4c79-978b-52649ce5a5ed-credential-socket\") pod \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\" (UID: \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\") " Oct 30 20:08:32.547949 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.547908 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d44744f7-8f4c-4c79-978b-52649ce5a5ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d44744f7-8f4c-4c79-978b-52649ce5a5ed-istio-data\") pod \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\" (UID: \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\") " Oct 30 20:08:32.548062 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.548046 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d44744f7-8f4c-4c79-978b-52649ce5a5ed-kube-api-access-hxt7j\" (UniqueName: \"kubernetes.io/projected/d44744f7-8f4c-4c79-978b-52649ce5a5ed-kube-api-access-hxt7j\") pod \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\" (UID: \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\") " Oct 30 20:08:32.548112 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.548067 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d44744f7-8f4c-4c79-978b-52649ce5a5ed-workload-certs" pod "d44744f7-8f4c-4c79-978b-52649ce5a5ed" (UID: "d44744f7-8f4c-4c79-978b-52649ce5a5ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.548112 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.548078 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d44744f7-8f4c-4c79-978b-52649ce5a5ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d44744f7-8f4c-4c79-978b-52649ce5a5ed-istiod-ca-cert\") pod \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\" (UID: \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\") " Oct 30 20:08:32.549087 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.548152 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d44744f7-8f4c-4c79-978b-52649ce5a5ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d44744f7-8f4c-4c79-978b-52649ce5a5ed-workload-socket\") pod \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\" (UID: \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\") " Oct 30 20:08:32.549087 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.548232 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d44744f7-8f4c-4c79-978b-52649ce5a5ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d44744f7-8f4c-4c79-978b-52649ce5a5ed-istio-podinfo\") pod \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\" (UID: \"d44744f7-8f4c-4c79-978b-52649ce5a5ed\") " Oct 30 20:08:32.549087 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.548388 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d44744f7-8f4c-4c79-978b-52649ce5a5ed-istio-data" pod "d44744f7-8f4c-4c79-978b-52649ce5a5ed" (UID: "d44744f7-8f4c-4c79-978b-52649ce5a5ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.549087 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.548411 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d44744f7-8f4c-4c79-978b-52649ce5a5ed-istiod-ca-cert" pod "d44744f7-8f4c-4c79-978b-52649ce5a5ed" (UID: "d44744f7-8f4c-4c79-978b-52649ce5a5ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:32.549087 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.548546 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d44744f7-8f4c-4c79-978b-52649ce5a5ed-workload-socket" pod "d44744f7-8f4c-4c79-978b-52649ce5a5ed" (UID: "d44744f7-8f4c-4c79-978b-52649ce5a5ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.549087 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.548858 5088 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d44744f7-8f4c-4c79-978b-52649ce5a5ed-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.549087 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.548887 5088 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d44744f7-8f4c-4c79-978b-52649ce5a5ed-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.549087 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.548901 5088 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d44744f7-8f4c-4c79-978b-52649ce5a5ed-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.549087 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.548916 5088 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d44744f7-8f4c-4c79-978b-52649ce5a5ed-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.549087 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.548981 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d44744f7-8f4c-4c79-978b-52649ce5a5ed-credential-socket" pod "d44744f7-8f4c-4c79-978b-52649ce5a5ed" (UID: "d44744f7-8f4c-4c79-978b-52649ce5a5ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.550446 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.550425 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d44744f7-8f4c-4c79-978b-52649ce5a5ed-istio-token" pod "d44744f7-8f4c-4c79-978b-52649ce5a5ed" (UID: "d44744f7-8f4c-4c79-978b-52649ce5a5ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:32.550994 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.550977 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d44744f7-8f4c-4c79-978b-52649ce5a5ed-istio-envoy" pod "d44744f7-8f4c-4c79-978b-52649ce5a5ed" (UID: "d44744f7-8f4c-4c79-978b-52649ce5a5ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.551075 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.550995 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d44744f7-8f4c-4c79-978b-52649ce5a5ed-kube-api-access-hxt7j" pod "d44744f7-8f4c-4c79-978b-52649ce5a5ed" (UID: "d44744f7-8f4c-4c79-978b-52649ce5a5ed"). InnerVolumeSpecName "kube-api-access-hxt7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:32.551377 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.551363 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d44744f7-8f4c-4c79-978b-52649ce5a5ed-istio-podinfo" pod "d44744f7-8f4c-4c79-978b-52649ce5a5ed" (UID: "d44744f7-8f4c-4c79-978b-52649ce5a5ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:08:32.585231 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.585203 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" Oct 30 20:08:32.649741 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.649705 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00264202-1595-4616-8796-2ecd7415c577-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00264202-1595-4616-8796-2ecd7415c577-istio-podinfo\") pod \"00264202-1595-4616-8796-2ecd7415c577\" (UID: \"00264202-1595-4616-8796-2ecd7415c577\") " Oct 30 20:08:32.649904 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.649849 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00264202-1595-4616-8796-2ecd7415c577-kube-api-access-gn6rq\" (UniqueName: \"kubernetes.io/projected/00264202-1595-4616-8796-2ecd7415c577-kube-api-access-gn6rq\") pod \"00264202-1595-4616-8796-2ecd7415c577\" (UID: \"00264202-1595-4616-8796-2ecd7415c577\") " Oct 30 20:08:32.649904 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.649896 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00264202-1595-4616-8796-2ecd7415c577-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00264202-1595-4616-8796-2ecd7415c577-istio-data\") pod \"00264202-1595-4616-8796-2ecd7415c577\" (UID: \"00264202-1595-4616-8796-2ecd7415c577\") " Oct 30 20:08:32.650101 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.649926 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00264202-1595-4616-8796-2ecd7415c577-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00264202-1595-4616-8796-2ecd7415c577-credential-socket\") pod \"00264202-1595-4616-8796-2ecd7415c577\" (UID: \"00264202-1595-4616-8796-2ecd7415c577\") " Oct 30 20:08:32.650101 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.649980 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00264202-1595-4616-8796-2ecd7415c577-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00264202-1595-4616-8796-2ecd7415c577-istio-envoy\") pod \"00264202-1595-4616-8796-2ecd7415c577\" (UID: \"00264202-1595-4616-8796-2ecd7415c577\") " Oct 30 20:08:32.650101 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.650024 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00264202-1595-4616-8796-2ecd7415c577-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00264202-1595-4616-8796-2ecd7415c577-istiod-ca-cert\") pod \"00264202-1595-4616-8796-2ecd7415c577\" (UID: \"00264202-1595-4616-8796-2ecd7415c577\") " Oct 30 20:08:32.650256 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.650127 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00264202-1595-4616-8796-2ecd7415c577-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00264202-1595-4616-8796-2ecd7415c577-workload-socket\") pod \"00264202-1595-4616-8796-2ecd7415c577\" (UID: \"00264202-1595-4616-8796-2ecd7415c577\") " Oct 30 20:08:32.650256 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.650159 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00264202-1595-4616-8796-2ecd7415c577-istio-token\" (UniqueName: \"kubernetes.io/projected/00264202-1595-4616-8796-2ecd7415c577-istio-token\") pod \"00264202-1595-4616-8796-2ecd7415c577\" (UID: \"00264202-1595-4616-8796-2ecd7415c577\") " Oct 30 20:08:32.650377 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.650287 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00264202-1595-4616-8796-2ecd7415c577-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00264202-1595-4616-8796-2ecd7415c577-workload-certs\") pod \"00264202-1595-4616-8796-2ecd7415c577\" (UID: \"00264202-1595-4616-8796-2ecd7415c577\") " Oct 30 20:08:32.650377 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.650299 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00264202-1595-4616-8796-2ecd7415c577-istio-data" pod "00264202-1595-4616-8796-2ecd7415c577" (UID: "00264202-1595-4616-8796-2ecd7415c577"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.650522 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.650495 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00264202-1595-4616-8796-2ecd7415c577-istiod-ca-cert" pod "00264202-1595-4616-8796-2ecd7415c577" (UID: "00264202-1595-4616-8796-2ecd7415c577"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:32.650872 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.650768 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00264202-1595-4616-8796-2ecd7415c577-workload-socket" pod "00264202-1595-4616-8796-2ecd7415c577" (UID: "00264202-1595-4616-8796-2ecd7415c577"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.650872 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.650788 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00264202-1595-4616-8796-2ecd7415c577-workload-certs" pod "00264202-1595-4616-8796-2ecd7415c577" (UID: "00264202-1595-4616-8796-2ecd7415c577"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.650872 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.650840 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00264202-1595-4616-8796-2ecd7415c577-credential-socket" pod "00264202-1595-4616-8796-2ecd7415c577" (UID: "00264202-1595-4616-8796-2ecd7415c577"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.651191 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.650959 5088 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00264202-1595-4616-8796-2ecd7415c577-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.651191 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.650978 5088 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00264202-1595-4616-8796-2ecd7415c577-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.651191 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.650992 5088 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00264202-1595-4616-8796-2ecd7415c577-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.651191 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.651010 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxt7j\" (UniqueName: \"kubernetes.io/projected/d44744f7-8f4c-4c79-978b-52649ce5a5ed-kube-api-access-hxt7j\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.651191 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.651026 5088 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d44744f7-8f4c-4c79-978b-52649ce5a5ed-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.651191 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.651039 5088 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00264202-1595-4616-8796-2ecd7415c577-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.651191 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.651053 5088 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d44744f7-8f4c-4c79-978b-52649ce5a5ed-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.651191 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.651067 5088 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00264202-1595-4616-8796-2ecd7415c577-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.651191 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.651079 5088 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d44744f7-8f4c-4c79-978b-52649ce5a5ed-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.651191 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.651092 5088 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d44744f7-8f4c-4c79-978b-52649ce5a5ed-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.652714 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.652687 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00264202-1595-4616-8796-2ecd7415c577-kube-api-access-gn6rq" pod "00264202-1595-4616-8796-2ecd7415c577" (UID: "00264202-1595-4616-8796-2ecd7415c577"). InnerVolumeSpecName "kube-api-access-gn6rq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:32.652798 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.652747 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00264202-1595-4616-8796-2ecd7415c577-istio-envoy" pod "00264202-1595-4616-8796-2ecd7415c577" (UID: "00264202-1595-4616-8796-2ecd7415c577"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:32.652798 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.652769 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00264202-1595-4616-8796-2ecd7415c577-istio-podinfo" pod "00264202-1595-4616-8796-2ecd7415c577" (UID: "00264202-1595-4616-8796-2ecd7415c577"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:08:32.653020 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.652998 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00264202-1595-4616-8796-2ecd7415c577-istio-token" pod "00264202-1595-4616-8796-2ecd7415c577" (UID: "00264202-1595-4616-8796-2ecd7415c577"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:32.752986 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.752948 5088 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00264202-1595-4616-8796-2ecd7415c577-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.753252 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.753224 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn6rq\" (UniqueName: \"kubernetes.io/projected/00264202-1595-4616-8796-2ecd7415c577-kube-api-access-gn6rq\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.753252 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.753254 5088 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00264202-1595-4616-8796-2ecd7415c577-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.753386 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.753269 5088 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00264202-1595-4616-8796-2ecd7415c577-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:32.881742 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.881659 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" Oct 30 20:08:32.882180 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.881655 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d" event={"ID":"d44744f7-8f4c-4c79-978b-52649ce5a5ed","Type":"ContainerDied","Data":"1778982336a81f990b393dcaec7f58cea3668b8d23b07095d2c9a546bbeb8462"} Oct 30 20:08:32.882180 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.881789 5088 scope.go:122] "RemoveContainer" containerID="314ac6568ace40ce03f7bf80c6852789d144db290701be78a5ba82f0f10dc195" Oct 30 20:08:32.883090 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.883069 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcafb519-gmmj2" Oct 30 20:08:32.883198 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.883092 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcafb519-gmmj2" event={"ID":"442a352a-40cd-4152-97a5-88a598908ac1","Type":"ContainerDied","Data":"3527ad9fad34b82c55d3221cb5d8d3e13594207b8578be727e728c849ff85d70"} Oct 30 20:08:32.883198 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.883143 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3527ad9fad34b82c55d3221cb5d8d3e13594207b8578be727e728c849ff85d70" Oct 30 20:08:32.884747 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.884726 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" Oct 30 20:08:32.884860 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.884745 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s" event={"ID":"ff410c2e-9048-4e5c-965b-1261336561bb","Type":"ContainerDied","Data":"00c2f8bdf4d598a3c83d6f14f27706d3fb1b1403ec1042fe9186bedd748cb828"} Oct 30 20:08:32.886334 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.886314 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" Oct 30 20:08:32.886451 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.886340 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp" event={"ID":"d209c20c-000f-4cc4-9792-5a71eeb0a773","Type":"ContainerDied","Data":"05dee448ce51c2432d099153b6e1e14a8e6e397ee80ba7b921cd66dc3ced8cfb"} Oct 30 20:08:32.887913 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.887887 5088 generic.go:356] "Generic (PLEG): container finished" podID="ba76bfa7-6ba7-4862-8348-ca7c17487b9f" containerID="f54a4ec28ad66ba296a4360924c66536ec38efc33f4ff1e36c3bb003b0ba44ed" exitCode=0 Oct 30 20:08:32.888006 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.887937 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" Oct 30 20:08:32.888006 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.887969 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" event={"ID":"ba76bfa7-6ba7-4862-8348-ca7c17487b9f","Type":"ContainerDied","Data":"f54a4ec28ad66ba296a4360924c66536ec38efc33f4ff1e36c3bb003b0ba44ed"} Oct 30 20:08:32.888117 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.888016 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr" event={"ID":"ba76bfa7-6ba7-4862-8348-ca7c17487b9f","Type":"ContainerDied","Data":"3e810d14352a4a960f9a277b8f09fb5d57f7da797a280bc33e507e10764f5abf"} Oct 30 20:08:32.889589 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.889571 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" event={"ID":"b55ec6e2-9392-4d8d-8340-ef12f674970c","Type":"ContainerDied","Data":"0641400eeee93afd1646bb6f071446950ef57acbdd689d01a78b782efc03853a"} Oct 30 20:08:32.889589 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.889584 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj" Oct 30 20:08:32.891373 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.891348 5088 generic.go:356] "Generic (PLEG): container finished" podID="00264202-1595-4616-8796-2ecd7415c577" containerID="5e32dd83a9a502d3feb8fbe6ae10467c0acbc30fa91cd27137549b4abbe249df" exitCode=0 Oct 30 20:08:32.891463 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.891407 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" event={"ID":"00264202-1595-4616-8796-2ecd7415c577","Type":"ContainerDied","Data":"5e32dd83a9a502d3feb8fbe6ae10467c0acbc30fa91cd27137549b4abbe249df"} Oct 30 20:08:32.891546 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.891460 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" event={"ID":"00264202-1595-4616-8796-2ecd7415c577","Type":"ContainerDied","Data":"42eac92ea231f22e879fdbca88921a9ec3d6af992c0704e4cdb7ee6ccd57c3af"} Oct 30 20:08:32.891546 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.891470 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg" Oct 30 20:08:32.892047 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.891987 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r" Oct 30 20:08:32.892160 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.892131 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s" Oct 30 20:08:32.892309 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.892280 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2" Oct 30 20:08:32.892535 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.892408 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn" Oct 30 20:08:32.894975 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.894958 5088 scope.go:122] "RemoveContainer" containerID="6cfacb45b1d70e6292533f4fefcaba52f16eaac85d2922a2b8acf5e5fb5791d1" Oct 30 20:08:32.907145 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.907101 5088 scope.go:122] "RemoveContainer" containerID="1619335ad2605965c2c5a4f2c72a93cf39d2404688f2c83428c6b8155e763dc7" Oct 30 20:08:32.918256 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.918241 5088 scope.go:122] "RemoveContainer" containerID="f54a4ec28ad66ba296a4360924c66536ec38efc33f4ff1e36c3bb003b0ba44ed" Oct 30 20:08:32.931348 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.931324 5088 scope.go:122] "RemoveContainer" containerID="f54a4ec28ad66ba296a4360924c66536ec38efc33f4ff1e36c3bb003b0ba44ed" Oct 30 20:08:32.931673 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:08:32.931641 5088 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f54a4ec28ad66ba296a4360924c66536ec38efc33f4ff1e36c3bb003b0ba44ed\": container with ID starting with f54a4ec28ad66ba296a4360924c66536ec38efc33f4ff1e36c3bb003b0ba44ed not found: ID does not exist" containerID="f54a4ec28ad66ba296a4360924c66536ec38efc33f4ff1e36c3bb003b0ba44ed" Oct 30 20:08:32.931745 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.931681 5088 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f54a4ec28ad66ba296a4360924c66536ec38efc33f4ff1e36c3bb003b0ba44ed"} err="failed to get container status \"f54a4ec28ad66ba296a4360924c66536ec38efc33f4ff1e36c3bb003b0ba44ed\": rpc error: code = NotFound desc = could not find container \"f54a4ec28ad66ba296a4360924c66536ec38efc33f4ff1e36c3bb003b0ba44ed\": container with ID starting with f54a4ec28ad66ba296a4360924c66536ec38efc33f4ff1e36c3bb003b0ba44ed not found: ID does not exist" Oct 30 20:08:32.931745 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.931699 5088 scope.go:122] "RemoveContainer" containerID="5a1758a4fb019200747822b65a3e9e033faf2d04d992fa5b6cb39c925af93acd" Oct 30 20:08:32.941414 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.941394 5088 scope.go:122] "RemoveContainer" containerID="5e32dd83a9a502d3feb8fbe6ae10467c0acbc30fa91cd27137549b4abbe249df" Oct 30 20:08:32.950733 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.950714 5088 scope.go:122] "RemoveContainer" containerID="5e32dd83a9a502d3feb8fbe6ae10467c0acbc30fa91cd27137549b4abbe249df" Oct 30 20:08:32.951004 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:08:32.950986 5088 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e32dd83a9a502d3feb8fbe6ae10467c0acbc30fa91cd27137549b4abbe249df\": container with ID starting with 5e32dd83a9a502d3feb8fbe6ae10467c0acbc30fa91cd27137549b4abbe249df not found: ID does not exist" containerID="5e32dd83a9a502d3feb8fbe6ae10467c0acbc30fa91cd27137549b4abbe249df" Oct 30 20:08:32.951069 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.951011 5088 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e32dd83a9a502d3feb8fbe6ae10467c0acbc30fa91cd27137549b4abbe249df"} err="failed to get container status \"5e32dd83a9a502d3feb8fbe6ae10467c0acbc30fa91cd27137549b4abbe249df\": rpc error: code = NotFound desc = could not find container \"5e32dd83a9a502d3feb8fbe6ae10467c0acbc30fa91cd27137549b4abbe249df\": container with ID starting with 5e32dd83a9a502d3feb8fbe6ae10467c0acbc30fa91cd27137549b4abbe249df not found: ID does not exist" Oct 30 20:08:32.978354 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.978322 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s"] Oct 30 20:08:32.987451 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:32.987422 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-rnt9s"] Oct 30 20:08:33.047291 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:33.047255 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj"] Oct 30 20:08:33.052353 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:33.052325 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-92qbj"] Oct 30 20:08:33.112494 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:33.112450 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d"] Oct 30 20:08:33.116840 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:33.116815 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-6kl8d"] Oct 30 20:08:33.184442 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:33.184350 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr"] Oct 30 20:08:33.190354 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:33.190327 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4w2qr"] Oct 30 20:08:33.230302 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:33.230263 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp"] Oct 30 20:08:33.245775 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:33.245746 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-2hpqp"] Oct 30 20:08:33.304368 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:33.304334 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2"] Oct 30 20:08:33.312339 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:33.312304 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-jvbg2"] Oct 30 20:08:33.351559 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:33.351527 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s"] Oct 30 20:08:33.362624 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:33.362583 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-59p8s"] Oct 30 20:08:33.402011 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:33.401971 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg"] Oct 30 20:08:33.408437 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:33.408406 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-2jwfg"] Oct 30 20:08:33.450121 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:33.450042 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn"] Oct 30 20:08:33.455846 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:33.455817 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-62f4a1-openshift-default-7fc84fbccc-lgbvn"] Oct 30 20:08:33.497325 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:33.497293 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r"] Oct 30 20:08:33.505346 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:33.505316 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f10d2-openshift-default-6c6777d6f5-qff9r"] Oct 30 20:08:34.739957 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:34.739923 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00264202-1595-4616-8796-2ecd7415c577" path="/var/lib/kubelet/pods/00264202-1595-4616-8796-2ecd7415c577/volumes" Oct 30 20:08:34.740417 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:34.740403 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="419aecf7-a250-42ad-995b-e7ebd9630a76" path="/var/lib/kubelet/pods/419aecf7-a250-42ad-995b-e7ebd9630a76/volumes" Oct 30 20:08:34.740871 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:34.740856 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98275e84-2e20-46db-8264-27ec2ac2e2f2" path="/var/lib/kubelet/pods/98275e84-2e20-46db-8264-27ec2ac2e2f2/volumes" Oct 30 20:08:34.741628 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:34.741597 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0c9594e-8534-4538-a4bc-2d40d918e362" path="/var/lib/kubelet/pods/a0c9594e-8534-4538-a4bc-2d40d918e362/volumes" Oct 30 20:08:34.742046 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:34.742032 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b55ec6e2-9392-4d8d-8340-ef12f674970c" path="/var/lib/kubelet/pods/b55ec6e2-9392-4d8d-8340-ef12f674970c/volumes" Oct 30 20:08:34.742440 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:34.742428 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba76bfa7-6ba7-4862-8348-ca7c17487b9f" path="/var/lib/kubelet/pods/ba76bfa7-6ba7-4862-8348-ca7c17487b9f/volumes" Oct 30 20:08:34.743211 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:34.743193 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d209c20c-000f-4cc4-9792-5a71eeb0a773" path="/var/lib/kubelet/pods/d209c20c-000f-4cc4-9792-5a71eeb0a773/volumes" Oct 30 20:08:34.743660 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:34.743644 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d44744f7-8f4c-4c79-978b-52649ce5a5ed" path="/var/lib/kubelet/pods/d44744f7-8f4c-4c79-978b-52649ce5a5ed/volumes" Oct 30 20:08:34.744067 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:34.744054 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc3ed9fa-b7f0-418f-8ac9-422f9e765027" path="/var/lib/kubelet/pods/fc3ed9fa-b7f0-418f-8ac9-422f9e765027/volumes" Oct 30 20:08:34.744781 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:34.744766 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff410c2e-9048-4e5c-965b-1261336561bb" path="/var/lib/kubelet/pods/ff410c2e-9048-4e5c-965b-1261336561bb/volumes" Oct 30 20:08:40.578636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.578361 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-51dd62-2hs82"] Oct 30 20:08:40.579641 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.579211 5088 cpu_manager.go:414] "containerMap: removing container" podUID="b55ec6e2-9392-4d8d-8340-ef12f674970c" containerName="istio-proxy" Oct 30 20:08:40.579641 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.579239 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b55ec6e2-9392-4d8d-8340-ef12f674970c" containerName="istio-proxy" Oct 30 20:08:40.579641 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.579258 5088 cpu_manager.go:414] "containerMap: removing container" podUID="ba76bfa7-6ba7-4862-8348-ca7c17487b9f" containerName="istio-proxy" Oct 30 20:08:40.579641 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.579266 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba76bfa7-6ba7-4862-8348-ca7c17487b9f" containerName="istio-proxy" Oct 30 20:08:40.579641 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.579308 5088 cpu_manager.go:414] "containerMap: removing container" podUID="d44744f7-8f4c-4c79-978b-52649ce5a5ed" containerName="istio-proxy" Oct 30 20:08:40.579641 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.579316 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d44744f7-8f4c-4c79-978b-52649ce5a5ed" containerName="istio-proxy" Oct 30 20:08:40.579641 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.579326 5088 cpu_manager.go:414] "containerMap: removing container" podUID="98275e84-2e20-46db-8264-27ec2ac2e2f2" containerName="istio-proxy" Oct 30 20:08:40.579641 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.579333 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98275e84-2e20-46db-8264-27ec2ac2e2f2" containerName="istio-proxy" Oct 30 20:08:40.579641 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.579346 5088 cpu_manager.go:414] "containerMap: removing container" podUID="00264202-1595-4616-8796-2ecd7415c577" containerName="istio-proxy" Oct 30 20:08:40.579641 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.579353 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00264202-1595-4616-8796-2ecd7415c577" containerName="istio-proxy" Oct 30 20:08:40.579641 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.579363 5088 cpu_manager.go:414] "containerMap: removing container" podUID="442a352a-40cd-4152-97a5-88a598908ac1" containerName="revoke-keys" Oct 30 20:08:40.579641 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.579370 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="442a352a-40cd-4152-97a5-88a598908ac1" containerName="revoke-keys" Oct 30 20:08:40.579641 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.579383 5088 cpu_manager.go:414] "containerMap: removing container" podUID="d209c20c-000f-4cc4-9792-5a71eeb0a773" containerName="istio-proxy" Oct 30 20:08:40.579641 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.579390 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d209c20c-000f-4cc4-9792-5a71eeb0a773" containerName="istio-proxy" Oct 30 20:08:40.579641 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.579399 5088 cpu_manager.go:414] "containerMap: removing container" podUID="a0c9594e-8534-4538-a4bc-2d40d918e362" containerName="istio-proxy" Oct 30 20:08:40.579641 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.579406 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0c9594e-8534-4538-a4bc-2d40d918e362" containerName="istio-proxy" Oct 30 20:08:40.579641 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.579422 5088 cpu_manager.go:414] "containerMap: removing container" podUID="419aecf7-a250-42ad-995b-e7ebd9630a76" containerName="istio-proxy" Oct 30 20:08:40.579641 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.579429 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="419aecf7-a250-42ad-995b-e7ebd9630a76" containerName="istio-proxy" Oct 30 20:08:40.579641 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.579439 5088 cpu_manager.go:414] "containerMap: removing container" podUID="fc3ed9fa-b7f0-418f-8ac9-422f9e765027" containerName="istio-proxy" Oct 30 20:08:40.579641 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.579446 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc3ed9fa-b7f0-418f-8ac9-422f9e765027" containerName="istio-proxy" Oct 30 20:08:40.579641 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.579483 5088 cpu_manager.go:414] "containerMap: removing container" podUID="ff410c2e-9048-4e5c-965b-1261336561bb" containerName="istio-proxy" Oct 30 20:08:40.579641 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.579490 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff410c2e-9048-4e5c-965b-1261336561bb" containerName="istio-proxy" Oct 30 20:08:40.579641 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.579604 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="d44744f7-8f4c-4c79-978b-52649ce5a5ed" containerName="istio-proxy" Oct 30 20:08:40.579641 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.579635 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="98275e84-2e20-46db-8264-27ec2ac2e2f2" containerName="istio-proxy" Oct 30 20:08:40.579641 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.579644 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="00264202-1595-4616-8796-2ecd7415c577" containerName="istio-proxy" Oct 30 20:08:40.579641 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.579655 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="442a352a-40cd-4152-97a5-88a598908ac1" containerName="revoke-keys" Oct 30 20:08:40.579641 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.579663 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="d209c20c-000f-4cc4-9792-5a71eeb0a773" containerName="istio-proxy" Oct 30 20:08:40.580964 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.579671 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0c9594e-8534-4538-a4bc-2d40d918e362" containerName="istio-proxy" Oct 30 20:08:40.580964 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.579681 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="419aecf7-a250-42ad-995b-e7ebd9630a76" containerName="istio-proxy" Oct 30 20:08:40.580964 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.579689 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc3ed9fa-b7f0-418f-8ac9-422f9e765027" containerName="istio-proxy" Oct 30 20:08:40.580964 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.579703 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff410c2e-9048-4e5c-965b-1261336561bb" containerName="istio-proxy" Oct 30 20:08:40.580964 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.579712 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="b55ec6e2-9392-4d8d-8340-ef12f674970c" containerName="istio-proxy" Oct 30 20:08:40.580964 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.579719 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba76bfa7-6ba7-4862-8348-ca7c17487b9f" containerName="istio-proxy" Oct 30 20:08:40.580964 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.579732 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="05a46855-a143-4c6d-8c87-404dc271bdb3" containerName="istio-proxy" Oct 30 20:08:40.633345 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.633308 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-51dd62-2hs82"] Oct 30 20:08:40.633708 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.633687 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-51dd62-2hs82" Oct 30 20:08:40.683890 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.683753 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:08:40.876535 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.834368 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:08:40.876535 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.863070 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92e0bb64-54a3-4fb9-8423-c86b5c21f241-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-51dd62-2hs82\" (UID: \"92e0bb64-54a3-4fb9-8423-c86b5c21f241\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-51dd62-2hs82" Oct 30 20:08:40.876535 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.863131 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntfbf\" (UniqueName: \"kubernetes.io/projected/92e0bb64-54a3-4fb9-8423-c86b5c21f241-kube-api-access-ntfbf\") pod \"maas-api-revoke-keys-e2e-worker-w6-51dd62-2hs82\" (UID: \"92e0bb64-54a3-4fb9-8423-c86b5c21f241\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-51dd62-2hs82" Oct 30 20:08:40.876535 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.863293 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92e0bb64-54a3-4fb9-8423-c86b5c21f241-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-51dd62-2hs82\" (UID: \"92e0bb64-54a3-4fb9-8423-c86b5c21f241\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-51dd62-2hs82" Oct 30 20:08:40.964590 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.964549 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92e0bb64-54a3-4fb9-8423-c86b5c21f241-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-51dd62-2hs82\" (UID: \"92e0bb64-54a3-4fb9-8423-c86b5c21f241\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-51dd62-2hs82" Oct 30 20:08:40.964770 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.964715 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92e0bb64-54a3-4fb9-8423-c86b5c21f241-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-51dd62-2hs82\" (UID: \"92e0bb64-54a3-4fb9-8423-c86b5c21f241\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-51dd62-2hs82" Oct 30 20:08:40.964770 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.964748 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntfbf\" (UniqueName: \"kubernetes.io/projected/92e0bb64-54a3-4fb9-8423-c86b5c21f241-kube-api-access-ntfbf\") pod \"maas-api-revoke-keys-e2e-worker-w6-51dd62-2hs82\" (UID: \"92e0bb64-54a3-4fb9-8423-c86b5c21f241\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-51dd62-2hs82" Oct 30 20:08:40.965392 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.965367 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92e0bb64-54a3-4fb9-8423-c86b5c21f241-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-51dd62-2hs82\" (UID: \"92e0bb64-54a3-4fb9-8423-c86b5c21f241\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-51dd62-2hs82" Oct 30 20:08:40.967004 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.966981 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92e0bb64-54a3-4fb9-8423-c86b5c21f241-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-51dd62-2hs82\" (UID: \"92e0bb64-54a3-4fb9-8423-c86b5c21f241\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-51dd62-2hs82" Oct 30 20:08:40.992352 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:40.992324 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntfbf\" (UniqueName: \"kubernetes.io/projected/92e0bb64-54a3-4fb9-8423-c86b5c21f241-kube-api-access-ntfbf\") pod \"maas-api-revoke-keys-e2e-worker-w6-51dd62-2hs82\" (UID: \"92e0bb64-54a3-4fb9-8423-c86b5c21f241\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-51dd62-2hs82" Oct 30 20:08:41.183169 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:41.183078 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-51dd62-2hs82" Oct 30 20:08:41.517440 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:41.517409 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-51dd62-2hs82"] Oct 30 20:08:41.517992 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:08:41.517958 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92e0bb64_54a3_4fb9_8423_c86b5c21f241.slice/crio-e9afeb4b3199c306499b5269f32203c9a33b882f3bb4d1091767e7e10de49fa2 WatchSource:0}: Error finding container e9afeb4b3199c306499b5269f32203c9a33b882f3bb4d1091767e7e10de49fa2: Status 404 returned error can't find the container with id e9afeb4b3199c306499b5269f32203c9a33b882f3bb4d1091767e7e10de49fa2 Oct 30 20:08:41.789999 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:08:41.789923 5088 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92e0bb64_54a3_4fb9_8423_c86b5c21f241.slice/crio-f2793a1a24f84756d52de31e4f03449dfd4ea04b92b1177378505ee6b83e6dd0.scope\": RecentStats: unable to find data in memory cache]" Oct 30 20:08:41.993288 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:41.993247 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-51dd62-2hs82" event={"ID":"92e0bb64-54a3-4fb9-8423-c86b5c21f241","Type":"ContainerStarted","Data":"f2793a1a24f84756d52de31e4f03449dfd4ea04b92b1177378505ee6b83e6dd0"} Oct 30 20:08:41.993529 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:41.993309 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-51dd62-2hs82" event={"ID":"92e0bb64-54a3-4fb9-8423-c86b5c21f241","Type":"ContainerStarted","Data":"e9afeb4b3199c306499b5269f32203c9a33b882f3bb4d1091767e7e10de49fa2"} Oct 30 20:08:42.041335 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:42.041238 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-51dd62-2hs82" podStartSLOduration=2.04122273 podStartE2EDuration="2.04122273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:42.03815093 +0000 UTC m=+1283.450864757" watchObservedRunningTime="2026-10-30 20:08:42.04122273 +0000 UTC m=+1283.453936522" Oct 30 20:08:43.005049 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:43.004990 5088 generic.go:356] "Generic (PLEG): container finished" podID="92e0bb64-54a3-4fb9-8423-c86b5c21f241" containerID="f2793a1a24f84756d52de31e4f03449dfd4ea04b92b1177378505ee6b83e6dd0" exitCode=0 Oct 30 20:08:43.005525 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:43.005079 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-51dd62-2hs82" event={"ID":"92e0bb64-54a3-4fb9-8423-c86b5c21f241","Type":"ContainerDied","Data":"f2793a1a24f84756d52de31e4f03449dfd4ea04b92b1177378505ee6b83e6dd0"} Oct 30 20:08:44.163280 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:44.163253 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-51dd62-2hs82" Oct 30 20:08:44.304119 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:44.304027 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92e0bb64-54a3-4fb9-8423-c86b5c21f241-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92e0bb64-54a3-4fb9-8423-c86b5c21f241-maas-api-xks-ca\") pod \"92e0bb64-54a3-4fb9-8423-c86b5c21f241\" (UID: \"92e0bb64-54a3-4fb9-8423-c86b5c21f241\") " Oct 30 20:08:44.304285 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:44.304122 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92e0bb64-54a3-4fb9-8423-c86b5c21f241-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92e0bb64-54a3-4fb9-8423-c86b5c21f241-maas-api-openshift-ca\") pod \"92e0bb64-54a3-4fb9-8423-c86b5c21f241\" (UID: \"92e0bb64-54a3-4fb9-8423-c86b5c21f241\") " Oct 30 20:08:44.304285 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:44.304236 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92e0bb64-54a3-4fb9-8423-c86b5c21f241-kube-api-access-ntfbf\" (UniqueName: \"kubernetes.io/projected/92e0bb64-54a3-4fb9-8423-c86b5c21f241-kube-api-access-ntfbf\") pod \"92e0bb64-54a3-4fb9-8423-c86b5c21f241\" (UID: \"92e0bb64-54a3-4fb9-8423-c86b5c21f241\") " Oct 30 20:08:44.304491 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:44.304467 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92e0bb64-54a3-4fb9-8423-c86b5c21f241-maas-api-openshift-ca" pod "92e0bb64-54a3-4fb9-8423-c86b5c21f241" (UID: "92e0bb64-54a3-4fb9-8423-c86b5c21f241"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:44.306356 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:44.306329 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92e0bb64-54a3-4fb9-8423-c86b5c21f241-kube-api-access-ntfbf" pod "92e0bb64-54a3-4fb9-8423-c86b5c21f241" (UID: "92e0bb64-54a3-4fb9-8423-c86b5c21f241"). InnerVolumeSpecName "kube-api-access-ntfbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:44.306468 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:44.306388 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92e0bb64-54a3-4fb9-8423-c86b5c21f241-maas-api-xks-ca" pod "92e0bb64-54a3-4fb9-8423-c86b5c21f241" (UID: "92e0bb64-54a3-4fb9-8423-c86b5c21f241"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:08:44.405442 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:44.405409 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92e0bb64-54a3-4fb9-8423-c86b5c21f241-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:44.405442 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:44.405439 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92e0bb64-54a3-4fb9-8423-c86b5c21f241-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:44.405442 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:44.405449 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntfbf\" (UniqueName: \"kubernetes.io/projected/92e0bb64-54a3-4fb9-8423-c86b5c21f241-kube-api-access-ntfbf\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:45.022873 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:45.022794 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-51dd62-2hs82" Oct 30 20:08:45.022873 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:45.022824 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-51dd62-2hs82" event={"ID":"92e0bb64-54a3-4fb9-8423-c86b5c21f241","Type":"ContainerDied","Data":"e9afeb4b3199c306499b5269f32203c9a33b882f3bb4d1091767e7e10de49fa2"} Oct 30 20:08:45.023065 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:45.022877 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9afeb4b3199c306499b5269f32203c9a33b882f3bb4d1091767e7e10de49fa2" Oct 30 20:08:45.384800 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:45.384716 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b"] Oct 30 20:08:45.395752 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:45.395718 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t"] Oct 30 20:08:45.578754 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:45.578723 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:08:51.062193 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:51.062150 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-51dd62-5668978b9c-5djrs"] Oct 30 20:08:51.097255 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:51.097216 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-51dd62-79f5b56fd7-ltqtf"] Oct 30 20:08:52.437868 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:08:52.437833 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:08:54.136231 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:54.136196 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-475b131d-vbdxt"] Oct 30 20:08:54.136743 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:54.136727 5088 cpu_manager.go:414] "containerMap: removing container" podUID="92e0bb64-54a3-4fb9-8423-c86b5c21f241" containerName="revoke-keys" Oct 30 20:08:54.136793 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:54.136744 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92e0bb64-54a3-4fb9-8423-c86b5c21f241" containerName="revoke-keys" Oct 30 20:08:54.136862 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:54.136853 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="92e0bb64-54a3-4fb9-8423-c86b5c21f241" containerName="revoke-keys" Oct 30 20:08:54.140685 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:54.140668 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-475b131d-vbdxt" Oct 30 20:08:54.166369 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:54.166330 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-475b131d-vbdxt"] Oct 30 20:08:54.171647 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:54.171372 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:08:54.171647 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:54.171489 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:08:54.280166 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:54.280123 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c157f2b-3789-4333-ba93-4920242f6093-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-475b131d-vbdxt\" (UID: \"6c157f2b-3789-4333-ba93-4920242f6093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475b131d-vbdxt" Oct 30 20:08:54.280358 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:54.280273 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9psg\" (UniqueName: \"kubernetes.io/projected/6c157f2b-3789-4333-ba93-4920242f6093-kube-api-access-q9psg\") pod \"maas-api-revoke-sub-475b131d-vbdxt\" (UID: \"6c157f2b-3789-4333-ba93-4920242f6093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475b131d-vbdxt" Oct 30 20:08:54.280358 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:54.280323 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c157f2b-3789-4333-ba93-4920242f6093-maas-api-xks-ca\") pod \"maas-api-revoke-sub-475b131d-vbdxt\" (UID: \"6c157f2b-3789-4333-ba93-4920242f6093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475b131d-vbdxt" Oct 30 20:08:54.381214 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:54.381154 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9psg\" (UniqueName: \"kubernetes.io/projected/6c157f2b-3789-4333-ba93-4920242f6093-kube-api-access-q9psg\") pod \"maas-api-revoke-sub-475b131d-vbdxt\" (UID: \"6c157f2b-3789-4333-ba93-4920242f6093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475b131d-vbdxt" Oct 30 20:08:54.381214 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:54.381219 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c157f2b-3789-4333-ba93-4920242f6093-maas-api-xks-ca\") pod \"maas-api-revoke-sub-475b131d-vbdxt\" (UID: \"6c157f2b-3789-4333-ba93-4920242f6093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475b131d-vbdxt" Oct 30 20:08:54.381441 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:54.381381 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c157f2b-3789-4333-ba93-4920242f6093-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-475b131d-vbdxt\" (UID: \"6c157f2b-3789-4333-ba93-4920242f6093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475b131d-vbdxt" Oct 30 20:08:54.382102 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:54.382082 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c157f2b-3789-4333-ba93-4920242f6093-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-475b131d-vbdxt\" (UID: \"6c157f2b-3789-4333-ba93-4920242f6093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475b131d-vbdxt" Oct 30 20:08:54.383689 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:54.383660 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c157f2b-3789-4333-ba93-4920242f6093-maas-api-xks-ca\") pod \"maas-api-revoke-sub-475b131d-vbdxt\" (UID: \"6c157f2b-3789-4333-ba93-4920242f6093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475b131d-vbdxt" Oct 30 20:08:54.409559 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:54.409485 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9psg\" (UniqueName: \"kubernetes.io/projected/6c157f2b-3789-4333-ba93-4920242f6093-kube-api-access-q9psg\") pod \"maas-api-revoke-sub-475b131d-vbdxt\" (UID: \"6c157f2b-3789-4333-ba93-4920242f6093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475b131d-vbdxt" Oct 30 20:08:54.454605 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:54.454563 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-475b131d-vbdxt" Oct 30 20:08:54.763783 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:54.763756 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-475b131d-vbdxt"] Oct 30 20:08:54.763959 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:08:54.763937 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c157f2b_3789_4333_ba93_4920242f6093.slice/crio-be53fccf7d3447215bac119f6f63772d3e692aaab6c31d5abd4ace6f2b93453f WatchSource:0}: Error finding container be53fccf7d3447215bac119f6f63772d3e692aaab6c31d5abd4ace6f2b93453f: Status 404 returned error can't find the container with id be53fccf7d3447215bac119f6f63772d3e692aaab6c31d5abd4ace6f2b93453f Oct 30 20:08:55.171502 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:55.171459 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-475b131d-vbdxt" event={"ID":"6c157f2b-3789-4333-ba93-4920242f6093","Type":"ContainerStarted","Data":"bdce559f15a9ddd71a156d312b953b301abbec6193968167c3875a8224e1244b"} Oct 30 20:08:55.172017 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:55.171528 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-475b131d-vbdxt" event={"ID":"6c157f2b-3789-4333-ba93-4920242f6093","Type":"ContainerStarted","Data":"be53fccf7d3447215bac119f6f63772d3e692aaab6c31d5abd4ace6f2b93453f"} Oct 30 20:08:55.221664 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:55.221594 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-475b131d-vbdxt" podStartSLOduration=1.221581265 podStartE2EDuration="1.221581265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:55.217150023 +0000 UTC m=+1296.629863815" watchObservedRunningTime="2026-10-30 20:08:55.221581265 +0000 UTC m=+1296.634295044" Oct 30 20:08:56.063146 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:56.063103 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-51dd62-5668978b9c-5djrs" podUID="2f2223b9-aebe-4aab-9437-af457ece8c69" containerName="payload-processing" containerID="cri-o://a9b6f46f57963859ae28c7ec7bdbf404c5ec2de1c93b2e24e47bb988b171a3ca" gracePeriod=25 Oct 30 20:08:56.097733 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:56.097688 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-51dd62-79f5b56fd7-ltqtf" podUID="ab7bcd6d-197b-4c1e-9ce1-4aa111e96e54" containerName="payload-pre-processing" containerID="cri-o://36e4fdc76cc5e54e943202cd1add398d8d06a6e1797847f44f6ad6c8daa3b250" gracePeriod=25 Oct 30 20:08:56.180050 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:56.180015 5088 generic.go:356] "Generic (PLEG): container finished" podID="6c157f2b-3789-4333-ba93-4920242f6093" containerID="bdce559f15a9ddd71a156d312b953b301abbec6193968167c3875a8224e1244b" exitCode=0 Oct 30 20:08:56.180460 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:56.180110 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-475b131d-vbdxt" event={"ID":"6c157f2b-3789-4333-ba93-4920242f6093","Type":"ContainerDied","Data":"bdce559f15a9ddd71a156d312b953b301abbec6193968167c3875a8224e1244b"} Oct 30 20:08:56.283166 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:56.283130 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs"] Oct 30 20:08:57.326921 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:57.326894 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-475b131d-vbdxt" Oct 30 20:08:57.416757 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:57.416723 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c157f2b-3789-4333-ba93-4920242f6093-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c157f2b-3789-4333-ba93-4920242f6093-maas-api-openshift-ca\") pod \"6c157f2b-3789-4333-ba93-4920242f6093\" (UID: \"6c157f2b-3789-4333-ba93-4920242f6093\") " Oct 30 20:08:57.416966 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:57.416768 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c157f2b-3789-4333-ba93-4920242f6093-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c157f2b-3789-4333-ba93-4920242f6093-maas-api-xks-ca\") pod \"6c157f2b-3789-4333-ba93-4920242f6093\" (UID: \"6c157f2b-3789-4333-ba93-4920242f6093\") " Oct 30 20:08:57.416966 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:57.416829 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c157f2b-3789-4333-ba93-4920242f6093-kube-api-access-q9psg\" (UniqueName: \"kubernetes.io/projected/6c157f2b-3789-4333-ba93-4920242f6093-kube-api-access-q9psg\") pod \"6c157f2b-3789-4333-ba93-4920242f6093\" (UID: \"6c157f2b-3789-4333-ba93-4920242f6093\") " Oct 30 20:08:57.417190 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:57.417164 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c157f2b-3789-4333-ba93-4920242f6093-maas-api-openshift-ca" pod "6c157f2b-3789-4333-ba93-4920242f6093" (UID: "6c157f2b-3789-4333-ba93-4920242f6093"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:57.419015 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:57.418989 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c157f2b-3789-4333-ba93-4920242f6093-kube-api-access-q9psg" pod "6c157f2b-3789-4333-ba93-4920242f6093" (UID: "6c157f2b-3789-4333-ba93-4920242f6093"). InnerVolumeSpecName "kube-api-access-q9psg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:57.419128 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:57.418996 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c157f2b-3789-4333-ba93-4920242f6093-maas-api-xks-ca" pod "6c157f2b-3789-4333-ba93-4920242f6093" (UID: "6c157f2b-3789-4333-ba93-4920242f6093"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:08:57.518709 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:57.518594 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c157f2b-3789-4333-ba93-4920242f6093-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:57.518709 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:57.518705 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c157f2b-3789-4333-ba93-4920242f6093-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:57.518709 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:57.518720 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9psg\" (UniqueName: \"kubernetes.io/projected/6c157f2b-3789-4333-ba93-4920242f6093-kube-api-access-q9psg\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:08:58.196552 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:58.196518 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-475b131d-vbdxt" Oct 30 20:08:58.196755 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:58.196525 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-475b131d-vbdxt" event={"ID":"6c157f2b-3789-4333-ba93-4920242f6093","Type":"ContainerDied","Data":"be53fccf7d3447215bac119f6f63772d3e692aaab6c31d5abd4ace6f2b93453f"} Oct 30 20:08:58.196755 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:58.196691 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be53fccf7d3447215bac119f6f63772d3e692aaab6c31d5abd4ace6f2b93453f" Oct 30 20:08:58.952896 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:08:58.952859 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:09:00.406059 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:00.406012 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" containerID="cri-o://e41fb20108e4fc754dc7d7e0a0263fd9ea78afdba73dee850bce2bc8b933fe02" gracePeriod=45 Oct 30 20:09:00.417075 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:00.417035 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" containerID="cri-o://a577aa3fb052d00f2e6a84cdf22c1cb06b661a526ca1d38e74ebea0991977dad" gracePeriod=45 Oct 30 20:09:00.690845 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:00.690817 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" Oct 30 20:09:00.719006 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:00.718978 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:09:00.860268 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:00.860234 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/659701e4-4e82-4c4a-b651-4ad8641f7b65-kube-api-access-mk9pt\" (UniqueName: \"kubernetes.io/projected/659701e4-4e82-4c4a-b651-4ad8641f7b65-kube-api-access-mk9pt\") pod \"659701e4-4e82-4c4a-b651-4ad8641f7b65\" (UID: \"659701e4-4e82-4c4a-b651-4ad8641f7b65\") " Oct 30 20:09:00.860448 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:00.860298 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/659701e4-4e82-4c4a-b651-4ad8641f7b65-dshm\" (UniqueName: \"kubernetes.io/empty-dir/659701e4-4e82-4c4a-b651-4ad8641f7b65-dshm\") pod \"659701e4-4e82-4c4a-b651-4ad8641f7b65\" (UID: \"659701e4-4e82-4c4a-b651-4ad8641f7b65\") " Oct 30 20:09:00.860448 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:00.860361 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/659701e4-4e82-4c4a-b651-4ad8641f7b65-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/659701e4-4e82-4c4a-b651-4ad8641f7b65-model-cache\") pod \"659701e4-4e82-4c4a-b651-4ad8641f7b65\" (UID: \"659701e4-4e82-4c4a-b651-4ad8641f7b65\") " Oct 30 20:09:00.860448 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:00.860402 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b42fe97-2609-4fa1-a51a-c901e34997b5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b42fe97-2609-4fa1-a51a-c901e34997b5-tmp-dir\") pod \"5b42fe97-2609-4fa1-a51a-c901e34997b5\" (UID: \"5b42fe97-2609-4fa1-a51a-c901e34997b5\") " Oct 30 20:09:00.860651 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:00.860461 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b42fe97-2609-4fa1-a51a-c901e34997b5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b42fe97-2609-4fa1-a51a-c901e34997b5-model-cache\") pod \"5b42fe97-2609-4fa1-a51a-c901e34997b5\" (UID: \"5b42fe97-2609-4fa1-a51a-c901e34997b5\") " Oct 30 20:09:00.860651 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:00.860524 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b42fe97-2609-4fa1-a51a-c901e34997b5-home\" (UniqueName: \"kubernetes.io/empty-dir/5b42fe97-2609-4fa1-a51a-c901e34997b5-home\") pod \"5b42fe97-2609-4fa1-a51a-c901e34997b5\" (UID: \"5b42fe97-2609-4fa1-a51a-c901e34997b5\") " Oct 30 20:09:00.860651 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:00.860600 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/659701e4-4e82-4c4a-b651-4ad8641f7b65-tls-certs\" (UniqueName: \"kubernetes.io/secret/659701e4-4e82-4c4a-b651-4ad8641f7b65-tls-certs\") pod \"659701e4-4e82-4c4a-b651-4ad8641f7b65\" (UID: \"659701e4-4e82-4c4a-b651-4ad8641f7b65\") " Oct 30 20:09:00.860820 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:00.860685 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/659701e4-4e82-4c4a-b651-4ad8641f7b65-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/659701e4-4e82-4c4a-b651-4ad8641f7b65-tmp-dir\") pod \"659701e4-4e82-4c4a-b651-4ad8641f7b65\" (UID: \"659701e4-4e82-4c4a-b651-4ad8641f7b65\") " Oct 30 20:09:00.860820 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:00.860719 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b42fe97-2609-4fa1-a51a-c901e34997b5-kube-api-access-74hxr\" (UniqueName: \"kubernetes.io/projected/5b42fe97-2609-4fa1-a51a-c901e34997b5-kube-api-access-74hxr\") pod \"5b42fe97-2609-4fa1-a51a-c901e34997b5\" (UID: \"5b42fe97-2609-4fa1-a51a-c901e34997b5\") " Oct 30 20:09:00.860820 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:00.860765 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b42fe97-2609-4fa1-a51a-c901e34997b5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b42fe97-2609-4fa1-a51a-c901e34997b5-dshm\") pod \"5b42fe97-2609-4fa1-a51a-c901e34997b5\" (UID: \"5b42fe97-2609-4fa1-a51a-c901e34997b5\") " Oct 30 20:09:00.860820 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:00.860767 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b42fe97-2609-4fa1-a51a-c901e34997b5-tmp-dir" pod "5b42fe97-2609-4fa1-a51a-c901e34997b5" (UID: "5b42fe97-2609-4fa1-a51a-c901e34997b5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:00.860820 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:00.860812 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/659701e4-4e82-4c4a-b651-4ad8641f7b65-home\" (UniqueName: \"kubernetes.io/empty-dir/659701e4-4e82-4c4a-b651-4ad8641f7b65-home\") pod \"659701e4-4e82-4c4a-b651-4ad8641f7b65\" (UID: \"659701e4-4e82-4c4a-b651-4ad8641f7b65\") " Oct 30 20:09:00.861088 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:00.860840 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b42fe97-2609-4fa1-a51a-c901e34997b5-model-cache" pod "5b42fe97-2609-4fa1-a51a-c901e34997b5" (UID: "5b42fe97-2609-4fa1-a51a-c901e34997b5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:00.861088 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:00.860887 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b42fe97-2609-4fa1-a51a-c901e34997b5-tls-certs\" (UniqueName: \"kubernetes.io/secret/5b42fe97-2609-4fa1-a51a-c901e34997b5-tls-certs\") pod \"5b42fe97-2609-4fa1-a51a-c901e34997b5\" (UID: \"5b42fe97-2609-4fa1-a51a-c901e34997b5\") " Oct 30 20:09:00.861088 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:00.861013 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b42fe97-2609-4fa1-a51a-c901e34997b5-home" pod "5b42fe97-2609-4fa1-a51a-c901e34997b5" (UID: "5b42fe97-2609-4fa1-a51a-c901e34997b5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:00.861959 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:00.861333 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/659701e4-4e82-4c4a-b651-4ad8641f7b65-tmp-dir" pod "659701e4-4e82-4c4a-b651-4ad8641f7b65" (UID: "659701e4-4e82-4c4a-b651-4ad8641f7b65"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:00.861959 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:00.861497 5088 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b42fe97-2609-4fa1-a51a-c901e34997b5-tmp-dir\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:00.861959 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:00.861519 5088 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b42fe97-2609-4fa1-a51a-c901e34997b5-model-cache\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:00.861959 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:00.861535 5088 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b42fe97-2609-4fa1-a51a-c901e34997b5-home\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:00.861959 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:00.861549 5088 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/659701e4-4e82-4c4a-b651-4ad8641f7b65-tmp-dir\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:00.861959 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:00.861846 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/659701e4-4e82-4c4a-b651-4ad8641f7b65-home" pod "659701e4-4e82-4c4a-b651-4ad8641f7b65" (UID: "659701e4-4e82-4c4a-b651-4ad8641f7b65"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:00.861959 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:00.861933 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/659701e4-4e82-4c4a-b651-4ad8641f7b65-model-cache" pod "659701e4-4e82-4c4a-b651-4ad8641f7b65" (UID: "659701e4-4e82-4c4a-b651-4ad8641f7b65"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:00.863075 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:00.863049 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/659701e4-4e82-4c4a-b651-4ad8641f7b65-dshm" pod "659701e4-4e82-4c4a-b651-4ad8641f7b65" (UID: "659701e4-4e82-4c4a-b651-4ad8641f7b65"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:00.863211 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:00.863174 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/659701e4-4e82-4c4a-b651-4ad8641f7b65-kube-api-access-mk9pt" pod "659701e4-4e82-4c4a-b651-4ad8641f7b65" (UID: "659701e4-4e82-4c4a-b651-4ad8641f7b65"). InnerVolumeSpecName "kube-api-access-mk9pt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:00.863397 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:00.863379 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b42fe97-2609-4fa1-a51a-c901e34997b5-dshm" pod "5b42fe97-2609-4fa1-a51a-c901e34997b5" (UID: "5b42fe97-2609-4fa1-a51a-c901e34997b5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:00.863495 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:00.863471 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b42fe97-2609-4fa1-a51a-c901e34997b5-kube-api-access-74hxr" pod "5b42fe97-2609-4fa1-a51a-c901e34997b5" (UID: "5b42fe97-2609-4fa1-a51a-c901e34997b5"). InnerVolumeSpecName "kube-api-access-74hxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:00.863495 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:00.863473 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/659701e4-4e82-4c4a-b651-4ad8641f7b65-tls-certs" pod "659701e4-4e82-4c4a-b651-4ad8641f7b65" (UID: "659701e4-4e82-4c4a-b651-4ad8641f7b65"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:09:00.864200 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:00.864173 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b42fe97-2609-4fa1-a51a-c901e34997b5-tls-certs" pod "5b42fe97-2609-4fa1-a51a-c901e34997b5" (UID: "5b42fe97-2609-4fa1-a51a-c901e34997b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:09:00.962562 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:00.962449 5088 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/659701e4-4e82-4c4a-b651-4ad8641f7b65-tls-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:00.962562 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:00.962482 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74hxr\" (UniqueName: \"kubernetes.io/projected/5b42fe97-2609-4fa1-a51a-c901e34997b5-kube-api-access-74hxr\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:00.962562 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:00.962502 5088 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b42fe97-2609-4fa1-a51a-c901e34997b5-dshm\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:00.962562 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:00.962512 5088 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/659701e4-4e82-4c4a-b651-4ad8641f7b65-home\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:00.962562 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:00.962520 5088 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b42fe97-2609-4fa1-a51a-c901e34997b5-tls-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:00.962562 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:00.962529 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk9pt\" (UniqueName: \"kubernetes.io/projected/659701e4-4e82-4c4a-b651-4ad8641f7b65-kube-api-access-mk9pt\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:00.962562 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:00.962536 5088 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/659701e4-4e82-4c4a-b651-4ad8641f7b65-dshm\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:00.962562 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:00.962544 5088 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/659701e4-4e82-4c4a-b651-4ad8641f7b65-model-cache\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:01.221190 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:01.221103 5088 generic.go:356] "Generic (PLEG): container finished" podID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerID="a577aa3fb052d00f2e6a84cdf22c1cb06b661a526ca1d38e74ebea0991977dad" exitCode=0 Oct 30 20:09:01.221190 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:01.221157 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" Oct 30 20:09:01.221409 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:01.221188 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" event={"ID":"5b42fe97-2609-4fa1-a51a-c901e34997b5","Type":"ContainerDied","Data":"a577aa3fb052d00f2e6a84cdf22c1cb06b661a526ca1d38e74ebea0991977dad"} Oct 30 20:09:01.221409 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:01.221233 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" event={"ID":"5b42fe97-2609-4fa1-a51a-c901e34997b5","Type":"ContainerDied","Data":"3378cd2fe8872ab9cbfdbae6f7ab420764ba4aaf2cf54f1aa113814d652fd91c"} Oct 30 20:09:01.221409 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:01.221251 5088 scope.go:122] "RemoveContainer" containerID="a577aa3fb052d00f2e6a84cdf22c1cb06b661a526ca1d38e74ebea0991977dad" Oct 30 20:09:01.222895 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:01.222866 5088 generic.go:356] "Generic (PLEG): container finished" podID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerID="e41fb20108e4fc754dc7d7e0a0263fd9ea78afdba73dee850bce2bc8b933fe02" exitCode=0 Oct 30 20:09:01.223032 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:01.222907 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" Oct 30 20:09:01.223032 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:01.222949 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" event={"ID":"659701e4-4e82-4c4a-b651-4ad8641f7b65","Type":"ContainerDied","Data":"e41fb20108e4fc754dc7d7e0a0263fd9ea78afdba73dee850bce2bc8b933fe02"} Oct 30 20:09:01.223032 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:01.222988 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b" event={"ID":"659701e4-4e82-4c4a-b651-4ad8641f7b65","Type":"ContainerDied","Data":"3989bee64b116e24e78201adce2b4155d02e5ae841f0e326c4b0cc5f0ca01556"} Oct 30 20:09:01.233540 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:01.233519 5088 scope.go:122] "RemoveContainer" containerID="a577aa3fb052d00f2e6a84cdf22c1cb06b661a526ca1d38e74ebea0991977dad" Oct 30 20:09:01.233884 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:09:01.233867 5088 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a577aa3fb052d00f2e6a84cdf22c1cb06b661a526ca1d38e74ebea0991977dad\": container with ID starting with a577aa3fb052d00f2e6a84cdf22c1cb06b661a526ca1d38e74ebea0991977dad not found: ID does not exist" containerID="a577aa3fb052d00f2e6a84cdf22c1cb06b661a526ca1d38e74ebea0991977dad" Oct 30 20:09:01.233944 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:01.233896 5088 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a577aa3fb052d00f2e6a84cdf22c1cb06b661a526ca1d38e74ebea0991977dad"} err="failed to get container status \"a577aa3fb052d00f2e6a84cdf22c1cb06b661a526ca1d38e74ebea0991977dad\": rpc error: code = NotFound desc = could not find container \"a577aa3fb052d00f2e6a84cdf22c1cb06b661a526ca1d38e74ebea0991977dad\": container with ID starting with a577aa3fb052d00f2e6a84cdf22c1cb06b661a526ca1d38e74ebea0991977dad not found: ID does not exist" Oct 30 20:09:01.233944 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:01.233913 5088 scope.go:122] "RemoveContainer" containerID="e41fb20108e4fc754dc7d7e0a0263fd9ea78afdba73dee850bce2bc8b933fe02" Oct 30 20:09:01.243178 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:01.243160 5088 scope.go:122] "RemoveContainer" containerID="e41fb20108e4fc754dc7d7e0a0263fd9ea78afdba73dee850bce2bc8b933fe02" Oct 30 20:09:01.243437 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:09:01.243413 5088 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e41fb20108e4fc754dc7d7e0a0263fd9ea78afdba73dee850bce2bc8b933fe02\": container with ID starting with e41fb20108e4fc754dc7d7e0a0263fd9ea78afdba73dee850bce2bc8b933fe02 not found: ID does not exist" containerID="e41fb20108e4fc754dc7d7e0a0263fd9ea78afdba73dee850bce2bc8b933fe02" Oct 30 20:09:01.243500 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:01.243443 5088 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e41fb20108e4fc754dc7d7e0a0263fd9ea78afdba73dee850bce2bc8b933fe02"} err="failed to get container status \"e41fb20108e4fc754dc7d7e0a0263fd9ea78afdba73dee850bce2bc8b933fe02\": rpc error: code = NotFound desc = could not find container \"e41fb20108e4fc754dc7d7e0a0263fd9ea78afdba73dee850bce2bc8b933fe02\": container with ID starting with e41fb20108e4fc754dc7d7e0a0263fd9ea78afdba73dee850bce2bc8b933fe02 not found: ID does not exist" Oct 30 20:09:01.282941 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:01.282898 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t"] Oct 30 20:09:01.295741 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:01.295706 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t"] Oct 30 20:09:01.327084 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:01.327047 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b"] Oct 30 20:09:01.332001 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:01.331962 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-51dd62/distinct-w6-51dd62-kserve-54bffd8b69-v8k2b"] Oct 30 20:09:01.711532 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:01.711487 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-51dd62/premium-w6-51dd62-kserve-68487679fd-wwr8t" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 20:09:02.015340 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:02.015299 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-efbac3-66db987bc9-ptftf"] Oct 30 20:09:02.532828 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:02.532792 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-51dd62-79f5b56fd7-ltqtf" podUID="ab7bcd6d-197b-4c1e-9ce1-4aa111e96e54" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:09:02.533002 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:02.532792 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-51dd62-5668978b9c-5djrs" podUID="2f2223b9-aebe-4aab-9437-af457ece8c69" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:09:02.742323 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:02.742271 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" path="/var/lib/kubelet/pods/5b42fe97-2609-4fa1-a51a-c901e34997b5/volumes" Oct 30 20:09:02.743767 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:02.743740 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" path="/var/lib/kubelet/pods/659701e4-4e82-4c4a-b651-4ad8641f7b65/volumes" Oct 30 20:09:03.801172 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:03.801135 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd04ef0c-lt96r"] Oct 30 20:09:03.801693 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:03.801679 5088 cpu_manager.go:414] "containerMap: removing container" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" Oct 30 20:09:03.801752 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:03.801695 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" Oct 30 20:09:03.801752 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:03.801729 5088 cpu_manager.go:414] "containerMap: removing container" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" Oct 30 20:09:03.801752 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:03.801734 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" Oct 30 20:09:03.801752 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:03.801750 5088 cpu_manager.go:414] "containerMap: removing container" podUID="6c157f2b-3789-4333-ba93-4920242f6093" containerName="revoke-keys" Oct 30 20:09:03.801885 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:03.801756 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c157f2b-3789-4333-ba93-4920242f6093" containerName="revoke-keys" Oct 30 20:09:03.801885 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:03.801837 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="659701e4-4e82-4c4a-b651-4ad8641f7b65" containerName="main" Oct 30 20:09:03.801885 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:03.801847 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c157f2b-3789-4333-ba93-4920242f6093" containerName="revoke-keys" Oct 30 20:09:03.801885 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:03.801855 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b42fe97-2609-4fa1-a51a-c901e34997b5" containerName="main" Oct 30 20:09:03.807141 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:03.807122 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd04ef0c-lt96r" Oct 30 20:09:03.811040 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:03.811016 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c27da71-bbe3-4855-b6aa-d46fb82e42c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd04ef0c-lt96r\" (UID: \"4c27da71-bbe3-4855-b6aa-d46fb82e42c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd04ef0c-lt96r" Oct 30 20:09:03.811144 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:03.811050 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9652\" (UniqueName: \"kubernetes.io/projected/4c27da71-bbe3-4855-b6aa-d46fb82e42c7-kube-api-access-r9652\") pod \"maas-api-revoke-sub-cd04ef0c-lt96r\" (UID: \"4c27da71-bbe3-4855-b6aa-d46fb82e42c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd04ef0c-lt96r" Oct 30 20:09:03.811144 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:03.811133 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c27da71-bbe3-4855-b6aa-d46fb82e42c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd04ef0c-lt96r\" (UID: \"4c27da71-bbe3-4855-b6aa-d46fb82e42c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd04ef0c-lt96r" Oct 30 20:09:03.825414 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:03.825384 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:09:03.825696 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:03.825678 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:09:03.849355 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:03.849320 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd04ef0c-lt96r"] Oct 30 20:09:03.912295 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:03.912261 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c27da71-bbe3-4855-b6aa-d46fb82e42c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd04ef0c-lt96r\" (UID: \"4c27da71-bbe3-4855-b6aa-d46fb82e42c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd04ef0c-lt96r" Oct 30 20:09:03.912295 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:03.912295 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9652\" (UniqueName: \"kubernetes.io/projected/4c27da71-bbe3-4855-b6aa-d46fb82e42c7-kube-api-access-r9652\") pod \"maas-api-revoke-sub-cd04ef0c-lt96r\" (UID: \"4c27da71-bbe3-4855-b6aa-d46fb82e42c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd04ef0c-lt96r" Oct 30 20:09:03.912565 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:03.912328 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c27da71-bbe3-4855-b6aa-d46fb82e42c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd04ef0c-lt96r\" (UID: \"4c27da71-bbe3-4855-b6aa-d46fb82e42c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd04ef0c-lt96r" Oct 30 20:09:03.912917 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:03.912893 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c27da71-bbe3-4855-b6aa-d46fb82e42c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd04ef0c-lt96r\" (UID: \"4c27da71-bbe3-4855-b6aa-d46fb82e42c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd04ef0c-lt96r" Oct 30 20:09:03.914741 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:03.914721 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c27da71-bbe3-4855-b6aa-d46fb82e42c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd04ef0c-lt96r\" (UID: \"4c27da71-bbe3-4855-b6aa-d46fb82e42c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd04ef0c-lt96r" Oct 30 20:09:03.940017 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:03.939993 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9652\" (UniqueName: \"kubernetes.io/projected/4c27da71-bbe3-4855-b6aa-d46fb82e42c7-kube-api-access-r9652\") pod \"maas-api-revoke-sub-cd04ef0c-lt96r\" (UID: \"4c27da71-bbe3-4855-b6aa-d46fb82e42c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd04ef0c-lt96r" Oct 30 20:09:04.122001 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:04.121906 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd04ef0c-lt96r" Oct 30 20:09:04.438888 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:04.438858 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd04ef0c-lt96r"] Oct 30 20:09:04.438995 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:09:04.438897 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c27da71_bbe3_4855_b6aa_d46fb82e42c7.slice/crio-d3dd7883e089329279ca1615917e540c38f1d64de4f3c70a744cc1eb34746b88 WatchSource:0}: Error finding container d3dd7883e089329279ca1615917e540c38f1d64de4f3c70a744cc1eb34746b88: Status 404 returned error can't find the container with id d3dd7883e089329279ca1615917e540c38f1d64de4f3c70a744cc1eb34746b88 Oct 30 20:09:05.295750 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:05.295715 5088 generic.go:356] "Generic (PLEG): container finished" podID="4c27da71-bbe3-4855-b6aa-d46fb82e42c7" containerID="2f012f570ac80688faf6c9aa785fdab95bdac1c70cf4e79fdc31d380b24e0130" exitCode=0 Oct 30 20:09:05.296148 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:05.295806 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd04ef0c-lt96r" event={"ID":"4c27da71-bbe3-4855-b6aa-d46fb82e42c7","Type":"ContainerDied","Data":"2f012f570ac80688faf6c9aa785fdab95bdac1c70cf4e79fdc31d380b24e0130"} Oct 30 20:09:05.296148 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:05.295865 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd04ef0c-lt96r" event={"ID":"4c27da71-bbe3-4855-b6aa-d46fb82e42c7","Type":"ContainerStarted","Data":"d3dd7883e089329279ca1615917e540c38f1d64de4f3c70a744cc1eb34746b88"} Oct 30 20:09:06.458945 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:06.458921 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd04ef0c-lt96r" Oct 30 20:09:06.536940 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:06.536906 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c27da71-bbe3-4855-b6aa-d46fb82e42c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c27da71-bbe3-4855-b6aa-d46fb82e42c7-maas-api-xks-ca\") pod \"4c27da71-bbe3-4855-b6aa-d46fb82e42c7\" (UID: \"4c27da71-bbe3-4855-b6aa-d46fb82e42c7\") " Oct 30 20:09:06.538986 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:06.538954 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c27da71-bbe3-4855-b6aa-d46fb82e42c7-maas-api-xks-ca" pod "4c27da71-bbe3-4855-b6aa-d46fb82e42c7" (UID: "4c27da71-bbe3-4855-b6aa-d46fb82e42c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:09:06.638063 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:06.637961 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c27da71-bbe3-4855-b6aa-d46fb82e42c7-kube-api-access-r9652\" (UniqueName: \"kubernetes.io/projected/4c27da71-bbe3-4855-b6aa-d46fb82e42c7-kube-api-access-r9652\") pod \"4c27da71-bbe3-4855-b6aa-d46fb82e42c7\" (UID: \"4c27da71-bbe3-4855-b6aa-d46fb82e42c7\") " Oct 30 20:09:06.638063 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:06.638050 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c27da71-bbe3-4855-b6aa-d46fb82e42c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c27da71-bbe3-4855-b6aa-d46fb82e42c7-maas-api-openshift-ca\") pod \"4c27da71-bbe3-4855-b6aa-d46fb82e42c7\" (UID: \"4c27da71-bbe3-4855-b6aa-d46fb82e42c7\") " Oct 30 20:09:06.638379 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:06.638367 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c27da71-bbe3-4855-b6aa-d46fb82e42c7-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:06.638452 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:06.638418 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c27da71-bbe3-4855-b6aa-d46fb82e42c7-maas-api-openshift-ca" pod "4c27da71-bbe3-4855-b6aa-d46fb82e42c7" (UID: "4c27da71-bbe3-4855-b6aa-d46fb82e42c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:09:06.640204 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:06.640186 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c27da71-bbe3-4855-b6aa-d46fb82e42c7-kube-api-access-r9652" pod "4c27da71-bbe3-4855-b6aa-d46fb82e42c7" (UID: "4c27da71-bbe3-4855-b6aa-d46fb82e42c7"). InnerVolumeSpecName "kube-api-access-r9652". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:06.739351 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:06.739319 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9652\" (UniqueName: \"kubernetes.io/projected/4c27da71-bbe3-4855-b6aa-d46fb82e42c7-kube-api-access-r9652\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:06.739351 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:06.739346 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c27da71-bbe3-4855-b6aa-d46fb82e42c7-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:07.016424 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:07.016306 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-efbac3-66db987bc9-ptftf" podUID="0ee0e296-2cf3-4cdc-9340-d202e75c6a14" containerName="payload-pre-processing" containerID="cri-o://253e2f1b3689f4cf53015de4c05a89f5d579f55bad8438f33b5ab9f23e73e376" gracePeriod=25 Oct 30 20:09:07.251539 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:07.251501 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:09:07.313373 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:07.313281 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd04ef0c-lt96r" event={"ID":"4c27da71-bbe3-4855-b6aa-d46fb82e42c7","Type":"ContainerDied","Data":"d3dd7883e089329279ca1615917e540c38f1d64de4f3c70a744cc1eb34746b88"} Oct 30 20:09:07.313373 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:07.313339 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3dd7883e089329279ca1615917e540c38f1d64de4f3c70a744cc1eb34746b88" Oct 30 20:09:07.313373 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:07.313306 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd04ef0c-lt96r" Oct 30 20:09:08.783336 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:08.783303 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957"] Oct 30 20:09:08.810643 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:08.788723 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957" podUID="be2ffe6a-d355-4d95-a7ae-46adeb952736" containerName="maas-api" containerID="cri-o://f72efc6e822122441efed9c58d4563ca1068b6453cbace028b5bf3b95a20c799" gracePeriod=30 Oct 30 20:09:09.365814 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:09.365733 5088 generic.go:356] "Generic (PLEG): container finished" podID="be2ffe6a-d355-4d95-a7ae-46adeb952736" containerID="f72efc6e822122441efed9c58d4563ca1068b6453cbace028b5bf3b95a20c799" exitCode=0 Oct 30 20:09:09.366151 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:09.366134 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957" event={"ID":"be2ffe6a-d355-4d95-a7ae-46adeb952736","Type":"ContainerDied","Data":"f72efc6e822122441efed9c58d4563ca1068b6453cbace028b5bf3b95a20c799"} Oct 30 20:09:09.589634 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:09.588638 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-51dd62-2hs82"] Oct 30 20:09:09.589634 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:09.588811 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-51dd62-2hs82"] Oct 30 20:09:09.697936 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:09.697885 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957" Oct 30 20:09:09.735820 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:09.734679 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be2ffe6a-d355-4d95-a7ae-46adeb952736-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be2ffe6a-d355-4d95-a7ae-46adeb952736-openshift-service-ca\") pod \"be2ffe6a-d355-4d95-a7ae-46adeb952736\" (UID: \"be2ffe6a-d355-4d95-a7ae-46adeb952736\") " Oct 30 20:09:09.735820 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:09.734737 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be2ffe6a-d355-4d95-a7ae-46adeb952736-kube-api-access-dmd6k\" (UniqueName: \"kubernetes.io/projected/be2ffe6a-d355-4d95-a7ae-46adeb952736-kube-api-access-dmd6k\") pod \"be2ffe6a-d355-4d95-a7ae-46adeb952736\" (UID: \"be2ffe6a-d355-4d95-a7ae-46adeb952736\") " Oct 30 20:09:09.735820 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:09.734856 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be2ffe6a-d355-4d95-a7ae-46adeb952736-metrics-certs\" (UniqueName: \"kubernetes.io/secret/be2ffe6a-d355-4d95-a7ae-46adeb952736-metrics-certs\") pod \"be2ffe6a-d355-4d95-a7ae-46adeb952736\" (UID: \"be2ffe6a-d355-4d95-a7ae-46adeb952736\") " Oct 30 20:09:09.735820 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:09.734897 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be2ffe6a-d355-4d95-a7ae-46adeb952736-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be2ffe6a-d355-4d95-a7ae-46adeb952736-maas-api-tls\") pod \"be2ffe6a-d355-4d95-a7ae-46adeb952736\" (UID: \"be2ffe6a-d355-4d95-a7ae-46adeb952736\") " Oct 30 20:09:09.737184 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:09.736487 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be2ffe6a-d355-4d95-a7ae-46adeb952736-openshift-service-ca" pod "be2ffe6a-d355-4d95-a7ae-46adeb952736" (UID: "be2ffe6a-d355-4d95-a7ae-46adeb952736"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:09:09.741054 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:09.741007 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be2ffe6a-d355-4d95-a7ae-46adeb952736-metrics-certs" pod "be2ffe6a-d355-4d95-a7ae-46adeb952736" (UID: "be2ffe6a-d355-4d95-a7ae-46adeb952736"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:09:09.743204 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:09.743154 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be2ffe6a-d355-4d95-a7ae-46adeb952736-maas-api-tls" pod "be2ffe6a-d355-4d95-a7ae-46adeb952736" (UID: "be2ffe6a-d355-4d95-a7ae-46adeb952736"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:09:09.743939 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:09.743786 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be2ffe6a-d355-4d95-a7ae-46adeb952736-kube-api-access-dmd6k" pod "be2ffe6a-d355-4d95-a7ae-46adeb952736" (UID: "be2ffe6a-d355-4d95-a7ae-46adeb952736"). InnerVolumeSpecName "kube-api-access-dmd6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:09.836198 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:09.836166 5088 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be2ffe6a-d355-4d95-a7ae-46adeb952736-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:09.836198 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:09.836198 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be2ffe6a-d355-4d95-a7ae-46adeb952736-maas-api-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:09.836652 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:09.836211 5088 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be2ffe6a-d355-4d95-a7ae-46adeb952736-openshift-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:09.836652 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:09.836220 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmd6k\" (UniqueName: \"kubernetes.io/projected/be2ffe6a-d355-4d95-a7ae-46adeb952736-kube-api-access-dmd6k\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:10.375539 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:10.375500 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957" Oct 30 20:09:10.375539 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:10.375532 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957" event={"ID":"be2ffe6a-d355-4d95-a7ae-46adeb952736","Type":"ContainerDied","Data":"45fca52d914b3c98643cfa4ecdadc6d423319c7f201f72c2759a25d95984ec1c"} Oct 30 20:09:10.375780 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:10.375593 5088 scope.go:122] "RemoveContainer" containerID="f72efc6e822122441efed9c58d4563ca1068b6453cbace028b5bf3b95a20c799" Oct 30 20:09:10.440736 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:10.440699 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957"] Oct 30 20:09:10.454345 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:10.454308 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-51dd62-58b8fdb648-74957"] Oct 30 20:09:10.746349 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:10.746312 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92e0bb64-54a3-4fb9-8423-c86b5c21f241" path="/var/lib/kubelet/pods/92e0bb64-54a3-4fb9-8423-c86b5c21f241/volumes" Oct 30 20:09:10.746869 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:10.746848 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be2ffe6a-d355-4d95-a7ae-46adeb952736" path="/var/lib/kubelet/pods/be2ffe6a-d355-4d95-a7ae-46adeb952736/volumes" Oct 30 20:09:11.304079 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:11.304030 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" podUID="fec493c7-3f5a-459e-99e0-a4c816d5a907" containerName="main" containerID="cri-o://ac82bf60d85161dcf77d16cf1dd8fdc013432c2b30ea62b68562d08e33b9ebfe" gracePeriod=45 Oct 30 20:09:11.579289 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:11.579259 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" Oct 30 20:09:11.657594 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:11.657566 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fec493c7-3f5a-459e-99e0-a4c816d5a907-tls-certs\" (UniqueName: \"kubernetes.io/secret/fec493c7-3f5a-459e-99e0-a4c816d5a907-tls-certs\") pod \"fec493c7-3f5a-459e-99e0-a4c816d5a907\" (UID: \"fec493c7-3f5a-459e-99e0-a4c816d5a907\") " Oct 30 20:09:11.657796 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:11.657679 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fec493c7-3f5a-459e-99e0-a4c816d5a907-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fec493c7-3f5a-459e-99e0-a4c816d5a907-model-cache\") pod \"fec493c7-3f5a-459e-99e0-a4c816d5a907\" (UID: \"fec493c7-3f5a-459e-99e0-a4c816d5a907\") " Oct 30 20:09:11.657796 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:11.657760 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fec493c7-3f5a-459e-99e0-a4c816d5a907-kube-api-access-vg882\" (UniqueName: \"kubernetes.io/projected/fec493c7-3f5a-459e-99e0-a4c816d5a907-kube-api-access-vg882\") pod \"fec493c7-3f5a-459e-99e0-a4c816d5a907\" (UID: \"fec493c7-3f5a-459e-99e0-a4c816d5a907\") " Oct 30 20:09:11.657796 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:11.657791 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fec493c7-3f5a-459e-99e0-a4c816d5a907-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fec493c7-3f5a-459e-99e0-a4c816d5a907-dshm\") pod \"fec493c7-3f5a-459e-99e0-a4c816d5a907\" (UID: \"fec493c7-3f5a-459e-99e0-a4c816d5a907\") " Oct 30 20:09:11.657962 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:11.657835 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fec493c7-3f5a-459e-99e0-a4c816d5a907-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fec493c7-3f5a-459e-99e0-a4c816d5a907-tmp-dir\") pod \"fec493c7-3f5a-459e-99e0-a4c816d5a907\" (UID: \"fec493c7-3f5a-459e-99e0-a4c816d5a907\") " Oct 30 20:09:11.658022 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:11.657970 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fec493c7-3f5a-459e-99e0-a4c816d5a907-home\" (UniqueName: \"kubernetes.io/empty-dir/fec493c7-3f5a-459e-99e0-a4c816d5a907-home\") pod \"fec493c7-3f5a-459e-99e0-a4c816d5a907\" (UID: \"fec493c7-3f5a-459e-99e0-a4c816d5a907\") " Oct 30 20:09:11.658605 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:11.658581 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fec493c7-3f5a-459e-99e0-a4c816d5a907-home" pod "fec493c7-3f5a-459e-99e0-a4c816d5a907" (UID: "fec493c7-3f5a-459e-99e0-a4c816d5a907"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:11.658800 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:11.658781 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fec493c7-3f5a-459e-99e0-a4c816d5a907-model-cache" pod "fec493c7-3f5a-459e-99e0-a4c816d5a907" (UID: "fec493c7-3f5a-459e-99e0-a4c816d5a907"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:11.659351 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:11.659328 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fec493c7-3f5a-459e-99e0-a4c816d5a907-tmp-dir" pod "fec493c7-3f5a-459e-99e0-a4c816d5a907" (UID: "fec493c7-3f5a-459e-99e0-a4c816d5a907"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:11.661663 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:11.661642 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fec493c7-3f5a-459e-99e0-a4c816d5a907-tls-certs" pod "fec493c7-3f5a-459e-99e0-a4c816d5a907" (UID: "fec493c7-3f5a-459e-99e0-a4c816d5a907"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:09:11.661991 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:11.661965 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fec493c7-3f5a-459e-99e0-a4c816d5a907-kube-api-access-vg882" pod "fec493c7-3f5a-459e-99e0-a4c816d5a907" (UID: "fec493c7-3f5a-459e-99e0-a4c816d5a907"). InnerVolumeSpecName "kube-api-access-vg882". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:11.662589 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:11.662568 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fec493c7-3f5a-459e-99e0-a4c816d5a907-dshm" pod "fec493c7-3f5a-459e-99e0-a4c816d5a907" (UID: "fec493c7-3f5a-459e-99e0-a4c816d5a907"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:11.759033 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:11.758998 5088 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fec493c7-3f5a-459e-99e0-a4c816d5a907-dshm\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:11.759033 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:11.759027 5088 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fec493c7-3f5a-459e-99e0-a4c816d5a907-tmp-dir\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:11.759033 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:11.759037 5088 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fec493c7-3f5a-459e-99e0-a4c816d5a907-home\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:11.759289 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:11.759049 5088 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fec493c7-3f5a-459e-99e0-a4c816d5a907-tls-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:11.759289 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:11.759057 5088 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fec493c7-3f5a-459e-99e0-a4c816d5a907-model-cache\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:11.759289 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:11.759065 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg882\" (UniqueName: \"kubernetes.io/projected/fec493c7-3f5a-459e-99e0-a4c816d5a907-kube-api-access-vg882\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:12.393849 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:12.393813 5088 generic.go:356] "Generic (PLEG): container finished" podID="fec493c7-3f5a-459e-99e0-a4c816d5a907" containerID="ac82bf60d85161dcf77d16cf1dd8fdc013432c2b30ea62b68562d08e33b9ebfe" exitCode=0 Oct 30 20:09:12.394298 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:12.393865 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" Oct 30 20:09:12.394298 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:12.393896 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" event={"ID":"fec493c7-3f5a-459e-99e0-a4c816d5a907","Type":"ContainerDied","Data":"ac82bf60d85161dcf77d16cf1dd8fdc013432c2b30ea62b68562d08e33b9ebfe"} Oct 30 20:09:12.394298 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:12.393947 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs" event={"ID":"fec493c7-3f5a-459e-99e0-a4c816d5a907","Type":"ContainerDied","Data":"06afed01b949ca08c765a19d5e4ae7e6f7ac6f831b6b61480d34d26e42444a8b"} Oct 30 20:09:12.394298 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:12.393965 5088 scope.go:122] "RemoveContainer" containerID="ac82bf60d85161dcf77d16cf1dd8fdc013432c2b30ea62b68562d08e33b9ebfe" Oct 30 20:09:12.405434 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:12.405407 5088 scope.go:122] "RemoveContainer" containerID="ac82bf60d85161dcf77d16cf1dd8fdc013432c2b30ea62b68562d08e33b9ebfe" Oct 30 20:09:12.405865 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:09:12.405839 5088 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac82bf60d85161dcf77d16cf1dd8fdc013432c2b30ea62b68562d08e33b9ebfe\": container with ID starting with ac82bf60d85161dcf77d16cf1dd8fdc013432c2b30ea62b68562d08e33b9ebfe not found: ID does not exist" containerID="ac82bf60d85161dcf77d16cf1dd8fdc013432c2b30ea62b68562d08e33b9ebfe" Oct 30 20:09:12.405946 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:12.405875 5088 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac82bf60d85161dcf77d16cf1dd8fdc013432c2b30ea62b68562d08e33b9ebfe"} err="failed to get container status \"ac82bf60d85161dcf77d16cf1dd8fdc013432c2b30ea62b68562d08e33b9ebfe\": rpc error: code = NotFound desc = could not find container \"ac82bf60d85161dcf77d16cf1dd8fdc013432c2b30ea62b68562d08e33b9ebfe\": container with ID starting with ac82bf60d85161dcf77d16cf1dd8fdc013432c2b30ea62b68562d08e33b9ebfe not found: ID does not exist" Oct 30 20:09:12.473754 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:12.473695 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs"] Oct 30 20:09:12.486296 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:12.486267 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-efbac3/distinct-2-w2-efbac3-kserve-859fcccbd-2t8hs"] Oct 30 20:09:12.533066 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:12.533022 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-51dd62-5668978b9c-5djrs" podUID="2f2223b9-aebe-4aab-9437-af457ece8c69" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:09:12.533227 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:12.533022 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-51dd62-79f5b56fd7-ltqtf" podUID="ab7bcd6d-197b-4c1e-9ce1-4aa111e96e54" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:09:12.739173 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:12.739136 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fec493c7-3f5a-459e-99e0-a4c816d5a907" path="/var/lib/kubelet/pods/fec493c7-3f5a-459e-99e0-a4c816d5a907/volumes" Oct 30 20:09:16.373296 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:16.373261 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-51dd62-5668978b9c-5djrs" Oct 30 20:09:16.409340 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:16.409312 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f2223b9-aebe-4aab-9437-af457ece8c69-kube-api-access-pktm7\" (UniqueName: \"kubernetes.io/projected/2f2223b9-aebe-4aab-9437-af457ece8c69-kube-api-access-pktm7\") pod \"2f2223b9-aebe-4aab-9437-af457ece8c69\" (UID: \"2f2223b9-aebe-4aab-9437-af457ece8c69\") " Oct 30 20:09:16.409520 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:16.409377 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f2223b9-aebe-4aab-9437-af457ece8c69-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f2223b9-aebe-4aab-9437-af457ece8c69-plugins-config-volume\") pod \"2f2223b9-aebe-4aab-9437-af457ece8c69\" (UID: \"2f2223b9-aebe-4aab-9437-af457ece8c69\") " Oct 30 20:09:16.409520 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:16.409400 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-51dd62-79f5b56fd7-ltqtf" Oct 30 20:09:16.409813 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:16.409791 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f2223b9-aebe-4aab-9437-af457ece8c69-plugins-config-volume" pod "2f2223b9-aebe-4aab-9437-af457ece8c69" (UID: "2f2223b9-aebe-4aab-9437-af457ece8c69"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:09:16.411387 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:16.411367 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f2223b9-aebe-4aab-9437-af457ece8c69-kube-api-access-pktm7" pod "2f2223b9-aebe-4aab-9437-af457ece8c69" (UID: "2f2223b9-aebe-4aab-9437-af457ece8c69"). InnerVolumeSpecName "kube-api-access-pktm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:16.424831 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:16.424800 5088 generic.go:356] "Generic (PLEG): container finished" podID="2f2223b9-aebe-4aab-9437-af457ece8c69" containerID="a9b6f46f57963859ae28c7ec7bdbf404c5ec2de1c93b2e24e47bb988b171a3ca" exitCode=0 Oct 30 20:09:16.424986 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:16.424858 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-51dd62-5668978b9c-5djrs" Oct 30 20:09:16.424986 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:16.424878 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-51dd62-5668978b9c-5djrs" event={"ID":"2f2223b9-aebe-4aab-9437-af457ece8c69","Type":"ContainerDied","Data":"a9b6f46f57963859ae28c7ec7bdbf404c5ec2de1c93b2e24e47bb988b171a3ca"} Oct 30 20:09:16.424986 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:16.424972 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-51dd62-5668978b9c-5djrs" event={"ID":"2f2223b9-aebe-4aab-9437-af457ece8c69","Type":"ContainerDied","Data":"525b04fff0d6de0fcb97a0349c86f38750f52b631521c17c169eee1dfca2d29f"} Oct 30 20:09:16.425167 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:16.425004 5088 scope.go:122] "RemoveContainer" containerID="a9b6f46f57963859ae28c7ec7bdbf404c5ec2de1c93b2e24e47bb988b171a3ca" Oct 30 20:09:16.426519 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:16.426498 5088 generic.go:356] "Generic (PLEG): container finished" podID="ab7bcd6d-197b-4c1e-9ce1-4aa111e96e54" containerID="36e4fdc76cc5e54e943202cd1add398d8d06a6e1797847f44f6ad6c8daa3b250" exitCode=0 Oct 30 20:09:16.426646 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:16.426547 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-51dd62-79f5b56fd7-ltqtf" Oct 30 20:09:16.426646 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:16.426567 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-51dd62-79f5b56fd7-ltqtf" event={"ID":"ab7bcd6d-197b-4c1e-9ce1-4aa111e96e54","Type":"ContainerDied","Data":"36e4fdc76cc5e54e943202cd1add398d8d06a6e1797847f44f6ad6c8daa3b250"} Oct 30 20:09:16.426646 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:16.426607 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-51dd62-79f5b56fd7-ltqtf" event={"ID":"ab7bcd6d-197b-4c1e-9ce1-4aa111e96e54","Type":"ContainerDied","Data":"d2eb2124934140b1002aa7b022fecbcdbabba1bafdc94a9afdd430651f8f7228"} Oct 30 20:09:16.436261 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:16.436237 5088 scope.go:122] "RemoveContainer" containerID="a9b6f46f57963859ae28c7ec7bdbf404c5ec2de1c93b2e24e47bb988b171a3ca" Oct 30 20:09:16.436650 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:09:16.436598 5088 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9b6f46f57963859ae28c7ec7bdbf404c5ec2de1c93b2e24e47bb988b171a3ca\": container with ID starting with a9b6f46f57963859ae28c7ec7bdbf404c5ec2de1c93b2e24e47bb988b171a3ca not found: ID does not exist" containerID="a9b6f46f57963859ae28c7ec7bdbf404c5ec2de1c93b2e24e47bb988b171a3ca" Oct 30 20:09:16.436755 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:16.436657 5088 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9b6f46f57963859ae28c7ec7bdbf404c5ec2de1c93b2e24e47bb988b171a3ca"} err="failed to get container status \"a9b6f46f57963859ae28c7ec7bdbf404c5ec2de1c93b2e24e47bb988b171a3ca\": rpc error: code = NotFound desc = could not find container \"a9b6f46f57963859ae28c7ec7bdbf404c5ec2de1c93b2e24e47bb988b171a3ca\": container with ID starting with a9b6f46f57963859ae28c7ec7bdbf404c5ec2de1c93b2e24e47bb988b171a3ca not found: ID does not exist" Oct 30 20:09:16.436755 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:16.436681 5088 scope.go:122] "RemoveContainer" containerID="36e4fdc76cc5e54e943202cd1add398d8d06a6e1797847f44f6ad6c8daa3b250" Oct 30 20:09:16.447832 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:16.447796 5088 scope.go:122] "RemoveContainer" containerID="36e4fdc76cc5e54e943202cd1add398d8d06a6e1797847f44f6ad6c8daa3b250" Oct 30 20:09:16.454864 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:09:16.454834 5088 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36e4fdc76cc5e54e943202cd1add398d8d06a6e1797847f44f6ad6c8daa3b250\": container with ID starting with 36e4fdc76cc5e54e943202cd1add398d8d06a6e1797847f44f6ad6c8daa3b250 not found: ID does not exist" containerID="36e4fdc76cc5e54e943202cd1add398d8d06a6e1797847f44f6ad6c8daa3b250" Oct 30 20:09:16.455084 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:16.455049 5088 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36e4fdc76cc5e54e943202cd1add398d8d06a6e1797847f44f6ad6c8daa3b250"} err="failed to get container status \"36e4fdc76cc5e54e943202cd1add398d8d06a6e1797847f44f6ad6c8daa3b250\": rpc error: code = NotFound desc = could not find container \"36e4fdc76cc5e54e943202cd1add398d8d06a6e1797847f44f6ad6c8daa3b250\": container with ID starting with 36e4fdc76cc5e54e943202cd1add398d8d06a6e1797847f44f6ad6c8daa3b250 not found: ID does not exist" Oct 30 20:09:16.455599 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:16.455569 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-efbac3-66db987bc9-ptftf" podUID="0ee0e296-2cf3-4cdc-9340-d202e75c6a14" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:09:16.510131 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:16.510100 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab7bcd6d-197b-4c1e-9ce1-4aa111e96e54-kube-api-access-rx95j\" (UniqueName: \"kubernetes.io/projected/ab7bcd6d-197b-4c1e-9ce1-4aa111e96e54-kube-api-access-rx95j\") pod \"ab7bcd6d-197b-4c1e-9ce1-4aa111e96e54\" (UID: \"ab7bcd6d-197b-4c1e-9ce1-4aa111e96e54\") " Oct 30 20:09:16.510321 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:16.510167 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab7bcd6d-197b-4c1e-9ce1-4aa111e96e54-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab7bcd6d-197b-4c1e-9ce1-4aa111e96e54-plugins-config-volume\") pod \"ab7bcd6d-197b-4c1e-9ce1-4aa111e96e54\" (UID: \"ab7bcd6d-197b-4c1e-9ce1-4aa111e96e54\") " Oct 30 20:09:16.510552 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:16.510538 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pktm7\" (UniqueName: \"kubernetes.io/projected/2f2223b9-aebe-4aab-9437-af457ece8c69-kube-api-access-pktm7\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:16.510601 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:16.510554 5088 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f2223b9-aebe-4aab-9437-af457ece8c69-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:16.510696 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:16.510667 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab7bcd6d-197b-4c1e-9ce1-4aa111e96e54-plugins-config-volume" pod "ab7bcd6d-197b-4c1e-9ce1-4aa111e96e54" (UID: "ab7bcd6d-197b-4c1e-9ce1-4aa111e96e54"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:09:16.512345 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:16.512321 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab7bcd6d-197b-4c1e-9ce1-4aa111e96e54-kube-api-access-rx95j" pod "ab7bcd6d-197b-4c1e-9ce1-4aa111e96e54" (UID: "ab7bcd6d-197b-4c1e-9ce1-4aa111e96e54"). InnerVolumeSpecName "kube-api-access-rx95j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:16.547015 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:16.546981 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-51dd62-5668978b9c-5djrs"] Oct 30 20:09:16.554935 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:16.554908 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-51dd62-5668978b9c-5djrs"] Oct 30 20:09:16.611407 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:16.611326 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx95j\" (UniqueName: \"kubernetes.io/projected/ab7bcd6d-197b-4c1e-9ce1-4aa111e96e54-kube-api-access-rx95j\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:16.611407 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:16.611354 5088 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab7bcd6d-197b-4c1e-9ce1-4aa111e96e54-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:16.739172 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:16.739139 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f2223b9-aebe-4aab-9437-af457ece8c69" path="/var/lib/kubelet/pods/2f2223b9-aebe-4aab-9437-af457ece8c69/volumes" Oct 30 20:09:16.781429 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:16.781392 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-51dd62-79f5b56fd7-ltqtf"] Oct 30 20:09:16.786535 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:16.786505 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-51dd62-79f5b56fd7-ltqtf"] Oct 30 20:09:18.738921 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:18.738889 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab7bcd6d-197b-4c1e-9ce1-4aa111e96e54" path="/var/lib/kubelet/pods/ab7bcd6d-197b-4c1e-9ce1-4aa111e96e54/volumes" Oct 30 20:09:19.717008 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:19.716973 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf"] Oct 30 20:09:19.717204 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:19.717188 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" podUID="c0748781-63b9-4af1-8eab-5d0cef276da6" containerName="istio-proxy" containerID="cri-o://cf771118fedc5389d16f43ca5d7c4e0f0b24d2193c33d1c77df884e756cd9c0f" gracePeriod=30 Oct 30 20:09:19.730720 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:19.730688 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g"] Oct 30 20:09:19.730909 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:19.730888 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" podUID="1f2b960b-8db8-4b84-b6d1-9ac12823e278" containerName="istio-proxy" containerID="cri-o://ac6ef7d6f2e1e074bfa2ccd91e5df5aacd6d4d99966f02c65d29e197fcc55de3" gracePeriod=30 Oct 30 20:09:19.744921 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:19.744886 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h"] Oct 30 20:09:19.745337 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:19.745137 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" podUID="e402d580-e9a6-4e35-a2ff-e5ef2d78eb90" containerName="istio-proxy" containerID="cri-o://c681ce095c069d984ee329fe861fc5299eb161f9aeb2b45159bb8880bd808928" gracePeriod=30 Oct 30 20:09:20.978773 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:20.978726 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:09:22.437869 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:09:22.437825 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:09:22.659101 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:22.659064 5088 patch_prober.go:29] interesting pod/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:09:22.659270 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:22.659134 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" podUID="e402d580-e9a6-4e35-a2ff-e5ef2d78eb90" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:09:25.050184 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.050149 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" Oct 30 20:09:25.076441 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.076412 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" Oct 30 20:09:25.116370 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.116337 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" Oct 30 20:09:25.140579 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.140544 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f2b960b-8db8-4b84-b6d1-9ac12823e278-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f2b960b-8db8-4b84-b6d1-9ac12823e278-istio-podinfo\") pod \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\" (UID: \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\") " Oct 30 20:09:25.140796 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.140598 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f2b960b-8db8-4b84-b6d1-9ac12823e278-kube-api-access-zrcb9\" (UniqueName: \"kubernetes.io/projected/1f2b960b-8db8-4b84-b6d1-9ac12823e278-kube-api-access-zrcb9\") pod \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\" (UID: \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\") " Oct 30 20:09:25.140796 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.140661 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f2b960b-8db8-4b84-b6d1-9ac12823e278-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f2b960b-8db8-4b84-b6d1-9ac12823e278-istiod-ca-cert\") pod \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\" (UID: \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\") " Oct 30 20:09:25.140796 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.140696 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f2b960b-8db8-4b84-b6d1-9ac12823e278-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f2b960b-8db8-4b84-b6d1-9ac12823e278-credential-socket\") pod \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\" (UID: \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\") " Oct 30 20:09:25.140796 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.140752 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f2b960b-8db8-4b84-b6d1-9ac12823e278-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f2b960b-8db8-4b84-b6d1-9ac12823e278-istio-data\") pod \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\" (UID: \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\") " Oct 30 20:09:25.140796 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.140784 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f2b960b-8db8-4b84-b6d1-9ac12823e278-istio-token\" (UniqueName: \"kubernetes.io/projected/1f2b960b-8db8-4b84-b6d1-9ac12823e278-istio-token\") pod \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\" (UID: \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\") " Oct 30 20:09:25.141095 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.140828 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0748781-63b9-4af1-8eab-5d0cef276da6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0748781-63b9-4af1-8eab-5d0cef276da6-istiod-ca-cert\") pod \"c0748781-63b9-4af1-8eab-5d0cef276da6\" (UID: \"c0748781-63b9-4af1-8eab-5d0cef276da6\") " Oct 30 20:09:25.141095 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.140875 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0748781-63b9-4af1-8eab-5d0cef276da6-kube-api-access-k9lqb\" (UniqueName: \"kubernetes.io/projected/c0748781-63b9-4af1-8eab-5d0cef276da6-kube-api-access-k9lqb\") pod \"c0748781-63b9-4af1-8eab-5d0cef276da6\" (UID: \"c0748781-63b9-4af1-8eab-5d0cef276da6\") " Oct 30 20:09:25.141095 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.140912 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0748781-63b9-4af1-8eab-5d0cef276da6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0748781-63b9-4af1-8eab-5d0cef276da6-istio-podinfo\") pod \"c0748781-63b9-4af1-8eab-5d0cef276da6\" (UID: \"c0748781-63b9-4af1-8eab-5d0cef276da6\") " Oct 30 20:09:25.141095 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.140995 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0748781-63b9-4af1-8eab-5d0cef276da6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0748781-63b9-4af1-8eab-5d0cef276da6-workload-socket\") pod \"c0748781-63b9-4af1-8eab-5d0cef276da6\" (UID: \"c0748781-63b9-4af1-8eab-5d0cef276da6\") " Oct 30 20:09:25.141095 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.141039 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f2b960b-8db8-4b84-b6d1-9ac12823e278-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f2b960b-8db8-4b84-b6d1-9ac12823e278-istio-envoy\") pod \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\" (UID: \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\") " Oct 30 20:09:25.141095 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.141055 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f2b960b-8db8-4b84-b6d1-9ac12823e278-credential-socket" pod "1f2b960b-8db8-4b84-b6d1-9ac12823e278" (UID: "1f2b960b-8db8-4b84-b6d1-9ac12823e278"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:25.141412 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.141071 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f2b960b-8db8-4b84-b6d1-9ac12823e278-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f2b960b-8db8-4b84-b6d1-9ac12823e278-workload-socket\") pod \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\" (UID: \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\") " Oct 30 20:09:25.141412 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.141162 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0748781-63b9-4af1-8eab-5d0cef276da6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0748781-63b9-4af1-8eab-5d0cef276da6-istio-envoy\") pod \"c0748781-63b9-4af1-8eab-5d0cef276da6\" (UID: \"c0748781-63b9-4af1-8eab-5d0cef276da6\") " Oct 30 20:09:25.141412 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.141164 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f2b960b-8db8-4b84-b6d1-9ac12823e278-istio-data" pod "1f2b960b-8db8-4b84-b6d1-9ac12823e278" (UID: "1f2b960b-8db8-4b84-b6d1-9ac12823e278"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:25.141412 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.141192 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f2b960b-8db8-4b84-b6d1-9ac12823e278-istiod-ca-cert" pod "1f2b960b-8db8-4b84-b6d1-9ac12823e278" (UID: "1f2b960b-8db8-4b84-b6d1-9ac12823e278"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:09:25.141412 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.141303 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0748781-63b9-4af1-8eab-5d0cef276da6-istiod-ca-cert" pod "c0748781-63b9-4af1-8eab-5d0cef276da6" (UID: "c0748781-63b9-4af1-8eab-5d0cef276da6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:09:25.141696 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.141452 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0748781-63b9-4af1-8eab-5d0cef276da6-workload-socket" pod "c0748781-63b9-4af1-8eab-5d0cef276da6" (UID: "c0748781-63b9-4af1-8eab-5d0cef276da6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:25.141696 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.141573 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0748781-63b9-4af1-8eab-5d0cef276da6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0748781-63b9-4af1-8eab-5d0cef276da6-credential-socket\") pod \"c0748781-63b9-4af1-8eab-5d0cef276da6\" (UID: \"c0748781-63b9-4af1-8eab-5d0cef276da6\") " Oct 30 20:09:25.141696 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.141598 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f2b960b-8db8-4b84-b6d1-9ac12823e278-workload-socket" pod "1f2b960b-8db8-4b84-b6d1-9ac12823e278" (UID: "1f2b960b-8db8-4b84-b6d1-9ac12823e278"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:25.141696 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.141658 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0748781-63b9-4af1-8eab-5d0cef276da6-istio-token\" (UniqueName: \"kubernetes.io/projected/c0748781-63b9-4af1-8eab-5d0cef276da6-istio-token\") pod \"c0748781-63b9-4af1-8eab-5d0cef276da6\" (UID: \"c0748781-63b9-4af1-8eab-5d0cef276da6\") " Oct 30 20:09:25.141696 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.141697 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0748781-63b9-4af1-8eab-5d0cef276da6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0748781-63b9-4af1-8eab-5d0cef276da6-istio-data\") pod \"c0748781-63b9-4af1-8eab-5d0cef276da6\" (UID: \"c0748781-63b9-4af1-8eab-5d0cef276da6\") " Oct 30 20:09:25.141982 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.141734 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0748781-63b9-4af1-8eab-5d0cef276da6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0748781-63b9-4af1-8eab-5d0cef276da6-workload-certs\") pod \"c0748781-63b9-4af1-8eab-5d0cef276da6\" (UID: \"c0748781-63b9-4af1-8eab-5d0cef276da6\") " Oct 30 20:09:25.141982 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.141793 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f2b960b-8db8-4b84-b6d1-9ac12823e278-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f2b960b-8db8-4b84-b6d1-9ac12823e278-workload-certs\") pod \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\" (UID: \"1f2b960b-8db8-4b84-b6d1-9ac12823e278\") " Oct 30 20:09:25.141982 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.141915 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0748781-63b9-4af1-8eab-5d0cef276da6-credential-socket" pod "c0748781-63b9-4af1-8eab-5d0cef276da6" (UID: "c0748781-63b9-4af1-8eab-5d0cef276da6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:25.142205 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.142090 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0748781-63b9-4af1-8eab-5d0cef276da6-istio-data" pod "c0748781-63b9-4af1-8eab-5d0cef276da6" (UID: "c0748781-63b9-4af1-8eab-5d0cef276da6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:25.142319 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.142294 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0748781-63b9-4af1-8eab-5d0cef276da6-workload-certs" pod "c0748781-63b9-4af1-8eab-5d0cef276da6" (UID: "c0748781-63b9-4af1-8eab-5d0cef276da6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:25.142383 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.142322 5088 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f2b960b-8db8-4b84-b6d1-9ac12823e278-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:25.142383 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.142342 5088 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0748781-63b9-4af1-8eab-5d0cef276da6-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:25.142383 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.142356 5088 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0748781-63b9-4af1-8eab-5d0cef276da6-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:25.142383 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.142371 5088 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f2b960b-8db8-4b84-b6d1-9ac12823e278-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:25.142600 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.142384 5088 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0748781-63b9-4af1-8eab-5d0cef276da6-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:25.142600 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.142398 5088 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0748781-63b9-4af1-8eab-5d0cef276da6-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:25.142600 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.142409 5088 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f2b960b-8db8-4b84-b6d1-9ac12823e278-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:25.142600 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.142421 5088 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f2b960b-8db8-4b84-b6d1-9ac12823e278-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:25.142600 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.142494 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f2b960b-8db8-4b84-b6d1-9ac12823e278-workload-certs" pod "1f2b960b-8db8-4b84-b6d1-9ac12823e278" (UID: "1f2b960b-8db8-4b84-b6d1-9ac12823e278"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:25.144582 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.144558 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f2b960b-8db8-4b84-b6d1-9ac12823e278-istio-podinfo" pod "1f2b960b-8db8-4b84-b6d1-9ac12823e278" (UID: "1f2b960b-8db8-4b84-b6d1-9ac12823e278"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:09:25.145073 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.145052 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f2b960b-8db8-4b84-b6d1-9ac12823e278-istio-envoy" pod "1f2b960b-8db8-4b84-b6d1-9ac12823e278" (UID: "1f2b960b-8db8-4b84-b6d1-9ac12823e278"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:25.145165 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.145086 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f2b960b-8db8-4b84-b6d1-9ac12823e278-istio-token" pod "1f2b960b-8db8-4b84-b6d1-9ac12823e278" (UID: "1f2b960b-8db8-4b84-b6d1-9ac12823e278"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:25.145207 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.145169 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0748781-63b9-4af1-8eab-5d0cef276da6-istio-podinfo" pod "c0748781-63b9-4af1-8eab-5d0cef276da6" (UID: "c0748781-63b9-4af1-8eab-5d0cef276da6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:09:25.145365 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.145345 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f2b960b-8db8-4b84-b6d1-9ac12823e278-kube-api-access-zrcb9" pod "1f2b960b-8db8-4b84-b6d1-9ac12823e278" (UID: "1f2b960b-8db8-4b84-b6d1-9ac12823e278"). InnerVolumeSpecName "kube-api-access-zrcb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:25.145472 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.145450 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0748781-63b9-4af1-8eab-5d0cef276da6-istio-envoy" pod "c0748781-63b9-4af1-8eab-5d0cef276da6" (UID: "c0748781-63b9-4af1-8eab-5d0cef276da6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:25.145508 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.145456 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0748781-63b9-4af1-8eab-5d0cef276da6-istio-token" pod "c0748781-63b9-4af1-8eab-5d0cef276da6" (UID: "c0748781-63b9-4af1-8eab-5d0cef276da6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:25.145765 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.145745 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0748781-63b9-4af1-8eab-5d0cef276da6-kube-api-access-k9lqb" pod "c0748781-63b9-4af1-8eab-5d0cef276da6" (UID: "c0748781-63b9-4af1-8eab-5d0cef276da6"). InnerVolumeSpecName "kube-api-access-k9lqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:25.243601 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.243570 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-istio-envoy\") pod \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\" (UID: \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\") " Oct 30 20:09:25.243834 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.243651 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-istio-data\") pod \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\" (UID: \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\") " Oct 30 20:09:25.243834 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.243683 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-workload-socket\") pod \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\" (UID: \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\") " Oct 30 20:09:25.243834 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.243718 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-kube-api-access-rcvs2\" (UniqueName: \"kubernetes.io/projected/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-kube-api-access-rcvs2\") pod \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\" (UID: \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\") " Oct 30 20:09:25.243834 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.243759 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-workload-certs\") pod \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\" (UID: \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\") " Oct 30 20:09:25.243834 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.243796 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-istiod-ca-cert\") pod \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\" (UID: \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\") " Oct 30 20:09:25.244133 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.243839 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-credential-socket\") pod \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\" (UID: \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\") " Oct 30 20:09:25.244133 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.244048 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-istio-token\" (UniqueName: \"kubernetes.io/projected/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-istio-token\") pod \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\" (UID: \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\") " Oct 30 20:09:25.244133 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.244121 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-istio-podinfo\") pod \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\" (UID: \"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90\") " Oct 30 20:09:25.244465 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.244436 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-workload-socket" pod "e402d580-e9a6-4e35-a2ff-e5ef2d78eb90" (UID: "e402d580-e9a6-4e35-a2ff-e5ef2d78eb90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:25.244553 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.244470 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9lqb\" (UniqueName: \"kubernetes.io/projected/c0748781-63b9-4af1-8eab-5d0cef276da6-kube-api-access-k9lqb\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:25.244553 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.244492 5088 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0748781-63b9-4af1-8eab-5d0cef276da6-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:25.244553 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.244507 5088 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f2b960b-8db8-4b84-b6d1-9ac12823e278-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:25.244553 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.244521 5088 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0748781-63b9-4af1-8eab-5d0cef276da6-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:25.244553 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.244533 5088 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0748781-63b9-4af1-8eab-5d0cef276da6-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:25.244553 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.244546 5088 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0748781-63b9-4af1-8eab-5d0cef276da6-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:25.244931 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.244558 5088 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f2b960b-8db8-4b84-b6d1-9ac12823e278-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:25.244931 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.244571 5088 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f2b960b-8db8-4b84-b6d1-9ac12823e278-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:25.244931 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.244585 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrcb9\" (UniqueName: \"kubernetes.io/projected/1f2b960b-8db8-4b84-b6d1-9ac12823e278-kube-api-access-zrcb9\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:25.244931 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.244599 5088 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f2b960b-8db8-4b84-b6d1-9ac12823e278-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:25.245186 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.244670 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-istio-data" pod "e402d580-e9a6-4e35-a2ff-e5ef2d78eb90" (UID: "e402d580-e9a6-4e35-a2ff-e5ef2d78eb90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:25.245262 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.244844 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-workload-certs" pod "e402d580-e9a6-4e35-a2ff-e5ef2d78eb90" (UID: "e402d580-e9a6-4e35-a2ff-e5ef2d78eb90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:25.245262 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.244863 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-credential-socket" pod "e402d580-e9a6-4e35-a2ff-e5ef2d78eb90" (UID: "e402d580-e9a6-4e35-a2ff-e5ef2d78eb90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:25.245262 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.245085 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-istiod-ca-cert" pod "e402d580-e9a6-4e35-a2ff-e5ef2d78eb90" (UID: "e402d580-e9a6-4e35-a2ff-e5ef2d78eb90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:09:25.246852 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.246814 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-istio-envoy" pod "e402d580-e9a6-4e35-a2ff-e5ef2d78eb90" (UID: "e402d580-e9a6-4e35-a2ff-e5ef2d78eb90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:25.247175 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.247149 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-kube-api-access-rcvs2" pod "e402d580-e9a6-4e35-a2ff-e5ef2d78eb90" (UID: "e402d580-e9a6-4e35-a2ff-e5ef2d78eb90"). InnerVolumeSpecName "kube-api-access-rcvs2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:25.247294 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.247272 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-istio-token" pod "e402d580-e9a6-4e35-a2ff-e5ef2d78eb90" (UID: "e402d580-e9a6-4e35-a2ff-e5ef2d78eb90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:25.248989 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.248962 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-istio-podinfo" pod "e402d580-e9a6-4e35-a2ff-e5ef2d78eb90" (UID: "e402d580-e9a6-4e35-a2ff-e5ef2d78eb90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:09:25.346707 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.346600 5088 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:25.346707 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.346654 5088 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:25.346707 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.346665 5088 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:25.346707 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.346673 5088 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:25.346707 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.346682 5088 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:25.346707 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.346690 5088 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:25.346707 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.346698 5088 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:25.346707 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.346706 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcvs2\" (UniqueName: \"kubernetes.io/projected/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-kube-api-access-rcvs2\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:25.346707 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.346715 5088 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:25.501547 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.501508 5088 generic.go:356] "Generic (PLEG): container finished" podID="1f2b960b-8db8-4b84-b6d1-9ac12823e278" containerID="ac6ef7d6f2e1e074bfa2ccd91e5df5aacd6d4d99966f02c65d29e197fcc55de3" exitCode=0 Oct 30 20:09:25.501759 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.501555 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" Oct 30 20:09:25.501759 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.501589 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" event={"ID":"1f2b960b-8db8-4b84-b6d1-9ac12823e278","Type":"ContainerDied","Data":"ac6ef7d6f2e1e074bfa2ccd91e5df5aacd6d4d99966f02c65d29e197fcc55de3"} Oct 30 20:09:25.501759 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.501699 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g" event={"ID":"1f2b960b-8db8-4b84-b6d1-9ac12823e278","Type":"ContainerDied","Data":"3301c524dcd532234860794bb45de4e8819f1bec03fee43f69be6ef58fd55675"} Oct 30 20:09:25.501759 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.501729 5088 scope.go:122] "RemoveContainer" containerID="ac6ef7d6f2e1e074bfa2ccd91e5df5aacd6d4d99966f02c65d29e197fcc55de3" Oct 30 20:09:25.503570 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.503544 5088 generic.go:356] "Generic (PLEG): container finished" podID="c0748781-63b9-4af1-8eab-5d0cef276da6" containerID="cf771118fedc5389d16f43ca5d7c4e0f0b24d2193c33d1c77df884e756cd9c0f" exitCode=0 Oct 30 20:09:25.503688 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.503606 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" event={"ID":"c0748781-63b9-4af1-8eab-5d0cef276da6","Type":"ContainerDied","Data":"cf771118fedc5389d16f43ca5d7c4e0f0b24d2193c33d1c77df884e756cd9c0f"} Oct 30 20:09:25.503688 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.503638 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" Oct 30 20:09:25.503688 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.503678 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf" event={"ID":"c0748781-63b9-4af1-8eab-5d0cef276da6","Type":"ContainerDied","Data":"dc36d72d10480d8b242c759d9b9086439678cd8802b0a9740ddf1ef74a40618f"} Oct 30 20:09:25.505351 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.505331 5088 generic.go:356] "Generic (PLEG): container finished" podID="e402d580-e9a6-4e35-a2ff-e5ef2d78eb90" containerID="c681ce095c069d984ee329fe861fc5299eb161f9aeb2b45159bb8880bd808928" exitCode=0 Oct 30 20:09:25.505455 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.505385 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" Oct 30 20:09:25.505455 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.505404 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" event={"ID":"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90","Type":"ContainerDied","Data":"c681ce095c069d984ee329fe861fc5299eb161f9aeb2b45159bb8880bd808928"} Oct 30 20:09:25.505455 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.505438 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h" event={"ID":"e402d580-e9a6-4e35-a2ff-e5ef2d78eb90","Type":"ContainerDied","Data":"694bc27926151698b194dd89499b5182f2cfe6ef4844635dca205016eea82ecb"} Oct 30 20:09:25.515717 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.515694 5088 scope.go:122] "RemoveContainer" containerID="ac6ef7d6f2e1e074bfa2ccd91e5df5aacd6d4d99966f02c65d29e197fcc55de3" Oct 30 20:09:25.516033 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:09:25.516009 5088 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac6ef7d6f2e1e074bfa2ccd91e5df5aacd6d4d99966f02c65d29e197fcc55de3\": container with ID starting with ac6ef7d6f2e1e074bfa2ccd91e5df5aacd6d4d99966f02c65d29e197fcc55de3 not found: ID does not exist" containerID="ac6ef7d6f2e1e074bfa2ccd91e5df5aacd6d4d99966f02c65d29e197fcc55de3" Oct 30 20:09:25.516129 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.516043 5088 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac6ef7d6f2e1e074bfa2ccd91e5df5aacd6d4d99966f02c65d29e197fcc55de3"} err="failed to get container status \"ac6ef7d6f2e1e074bfa2ccd91e5df5aacd6d4d99966f02c65d29e197fcc55de3\": rpc error: code = NotFound desc = could not find container \"ac6ef7d6f2e1e074bfa2ccd91e5df5aacd6d4d99966f02c65d29e197fcc55de3\": container with ID starting with ac6ef7d6f2e1e074bfa2ccd91e5df5aacd6d4d99966f02c65d29e197fcc55de3 not found: ID does not exist" Oct 30 20:09:25.516129 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.516064 5088 scope.go:122] "RemoveContainer" containerID="cf771118fedc5389d16f43ca5d7c4e0f0b24d2193c33d1c77df884e756cd9c0f" Oct 30 20:09:25.527584 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.527559 5088 scope.go:122] "RemoveContainer" containerID="a9bd049c96e0f99442c1823a643467f15e55bf80eb26fd607ffad0a2abee78bf" Oct 30 20:09:25.537826 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.537802 5088 scope.go:122] "RemoveContainer" containerID="cf771118fedc5389d16f43ca5d7c4e0f0b24d2193c33d1c77df884e756cd9c0f" Oct 30 20:09:25.538153 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:09:25.538136 5088 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf771118fedc5389d16f43ca5d7c4e0f0b24d2193c33d1c77df884e756cd9c0f\": container with ID starting with cf771118fedc5389d16f43ca5d7c4e0f0b24d2193c33d1c77df884e756cd9c0f not found: ID does not exist" containerID="cf771118fedc5389d16f43ca5d7c4e0f0b24d2193c33d1c77df884e756cd9c0f" Oct 30 20:09:25.538196 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.538161 5088 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf771118fedc5389d16f43ca5d7c4e0f0b24d2193c33d1c77df884e756cd9c0f"} err="failed to get container status \"cf771118fedc5389d16f43ca5d7c4e0f0b24d2193c33d1c77df884e756cd9c0f\": rpc error: code = NotFound desc = could not find container \"cf771118fedc5389d16f43ca5d7c4e0f0b24d2193c33d1c77df884e756cd9c0f\": container with ID starting with cf771118fedc5389d16f43ca5d7c4e0f0b24d2193c33d1c77df884e756cd9c0f not found: ID does not exist" Oct 30 20:09:25.538196 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.538180 5088 scope.go:122] "RemoveContainer" containerID="a9bd049c96e0f99442c1823a643467f15e55bf80eb26fd607ffad0a2abee78bf" Oct 30 20:09:25.538417 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:09:25.538397 5088 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9bd049c96e0f99442c1823a643467f15e55bf80eb26fd607ffad0a2abee78bf\": container with ID starting with a9bd049c96e0f99442c1823a643467f15e55bf80eb26fd607ffad0a2abee78bf not found: ID does not exist" containerID="a9bd049c96e0f99442c1823a643467f15e55bf80eb26fd607ffad0a2abee78bf" Oct 30 20:09:25.538457 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.538422 5088 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9bd049c96e0f99442c1823a643467f15e55bf80eb26fd607ffad0a2abee78bf"} err="failed to get container status \"a9bd049c96e0f99442c1823a643467f15e55bf80eb26fd607ffad0a2abee78bf\": rpc error: code = NotFound desc = could not find container \"a9bd049c96e0f99442c1823a643467f15e55bf80eb26fd607ffad0a2abee78bf\": container with ID starting with a9bd049c96e0f99442c1823a643467f15e55bf80eb26fd607ffad0a2abee78bf not found: ID does not exist" Oct 30 20:09:25.538457 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.538436 5088 scope.go:122] "RemoveContainer" containerID="c681ce095c069d984ee329fe861fc5299eb161f9aeb2b45159bb8880bd808928" Oct 30 20:09:25.548775 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.548749 5088 scope.go:122] "RemoveContainer" containerID="c681ce095c069d984ee329fe861fc5299eb161f9aeb2b45159bb8880bd808928" Oct 30 20:09:25.549158 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:09:25.549133 5088 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c681ce095c069d984ee329fe861fc5299eb161f9aeb2b45159bb8880bd808928\": container with ID starting with c681ce095c069d984ee329fe861fc5299eb161f9aeb2b45159bb8880bd808928 not found: ID does not exist" containerID="c681ce095c069d984ee329fe861fc5299eb161f9aeb2b45159bb8880bd808928" Oct 30 20:09:25.549244 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.549167 5088 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c681ce095c069d984ee329fe861fc5299eb161f9aeb2b45159bb8880bd808928"} err="failed to get container status \"c681ce095c069d984ee329fe861fc5299eb161f9aeb2b45159bb8880bd808928\": rpc error: code = NotFound desc = could not find container \"c681ce095c069d984ee329fe861fc5299eb161f9aeb2b45159bb8880bd808928\": container with ID starting with c681ce095c069d984ee329fe861fc5299eb161f9aeb2b45159bb8880bd808928 not found: ID does not exist" Oct 30 20:09:25.570810 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.570767 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf"] Oct 30 20:09:25.584238 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.584202 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-zlxtf"] Oct 30 20:09:25.629332 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.629242 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h"] Oct 30 20:09:25.632796 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.632773 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-5vr2h"] Oct 30 20:09:25.684991 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.684957 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g"] Oct 30 20:09:25.693380 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:25.693348 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-51dd62-openshift-default-7954bd8797-h248g"] Oct 30 20:09:26.452507 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:26.452468 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-efbac3-66db987bc9-ptftf" podUID="0ee0e296-2cf3-4cdc-9340-d202e75c6a14" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:09:26.739260 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:26.739218 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f2b960b-8db8-4b84-b6d1-9ac12823e278" path="/var/lib/kubelet/pods/1f2b960b-8db8-4b84-b6d1-9ac12823e278/volumes" Oct 30 20:09:26.739834 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:26.739811 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0748781-63b9-4af1-8eab-5d0cef276da6" path="/var/lib/kubelet/pods/c0748781-63b9-4af1-8eab-5d0cef276da6/volumes" Oct 30 20:09:26.740332 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:26.740313 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e402d580-e9a6-4e35-a2ff-e5ef2d78eb90" path="/var/lib/kubelet/pods/e402d580-e9a6-4e35-a2ff-e5ef2d78eb90/volumes" Oct 30 20:09:26.893983 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:26.893946 5088 scope.go:122] "RemoveContainer" containerID="7973cb77382d83ca511eaaaa0c9f731f93cb413a97a9ccacff841279a2228925" Oct 30 20:09:26.903941 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:26.903912 5088 scope.go:122] "RemoveContainer" containerID="41517793b1607f48e8419b8ef0df6e2114c329899772b7f8cbdcd7c000efb1e9" Oct 30 20:09:26.913406 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:26.913384 5088 scope.go:122] "RemoveContainer" containerID="2397d8f8f2eb7160a3fa9c243030f3c589b299cfd4ac606570a17bda5269158c" Oct 30 20:09:26.922518 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:26.922490 5088 scope.go:122] "RemoveContainer" containerID="5bebaecf163264904fdb275b0369c6f6a91e01b5cc9e2c91094ea9efd8d92def" Oct 30 20:09:26.931376 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:26.931350 5088 scope.go:122] "RemoveContainer" containerID="c7abd59e44ced870d8ebb875bfa37f908ddd0b5eb248a581e9bf77ac41ef1c8e" Oct 30 20:09:26.940266 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:26.940241 5088 scope.go:122] "RemoveContainer" containerID="b05e9b0d77de96b9f1159bdf32750a69a00a0f90e63cece81107203cb2040bc2" Oct 30 20:09:26.949571 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:26.949545 5088 scope.go:122] "RemoveContainer" containerID="f6aabab1231d6244f5ca2ce70e510ac29ac7e567fcf76a684f741161dc0d2342" Oct 30 20:09:26.959425 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:26.959401 5088 scope.go:122] "RemoveContainer" containerID="8782d39e2295528eba8400edb6578f331c4c348500de3da94aaf50029c2ab628" Oct 30 20:09:26.971593 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:26.970792 5088 scope.go:122] "RemoveContainer" containerID="29568f1c1eda0617249dabb49e01503c963884e62d977c2e715649bfbadacb7b" Oct 30 20:09:27.296002 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:27.295967 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-efbac3-66db987bc9-ptftf" Oct 30 20:09:27.364131 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:27.364098 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ee0e296-2cf3-4cdc-9340-d202e75c6a14-kube-api-access-t268w\" (UniqueName: \"kubernetes.io/projected/0ee0e296-2cf3-4cdc-9340-d202e75c6a14-kube-api-access-t268w\") pod \"0ee0e296-2cf3-4cdc-9340-d202e75c6a14\" (UID: \"0ee0e296-2cf3-4cdc-9340-d202e75c6a14\") " Oct 30 20:09:27.364309 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:27.364249 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ee0e296-2cf3-4cdc-9340-d202e75c6a14-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ee0e296-2cf3-4cdc-9340-d202e75c6a14-plugins-config-volume\") pod \"0ee0e296-2cf3-4cdc-9340-d202e75c6a14\" (UID: \"0ee0e296-2cf3-4cdc-9340-d202e75c6a14\") " Oct 30 20:09:27.364708 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:27.364680 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ee0e296-2cf3-4cdc-9340-d202e75c6a14-plugins-config-volume" pod "0ee0e296-2cf3-4cdc-9340-d202e75c6a14" (UID: "0ee0e296-2cf3-4cdc-9340-d202e75c6a14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:09:27.366283 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:27.366252 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ee0e296-2cf3-4cdc-9340-d202e75c6a14-kube-api-access-t268w" pod "0ee0e296-2cf3-4cdc-9340-d202e75c6a14" (UID: "0ee0e296-2cf3-4cdc-9340-d202e75c6a14"). InnerVolumeSpecName "kube-api-access-t268w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:27.465667 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:27.465603 5088 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ee0e296-2cf3-4cdc-9340-d202e75c6a14-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:27.465667 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:27.465661 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t268w\" (UniqueName: \"kubernetes.io/projected/0ee0e296-2cf3-4cdc-9340-d202e75c6a14-kube-api-access-t268w\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:27.523902 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:27.523812 5088 generic.go:356] "Generic (PLEG): container finished" podID="0ee0e296-2cf3-4cdc-9340-d202e75c6a14" containerID="253e2f1b3689f4cf53015de4c05a89f5d579f55bad8438f33b5ab9f23e73e376" exitCode=0 Oct 30 20:09:27.523902 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:27.523853 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-efbac3-66db987bc9-ptftf" Oct 30 20:09:27.523902 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:27.523887 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-efbac3-66db987bc9-ptftf" event={"ID":"0ee0e296-2cf3-4cdc-9340-d202e75c6a14","Type":"ContainerDied","Data":"253e2f1b3689f4cf53015de4c05a89f5d579f55bad8438f33b5ab9f23e73e376"} Oct 30 20:09:27.524160 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:27.523939 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-efbac3-66db987bc9-ptftf" event={"ID":"0ee0e296-2cf3-4cdc-9340-d202e75c6a14","Type":"ContainerDied","Data":"69f193bc35b3e7c0d5e2ef2d18210b21866787ab2d67b7c4ee3c1f5e8895e51c"} Oct 30 20:09:27.524160 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:27.523977 5088 scope.go:122] "RemoveContainer" containerID="253e2f1b3689f4cf53015de4c05a89f5d579f55bad8438f33b5ab9f23e73e376" Oct 30 20:09:27.534349 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:27.534326 5088 scope.go:122] "RemoveContainer" containerID="253e2f1b3689f4cf53015de4c05a89f5d579f55bad8438f33b5ab9f23e73e376" Oct 30 20:09:27.534635 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:09:27.534600 5088 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"253e2f1b3689f4cf53015de4c05a89f5d579f55bad8438f33b5ab9f23e73e376\": container with ID starting with 253e2f1b3689f4cf53015de4c05a89f5d579f55bad8438f33b5ab9f23e73e376 not found: ID does not exist" containerID="253e2f1b3689f4cf53015de4c05a89f5d579f55bad8438f33b5ab9f23e73e376" Oct 30 20:09:27.534751 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:27.534647 5088 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"253e2f1b3689f4cf53015de4c05a89f5d579f55bad8438f33b5ab9f23e73e376"} err="failed to get container status \"253e2f1b3689f4cf53015de4c05a89f5d579f55bad8438f33b5ab9f23e73e376\": rpc error: code = NotFound desc = could not find container \"253e2f1b3689f4cf53015de4c05a89f5d579f55bad8438f33b5ab9f23e73e376\": container with ID starting with 253e2f1b3689f4cf53015de4c05a89f5d579f55bad8438f33b5ab9f23e73e376 not found: ID does not exist" Oct 30 20:09:27.597034 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:27.596994 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-efbac3-66db987bc9-ptftf"] Oct 30 20:09:27.600271 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:27.600244 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-efbac3-66db987bc9-ptftf"] Oct 30 20:09:27.903274 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:27.903178 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98"] Oct 30 20:09:27.903559 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:27.903527 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" podUID="ce337cd9-bd40-432b-998d-4357f13b46d1" containerName="istio-proxy" containerID="cri-o://e6aa99641e4b078628025af63172e36704b9d007e3d6997c78f025449ecab0cb" gracePeriod=30 Oct 30 20:09:27.915406 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:27.915372 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2"] Oct 30 20:09:27.915751 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:27.915693 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" podUID="a51889ad-a7b6-41f2-9857-679ac04b7bea" containerName="istio-proxy" containerID="cri-o://d9162f37e426872971fc72411af9413abb14758a4fd7f0a4977b158942bc274f" gracePeriod=30 Oct 30 20:09:27.918969 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:27.918907 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5"] Oct 30 20:09:27.919114 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:27.919092 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" podUID="cf9a8162-01f0-4226-b8f4-783125c277a2" containerName="istio-proxy" containerID="cri-o://4c13990224bbb25058dad815e340b413c6171b14b68a186e923c3ecf17a7c4fa" gracePeriod=30 Oct 30 20:09:27.922879 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:27.922852 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf"] Oct 30 20:09:27.923106 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:27.923082 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" podUID="811db7c8-11d5-4e9d-8c8b-254c17906e84" containerName="istio-proxy" containerID="cri-o://daeffcde17c313caceb0a68272537ab468c4b529d8d60af9e473a9c6ffd1ffd3" gracePeriod=30 Oct 30 20:09:28.066930 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:28.066897 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de4e2008-r28vt"] Oct 30 20:09:28.081394 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:28.081363 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de4e2008-r28vt"] Oct 30 20:09:28.244660 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:28.244622 5088 patch_prober.go:29] interesting pod/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:09:28.244802 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:28.244692 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" podUID="a51889ad-a7b6-41f2-9857-679ac04b7bea" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:09:28.738215 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:28.738183 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ee0e296-2cf3-4cdc-9340-d202e75c6a14" path="/var/lib/kubelet/pods/0ee0e296-2cf3-4cdc-9340-d202e75c6a14/volumes" Oct 30 20:09:28.738608 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:28.738588 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8eacb59-ba64-450b-b1bb-3bc6309d0b75" path="/var/lib/kubelet/pods/f8eacb59-ba64-450b-b1bb-3bc6309d0b75/volumes" Oct 30 20:09:29.054355 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:29.054265 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae2ba856-r8kv6"] Oct 30 20:09:29.074749 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:29.074712 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae2ba856-r8kv6"] Oct 30 20:09:29.546939 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:29.546905 5088 patch_prober.go:29] interesting pod/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:09:29.547109 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:29.546958 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" podUID="ce337cd9-bd40-432b-998d-4357f13b46d1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:09:30.738655 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:30.738606 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5d7b01c-5886-461d-aff5-c6646d23b78a" path="/var/lib/kubelet/pods/d5d7b01c-5886-461d-aff5-c6646d23b78a/volumes" Oct 30 20:09:33.214121 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.214033 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" Oct 30 20:09:33.341636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.316280 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" Oct 30 20:09:33.350819 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.342183 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf9a8162-01f0-4226-b8f4-783125c277a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9a8162-01f0-4226-b8f4-783125c277a2-workload-socket\") pod \"cf9a8162-01f0-4226-b8f4-783125c277a2\" (UID: \"cf9a8162-01f0-4226-b8f4-783125c277a2\") " Oct 30 20:09:33.359243 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.350773 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf9a8162-01f0-4226-b8f4-783125c277a2-workload-socket" pod "cf9a8162-01f0-4226-b8f4-783125c277a2" (UID: "cf9a8162-01f0-4226-b8f4-783125c277a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:33.359243 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.358777 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf9a8162-01f0-4226-b8f4-783125c277a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf9a8162-01f0-4226-b8f4-783125c277a2-istio-data\") pod \"cf9a8162-01f0-4226-b8f4-783125c277a2\" (UID: \"cf9a8162-01f0-4226-b8f4-783125c277a2\") " Oct 30 20:09:33.359243 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.358829 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf9a8162-01f0-4226-b8f4-783125c277a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf9a8162-01f0-4226-b8f4-783125c277a2-istiod-ca-cert\") pod \"cf9a8162-01f0-4226-b8f4-783125c277a2\" (UID: \"cf9a8162-01f0-4226-b8f4-783125c277a2\") " Oct 30 20:09:33.359243 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.358866 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf9a8162-01f0-4226-b8f4-783125c277a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf9a8162-01f0-4226-b8f4-783125c277a2-istio-envoy\") pod \"cf9a8162-01f0-4226-b8f4-783125c277a2\" (UID: \"cf9a8162-01f0-4226-b8f4-783125c277a2\") " Oct 30 20:09:33.359243 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.358980 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf9a8162-01f0-4226-b8f4-783125c277a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9a8162-01f0-4226-b8f4-783125c277a2-credential-socket\") pod \"cf9a8162-01f0-4226-b8f4-783125c277a2\" (UID: \"cf9a8162-01f0-4226-b8f4-783125c277a2\") " Oct 30 20:09:33.359243 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.359017 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce337cd9-bd40-432b-998d-4357f13b46d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce337cd9-bd40-432b-998d-4357f13b46d1-istio-podinfo\") pod \"ce337cd9-bd40-432b-998d-4357f13b46d1\" (UID: \"ce337cd9-bd40-432b-998d-4357f13b46d1\") " Oct 30 20:09:33.359243 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.359053 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce337cd9-bd40-432b-998d-4357f13b46d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce337cd9-bd40-432b-998d-4357f13b46d1-istiod-ca-cert\") pod \"ce337cd9-bd40-432b-998d-4357f13b46d1\" (UID: \"ce337cd9-bd40-432b-998d-4357f13b46d1\") " Oct 30 20:09:33.359243 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.359080 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf9a8162-01f0-4226-b8f4-783125c277a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf9a8162-01f0-4226-b8f4-783125c277a2-istio-podinfo\") pod \"cf9a8162-01f0-4226-b8f4-783125c277a2\" (UID: \"cf9a8162-01f0-4226-b8f4-783125c277a2\") " Oct 30 20:09:33.359243 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.359114 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf9a8162-01f0-4226-b8f4-783125c277a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf9a8162-01f0-4226-b8f4-783125c277a2-workload-certs\") pod \"cf9a8162-01f0-4226-b8f4-783125c277a2\" (UID: \"cf9a8162-01f0-4226-b8f4-783125c277a2\") " Oct 30 20:09:33.359243 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.359156 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce337cd9-bd40-432b-998d-4357f13b46d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce337cd9-bd40-432b-998d-4357f13b46d1-workload-socket\") pod \"ce337cd9-bd40-432b-998d-4357f13b46d1\" (UID: \"ce337cd9-bd40-432b-998d-4357f13b46d1\") " Oct 30 20:09:33.359243 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.359194 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce337cd9-bd40-432b-998d-4357f13b46d1-istio-token\" (UniqueName: \"kubernetes.io/projected/ce337cd9-bd40-432b-998d-4357f13b46d1-istio-token\") pod \"ce337cd9-bd40-432b-998d-4357f13b46d1\" (UID: \"ce337cd9-bd40-432b-998d-4357f13b46d1\") " Oct 30 20:09:33.359243 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.359222 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf9a8162-01f0-4226-b8f4-783125c277a2-istio-token\" (UniqueName: \"kubernetes.io/projected/cf9a8162-01f0-4226-b8f4-783125c277a2-istio-token\") pod \"cf9a8162-01f0-4226-b8f4-783125c277a2\" (UID: \"cf9a8162-01f0-4226-b8f4-783125c277a2\") " Oct 30 20:09:33.359948 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.359314 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce337cd9-bd40-432b-998d-4357f13b46d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce337cd9-bd40-432b-998d-4357f13b46d1-workload-certs\") pod \"ce337cd9-bd40-432b-998d-4357f13b46d1\" (UID: \"ce337cd9-bd40-432b-998d-4357f13b46d1\") " Oct 30 20:09:33.359948 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.359348 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf9a8162-01f0-4226-b8f4-783125c277a2-kube-api-access-ggqxg\" (UniqueName: \"kubernetes.io/projected/cf9a8162-01f0-4226-b8f4-783125c277a2-kube-api-access-ggqxg\") pod \"cf9a8162-01f0-4226-b8f4-783125c277a2\" (UID: \"cf9a8162-01f0-4226-b8f4-783125c277a2\") " Oct 30 20:09:33.359948 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.359797 5088 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9a8162-01f0-4226-b8f4-783125c277a2-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:33.361253 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.360360 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf9a8162-01f0-4226-b8f4-783125c277a2-istio-data" pod "cf9a8162-01f0-4226-b8f4-783125c277a2" (UID: "cf9a8162-01f0-4226-b8f4-783125c277a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:33.361253 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.360699 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf9a8162-01f0-4226-b8f4-783125c277a2-istiod-ca-cert" pod "cf9a8162-01f0-4226-b8f4-783125c277a2" (UID: "cf9a8162-01f0-4226-b8f4-783125c277a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:09:33.361253 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.360952 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf9a8162-01f0-4226-b8f4-783125c277a2-credential-socket" pod "cf9a8162-01f0-4226-b8f4-783125c277a2" (UID: "cf9a8162-01f0-4226-b8f4-783125c277a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:33.361253 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.360967 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf9a8162-01f0-4226-b8f4-783125c277a2-workload-certs" pod "cf9a8162-01f0-4226-b8f4-783125c277a2" (UID: "cf9a8162-01f0-4226-b8f4-783125c277a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:33.361527 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.361468 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce337cd9-bd40-432b-998d-4357f13b46d1-istiod-ca-cert" pod "ce337cd9-bd40-432b-998d-4357f13b46d1" (UID: "ce337cd9-bd40-432b-998d-4357f13b46d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:09:33.362036 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.361783 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce337cd9-bd40-432b-998d-4357f13b46d1-workload-socket" pod "ce337cd9-bd40-432b-998d-4357f13b46d1" (UID: "ce337cd9-bd40-432b-998d-4357f13b46d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:33.362208 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.362173 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce337cd9-bd40-432b-998d-4357f13b46d1-workload-certs" pod "ce337cd9-bd40-432b-998d-4357f13b46d1" (UID: "ce337cd9-bd40-432b-998d-4357f13b46d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:33.364142 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.364087 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce337cd9-bd40-432b-998d-4357f13b46d1-istio-podinfo" pod "ce337cd9-bd40-432b-998d-4357f13b46d1" (UID: "ce337cd9-bd40-432b-998d-4357f13b46d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:09:33.365742 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.365709 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf9a8162-01f0-4226-b8f4-783125c277a2-kube-api-access-ggqxg" pod "cf9a8162-01f0-4226-b8f4-783125c277a2" (UID: "cf9a8162-01f0-4226-b8f4-783125c277a2"). InnerVolumeSpecName "kube-api-access-ggqxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:33.366027 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.365993 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf9a8162-01f0-4226-b8f4-783125c277a2-istio-token" pod "cf9a8162-01f0-4226-b8f4-783125c277a2" (UID: "cf9a8162-01f0-4226-b8f4-783125c277a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:33.367367 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.367269 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf9a8162-01f0-4226-b8f4-783125c277a2-istio-podinfo" pod "cf9a8162-01f0-4226-b8f4-783125c277a2" (UID: "cf9a8162-01f0-4226-b8f4-783125c277a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:09:33.367367 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.367335 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce337cd9-bd40-432b-998d-4357f13b46d1-istio-token" pod "ce337cd9-bd40-432b-998d-4357f13b46d1" (UID: "ce337cd9-bd40-432b-998d-4357f13b46d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:33.370042 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.369331 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf9a8162-01f0-4226-b8f4-783125c277a2-istio-envoy" pod "cf9a8162-01f0-4226-b8f4-783125c277a2" (UID: "cf9a8162-01f0-4226-b8f4-783125c277a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:33.380602 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.380574 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" Oct 30 20:09:33.402662 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.397894 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" Oct 30 20:09:33.460637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.460314 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/811db7c8-11d5-4e9d-8c8b-254c17906e84-kube-api-access-98t24\" (UniqueName: \"kubernetes.io/projected/811db7c8-11d5-4e9d-8c8b-254c17906e84-kube-api-access-98t24\") pod \"811db7c8-11d5-4e9d-8c8b-254c17906e84\" (UID: \"811db7c8-11d5-4e9d-8c8b-254c17906e84\") " Oct 30 20:09:33.460637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.460362 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce337cd9-bd40-432b-998d-4357f13b46d1-kube-api-access-dhzwd\" (UniqueName: \"kubernetes.io/projected/ce337cd9-bd40-432b-998d-4357f13b46d1-kube-api-access-dhzwd\") pod \"ce337cd9-bd40-432b-998d-4357f13b46d1\" (UID: \"ce337cd9-bd40-432b-998d-4357f13b46d1\") " Oct 30 20:09:33.460637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.460411 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a51889ad-a7b6-41f2-9857-679ac04b7bea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51889ad-a7b6-41f2-9857-679ac04b7bea-workload-socket\") pod \"a51889ad-a7b6-41f2-9857-679ac04b7bea\" (UID: \"a51889ad-a7b6-41f2-9857-679ac04b7bea\") " Oct 30 20:09:33.460637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.460438 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a51889ad-a7b6-41f2-9857-679ac04b7bea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a51889ad-a7b6-41f2-9857-679ac04b7bea-workload-certs\") pod \"a51889ad-a7b6-41f2-9857-679ac04b7bea\" (UID: \"a51889ad-a7b6-41f2-9857-679ac04b7bea\") " Oct 30 20:09:33.460637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.460455 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a51889ad-a7b6-41f2-9857-679ac04b7bea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a51889ad-a7b6-41f2-9857-679ac04b7bea-istio-data\") pod \"a51889ad-a7b6-41f2-9857-679ac04b7bea\" (UID: \"a51889ad-a7b6-41f2-9857-679ac04b7bea\") " Oct 30 20:09:33.460637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.460472 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/811db7c8-11d5-4e9d-8c8b-254c17906e84-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/811db7c8-11d5-4e9d-8c8b-254c17906e84-istiod-ca-cert\") pod \"811db7c8-11d5-4e9d-8c8b-254c17906e84\" (UID: \"811db7c8-11d5-4e9d-8c8b-254c17906e84\") " Oct 30 20:09:33.460637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.460492 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a51889ad-a7b6-41f2-9857-679ac04b7bea-istio-token\" (UniqueName: \"kubernetes.io/projected/a51889ad-a7b6-41f2-9857-679ac04b7bea-istio-token\") pod \"a51889ad-a7b6-41f2-9857-679ac04b7bea\" (UID: \"a51889ad-a7b6-41f2-9857-679ac04b7bea\") " Oct 30 20:09:33.460637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.460517 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce337cd9-bd40-432b-998d-4357f13b46d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce337cd9-bd40-432b-998d-4357f13b46d1-istio-envoy\") pod \"ce337cd9-bd40-432b-998d-4357f13b46d1\" (UID: \"ce337cd9-bd40-432b-998d-4357f13b46d1\") " Oct 30 20:09:33.460637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.460541 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a51889ad-a7b6-41f2-9857-679ac04b7bea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a51889ad-a7b6-41f2-9857-679ac04b7bea-istio-podinfo\") pod \"a51889ad-a7b6-41f2-9857-679ac04b7bea\" (UID: \"a51889ad-a7b6-41f2-9857-679ac04b7bea\") " Oct 30 20:09:33.460637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.460564 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/811db7c8-11d5-4e9d-8c8b-254c17906e84-istio-token\" (UniqueName: \"kubernetes.io/projected/811db7c8-11d5-4e9d-8c8b-254c17906e84-istio-token\") pod \"811db7c8-11d5-4e9d-8c8b-254c17906e84\" (UID: \"811db7c8-11d5-4e9d-8c8b-254c17906e84\") " Oct 30 20:09:33.460637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.460584 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce337cd9-bd40-432b-998d-4357f13b46d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce337cd9-bd40-432b-998d-4357f13b46d1-credential-socket\") pod \"ce337cd9-bd40-432b-998d-4357f13b46d1\" (UID: \"ce337cd9-bd40-432b-998d-4357f13b46d1\") " Oct 30 20:09:33.460637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.460607 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce337cd9-bd40-432b-998d-4357f13b46d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce337cd9-bd40-432b-998d-4357f13b46d1-istio-data\") pod \"ce337cd9-bd40-432b-998d-4357f13b46d1\" (UID: \"ce337cd9-bd40-432b-998d-4357f13b46d1\") " Oct 30 20:09:33.461453 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.460669 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a51889ad-a7b6-41f2-9857-679ac04b7bea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a51889ad-a7b6-41f2-9857-679ac04b7bea-istio-envoy\") pod \"a51889ad-a7b6-41f2-9857-679ac04b7bea\" (UID: \"a51889ad-a7b6-41f2-9857-679ac04b7bea\") " Oct 30 20:09:33.461453 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.460720 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/811db7c8-11d5-4e9d-8c8b-254c17906e84-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/811db7c8-11d5-4e9d-8c8b-254c17906e84-istio-data\") pod \"811db7c8-11d5-4e9d-8c8b-254c17906e84\" (UID: \"811db7c8-11d5-4e9d-8c8b-254c17906e84\") " Oct 30 20:09:33.461453 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.460745 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/811db7c8-11d5-4e9d-8c8b-254c17906e84-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/811db7c8-11d5-4e9d-8c8b-254c17906e84-workload-certs\") pod \"811db7c8-11d5-4e9d-8c8b-254c17906e84\" (UID: \"811db7c8-11d5-4e9d-8c8b-254c17906e84\") " Oct 30 20:09:33.461453 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.460812 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a51889ad-a7b6-41f2-9857-679ac04b7bea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a51889ad-a7b6-41f2-9857-679ac04b7bea-istiod-ca-cert\") pod \"a51889ad-a7b6-41f2-9857-679ac04b7bea\" (UID: \"a51889ad-a7b6-41f2-9857-679ac04b7bea\") " Oct 30 20:09:33.461453 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.460841 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/811db7c8-11d5-4e9d-8c8b-254c17906e84-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/811db7c8-11d5-4e9d-8c8b-254c17906e84-credential-socket\") pod \"811db7c8-11d5-4e9d-8c8b-254c17906e84\" (UID: \"811db7c8-11d5-4e9d-8c8b-254c17906e84\") " Oct 30 20:09:33.461453 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.460871 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/811db7c8-11d5-4e9d-8c8b-254c17906e84-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/811db7c8-11d5-4e9d-8c8b-254c17906e84-istio-podinfo\") pod \"811db7c8-11d5-4e9d-8c8b-254c17906e84\" (UID: \"811db7c8-11d5-4e9d-8c8b-254c17906e84\") " Oct 30 20:09:33.461453 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.460898 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/811db7c8-11d5-4e9d-8c8b-254c17906e84-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/811db7c8-11d5-4e9d-8c8b-254c17906e84-istio-envoy\") pod \"811db7c8-11d5-4e9d-8c8b-254c17906e84\" (UID: \"811db7c8-11d5-4e9d-8c8b-254c17906e84\") " Oct 30 20:09:33.461453 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.460942 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a51889ad-a7b6-41f2-9857-679ac04b7bea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51889ad-a7b6-41f2-9857-679ac04b7bea-credential-socket\") pod \"a51889ad-a7b6-41f2-9857-679ac04b7bea\" (UID: \"a51889ad-a7b6-41f2-9857-679ac04b7bea\") " Oct 30 20:09:33.461453 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.460953 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce337cd9-bd40-432b-998d-4357f13b46d1-credential-socket" pod "ce337cd9-bd40-432b-998d-4357f13b46d1" (UID: "ce337cd9-bd40-432b-998d-4357f13b46d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:33.461453 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.460985 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/811db7c8-11d5-4e9d-8c8b-254c17906e84-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/811db7c8-11d5-4e9d-8c8b-254c17906e84-workload-socket\") pod \"811db7c8-11d5-4e9d-8c8b-254c17906e84\" (UID: \"811db7c8-11d5-4e9d-8c8b-254c17906e84\") " Oct 30 20:09:33.461453 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.461034 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a51889ad-a7b6-41f2-9857-679ac04b7bea-kube-api-access-rm29m\" (UniqueName: \"kubernetes.io/projected/a51889ad-a7b6-41f2-9857-679ac04b7bea-kube-api-access-rm29m\") pod \"a51889ad-a7b6-41f2-9857-679ac04b7bea\" (UID: \"a51889ad-a7b6-41f2-9857-679ac04b7bea\") " Oct 30 20:09:33.461453 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.461382 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a51889ad-a7b6-41f2-9857-679ac04b7bea-workload-socket" pod "a51889ad-a7b6-41f2-9857-679ac04b7bea" (UID: "a51889ad-a7b6-41f2-9857-679ac04b7bea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:33.461453 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.461453 5088 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce337cd9-bd40-432b-998d-4357f13b46d1-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:33.462108 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.461478 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggqxg\" (UniqueName: \"kubernetes.io/projected/cf9a8162-01f0-4226-b8f4-783125c277a2-kube-api-access-ggqxg\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:33.462108 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.461494 5088 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf9a8162-01f0-4226-b8f4-783125c277a2-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:33.462108 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.461507 5088 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce337cd9-bd40-432b-998d-4357f13b46d1-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:33.462108 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.461519 5088 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf9a8162-01f0-4226-b8f4-783125c277a2-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:33.462108 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.461532 5088 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf9a8162-01f0-4226-b8f4-783125c277a2-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:33.462108 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.461545 5088 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9a8162-01f0-4226-b8f4-783125c277a2-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:33.462108 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.461557 5088 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce337cd9-bd40-432b-998d-4357f13b46d1-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:33.462108 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.461570 5088 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce337cd9-bd40-432b-998d-4357f13b46d1-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:33.462108 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.461583 5088 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf9a8162-01f0-4226-b8f4-783125c277a2-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:33.462108 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.461595 5088 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf9a8162-01f0-4226-b8f4-783125c277a2-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:33.462108 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.461626 5088 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce337cd9-bd40-432b-998d-4357f13b46d1-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:33.462108 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.461639 5088 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce337cd9-bd40-432b-998d-4357f13b46d1-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:33.462108 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.461653 5088 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf9a8162-01f0-4226-b8f4-783125c277a2-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:33.462108 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.461992 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce337cd9-bd40-432b-998d-4357f13b46d1-istio-data" pod "ce337cd9-bd40-432b-998d-4357f13b46d1" (UID: "ce337cd9-bd40-432b-998d-4357f13b46d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:33.462797 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.462780 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/811db7c8-11d5-4e9d-8c8b-254c17906e84-istiod-ca-cert" pod "811db7c8-11d5-4e9d-8c8b-254c17906e84" (UID: "811db7c8-11d5-4e9d-8c8b-254c17906e84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:09:33.463224 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.463195 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a51889ad-a7b6-41f2-9857-679ac04b7bea-istio-data" pod "a51889ad-a7b6-41f2-9857-679ac04b7bea" (UID: "a51889ad-a7b6-41f2-9857-679ac04b7bea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:33.463339 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.463253 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a51889ad-a7b6-41f2-9857-679ac04b7bea-workload-certs" pod "a51889ad-a7b6-41f2-9857-679ac04b7bea" (UID: "a51889ad-a7b6-41f2-9857-679ac04b7bea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:33.464032 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.463786 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a51889ad-a7b6-41f2-9857-679ac04b7bea-credential-socket" pod "a51889ad-a7b6-41f2-9857-679ac04b7bea" (UID: "a51889ad-a7b6-41f2-9857-679ac04b7bea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:33.465444 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.465358 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/811db7c8-11d5-4e9d-8c8b-254c17906e84-workload-socket" pod "811db7c8-11d5-4e9d-8c8b-254c17906e84" (UID: "811db7c8-11d5-4e9d-8c8b-254c17906e84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:33.465444 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.465425 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/811db7c8-11d5-4e9d-8c8b-254c17906e84-workload-certs" pod "811db7c8-11d5-4e9d-8c8b-254c17906e84" (UID: "811db7c8-11d5-4e9d-8c8b-254c17906e84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:33.465679 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.465657 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/811db7c8-11d5-4e9d-8c8b-254c17906e84-istio-data" pod "811db7c8-11d5-4e9d-8c8b-254c17906e84" (UID: "811db7c8-11d5-4e9d-8c8b-254c17906e84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:33.465972 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.465950 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a51889ad-a7b6-41f2-9857-679ac04b7bea-istiod-ca-cert" pod "a51889ad-a7b6-41f2-9857-679ac04b7bea" (UID: "a51889ad-a7b6-41f2-9857-679ac04b7bea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:09:33.466163 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.466125 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/811db7c8-11d5-4e9d-8c8b-254c17906e84-credential-socket" pod "811db7c8-11d5-4e9d-8c8b-254c17906e84" (UID: "811db7c8-11d5-4e9d-8c8b-254c17906e84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:33.467654 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.467110 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/811db7c8-11d5-4e9d-8c8b-254c17906e84-istio-envoy" pod "811db7c8-11d5-4e9d-8c8b-254c17906e84" (UID: "811db7c8-11d5-4e9d-8c8b-254c17906e84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:33.467862 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.467827 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce337cd9-bd40-432b-998d-4357f13b46d1-kube-api-access-dhzwd" pod "ce337cd9-bd40-432b-998d-4357f13b46d1" (UID: "ce337cd9-bd40-432b-998d-4357f13b46d1"). InnerVolumeSpecName "kube-api-access-dhzwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:33.472438 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.472410 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/811db7c8-11d5-4e9d-8c8b-254c17906e84-kube-api-access-98t24" pod "811db7c8-11d5-4e9d-8c8b-254c17906e84" (UID: "811db7c8-11d5-4e9d-8c8b-254c17906e84"). InnerVolumeSpecName "kube-api-access-98t24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:33.472563 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.472523 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a51889ad-a7b6-41f2-9857-679ac04b7bea-istio-token" pod "a51889ad-a7b6-41f2-9857-679ac04b7bea" (UID: "a51889ad-a7b6-41f2-9857-679ac04b7bea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:33.472657 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.472561 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a51889ad-a7b6-41f2-9857-679ac04b7bea-istio-podinfo" pod "a51889ad-a7b6-41f2-9857-679ac04b7bea" (UID: "a51889ad-a7b6-41f2-9857-679ac04b7bea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:09:33.472720 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.472698 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/811db7c8-11d5-4e9d-8c8b-254c17906e84-istio-token" pod "811db7c8-11d5-4e9d-8c8b-254c17906e84" (UID: "811db7c8-11d5-4e9d-8c8b-254c17906e84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:33.472940 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.472911 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce337cd9-bd40-432b-998d-4357f13b46d1-istio-envoy" pod "ce337cd9-bd40-432b-998d-4357f13b46d1" (UID: "ce337cd9-bd40-432b-998d-4357f13b46d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:33.473089 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.473069 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a51889ad-a7b6-41f2-9857-679ac04b7bea-istio-envoy" pod "a51889ad-a7b6-41f2-9857-679ac04b7bea" (UID: "a51889ad-a7b6-41f2-9857-679ac04b7bea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:33.473331 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.473307 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/811db7c8-11d5-4e9d-8c8b-254c17906e84-istio-podinfo" pod "811db7c8-11d5-4e9d-8c8b-254c17906e84" (UID: "811db7c8-11d5-4e9d-8c8b-254c17906e84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:09:33.474386 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.474367 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a51889ad-a7b6-41f2-9857-679ac04b7bea-kube-api-access-rm29m" pod "a51889ad-a7b6-41f2-9857-679ac04b7bea" (UID: "a51889ad-a7b6-41f2-9857-679ac04b7bea"). InnerVolumeSpecName "kube-api-access-rm29m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:33.562663 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.562631 5088 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a51889ad-a7b6-41f2-9857-679ac04b7bea-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:33.562663 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.562658 5088 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/811db7c8-11d5-4e9d-8c8b-254c17906e84-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:33.562663 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.562667 5088 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/811db7c8-11d5-4e9d-8c8b-254c17906e84-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:33.562923 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.562677 5088 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a51889ad-a7b6-41f2-9857-679ac04b7bea-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:33.562923 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.562687 5088 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/811db7c8-11d5-4e9d-8c8b-254c17906e84-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:33.562923 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.562701 5088 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/811db7c8-11d5-4e9d-8c8b-254c17906e84-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:33.562923 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.562713 5088 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/811db7c8-11d5-4e9d-8c8b-254c17906e84-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:33.562923 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.562725 5088 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51889ad-a7b6-41f2-9857-679ac04b7bea-credential-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:33.562923 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.562737 5088 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/811db7c8-11d5-4e9d-8c8b-254c17906e84-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:33.562923 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.562750 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm29m\" (UniqueName: \"kubernetes.io/projected/a51889ad-a7b6-41f2-9857-679ac04b7bea-kube-api-access-rm29m\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:33.562923 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.562765 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98t24\" (UniqueName: \"kubernetes.io/projected/811db7c8-11d5-4e9d-8c8b-254c17906e84-kube-api-access-98t24\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:33.562923 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.562778 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhzwd\" (UniqueName: \"kubernetes.io/projected/ce337cd9-bd40-432b-998d-4357f13b46d1-kube-api-access-dhzwd\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:33.562923 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.562791 5088 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51889ad-a7b6-41f2-9857-679ac04b7bea-workload-socket\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:33.562923 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.562803 5088 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a51889ad-a7b6-41f2-9857-679ac04b7bea-workload-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:33.562923 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.562813 5088 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a51889ad-a7b6-41f2-9857-679ac04b7bea-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:33.562923 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.562820 5088 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/811db7c8-11d5-4e9d-8c8b-254c17906e84-istiod-ca-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:33.562923 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.562828 5088 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a51889ad-a7b6-41f2-9857-679ac04b7bea-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:33.562923 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.562838 5088 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce337cd9-bd40-432b-998d-4357f13b46d1-istio-envoy\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:33.562923 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.562846 5088 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a51889ad-a7b6-41f2-9857-679ac04b7bea-istio-podinfo\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:33.562923 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.562854 5088 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/811db7c8-11d5-4e9d-8c8b-254c17906e84-istio-token\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:33.562923 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.562862 5088 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce337cd9-bd40-432b-998d-4357f13b46d1-istio-data\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:09:33.570195 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.570145 5088 generic.go:356] "Generic (PLEG): container finished" podID="ce337cd9-bd40-432b-998d-4357f13b46d1" containerID="e6aa99641e4b078628025af63172e36704b9d007e3d6997c78f025449ecab0cb" exitCode=0 Oct 30 20:09:33.570195 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.570192 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" Oct 30 20:09:33.570427 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.570319 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" event={"ID":"ce337cd9-bd40-432b-998d-4357f13b46d1","Type":"ContainerDied","Data":"e6aa99641e4b078628025af63172e36704b9d007e3d6997c78f025449ecab0cb"} Oct 30 20:09:33.570427 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.570369 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98" event={"ID":"ce337cd9-bd40-432b-998d-4357f13b46d1","Type":"ContainerDied","Data":"0ad5cb9ad0b2e2b692c429adfe2239453e5d28cc2d65b300f0a0d411a9e4738d"} Oct 30 20:09:33.570427 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.570401 5088 scope.go:122] "RemoveContainer" containerID="e6aa99641e4b078628025af63172e36704b9d007e3d6997c78f025449ecab0cb" Oct 30 20:09:33.572745 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.572507 5088 generic.go:356] "Generic (PLEG): container finished" podID="a51889ad-a7b6-41f2-9857-679ac04b7bea" containerID="d9162f37e426872971fc72411af9413abb14758a4fd7f0a4977b158942bc274f" exitCode=0 Oct 30 20:09:33.572745 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.572563 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" Oct 30 20:09:33.572745 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.572722 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" event={"ID":"a51889ad-a7b6-41f2-9857-679ac04b7bea","Type":"ContainerDied","Data":"d9162f37e426872971fc72411af9413abb14758a4fd7f0a4977b158942bc274f"} Oct 30 20:09:33.572982 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.572792 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2" event={"ID":"a51889ad-a7b6-41f2-9857-679ac04b7bea","Type":"ContainerDied","Data":"87ea412953d2a90266ea9dccc100fff17edf29ac885beca0537e6f9347b70ec5"} Oct 30 20:09:33.577968 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.575006 5088 generic.go:356] "Generic (PLEG): container finished" podID="811db7c8-11d5-4e9d-8c8b-254c17906e84" containerID="daeffcde17c313caceb0a68272537ab468c4b529d8d60af9e473a9c6ffd1ffd3" exitCode=0 Oct 30 20:09:33.577968 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.575100 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" event={"ID":"811db7c8-11d5-4e9d-8c8b-254c17906e84","Type":"ContainerDied","Data":"daeffcde17c313caceb0a68272537ab468c4b529d8d60af9e473a9c6ffd1ffd3"} Oct 30 20:09:33.577968 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.575139 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" event={"ID":"811db7c8-11d5-4e9d-8c8b-254c17906e84","Type":"ContainerDied","Data":"e6990407150c0016c13b03b54d0cfaeadc6ae4d38c28904bbe60b2589eb611e9"} Oct 30 20:09:33.577968 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.575206 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf" Oct 30 20:09:33.578979 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.578767 5088 generic.go:356] "Generic (PLEG): container finished" podID="cf9a8162-01f0-4226-b8f4-783125c277a2" containerID="4c13990224bbb25058dad815e340b413c6171b14b68a186e923c3ecf17a7c4fa" exitCode=0 Oct 30 20:09:33.578979 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.578813 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" Oct 30 20:09:33.578979 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.578897 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" event={"ID":"cf9a8162-01f0-4226-b8f4-783125c277a2","Type":"ContainerDied","Data":"4c13990224bbb25058dad815e340b413c6171b14b68a186e923c3ecf17a7c4fa"} Oct 30 20:09:33.578979 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.578941 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5" event={"ID":"cf9a8162-01f0-4226-b8f4-783125c277a2","Type":"ContainerDied","Data":"38c6e49fb5aea702a7e3e884decb08e20b853a19e9cbb2319942d6a927ef2598"} Oct 30 20:09:33.639123 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.608552 5088 scope.go:122] "RemoveContainer" containerID="061917a119072909f856fe9e1d06cc2fc6221402faadd09ccfbf02ce0fff4c45" Oct 30 20:09:33.651108 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.650813 5088 scope.go:122] "RemoveContainer" containerID="e6aa99641e4b078628025af63172e36704b9d007e3d6997c78f025449ecab0cb" Oct 30 20:09:33.655857 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:09:33.655536 5088 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6aa99641e4b078628025af63172e36704b9d007e3d6997c78f025449ecab0cb\": container with ID starting with e6aa99641e4b078628025af63172e36704b9d007e3d6997c78f025449ecab0cb not found: ID does not exist" containerID="e6aa99641e4b078628025af63172e36704b9d007e3d6997c78f025449ecab0cb" Oct 30 20:09:33.655857 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.655575 5088 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6aa99641e4b078628025af63172e36704b9d007e3d6997c78f025449ecab0cb"} err="failed to get container status \"e6aa99641e4b078628025af63172e36704b9d007e3d6997c78f025449ecab0cb\": rpc error: code = NotFound desc = could not find container \"e6aa99641e4b078628025af63172e36704b9d007e3d6997c78f025449ecab0cb\": container with ID starting with e6aa99641e4b078628025af63172e36704b9d007e3d6997c78f025449ecab0cb not found: ID does not exist" Oct 30 20:09:33.655857 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.655599 5088 scope.go:122] "RemoveContainer" containerID="061917a119072909f856fe9e1d06cc2fc6221402faadd09ccfbf02ce0fff4c45" Oct 30 20:09:33.656726 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:09:33.656498 5088 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"061917a119072909f856fe9e1d06cc2fc6221402faadd09ccfbf02ce0fff4c45\": container with ID starting with 061917a119072909f856fe9e1d06cc2fc6221402faadd09ccfbf02ce0fff4c45 not found: ID does not exist" containerID="061917a119072909f856fe9e1d06cc2fc6221402faadd09ccfbf02ce0fff4c45" Oct 30 20:09:33.656726 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.656535 5088 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"061917a119072909f856fe9e1d06cc2fc6221402faadd09ccfbf02ce0fff4c45"} err="failed to get container status \"061917a119072909f856fe9e1d06cc2fc6221402faadd09ccfbf02ce0fff4c45\": rpc error: code = NotFound desc = could not find container \"061917a119072909f856fe9e1d06cc2fc6221402faadd09ccfbf02ce0fff4c45\": container with ID starting with 061917a119072909f856fe9e1d06cc2fc6221402faadd09ccfbf02ce0fff4c45 not found: ID does not exist" Oct 30 20:09:33.656726 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.656557 5088 scope.go:122] "RemoveContainer" containerID="d9162f37e426872971fc72411af9413abb14758a4fd7f0a4977b158942bc274f" Oct 30 20:09:33.666964 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.666943 5088 scope.go:122] "RemoveContainer" containerID="d9162f37e426872971fc72411af9413abb14758a4fd7f0a4977b158942bc274f" Oct 30 20:09:33.667294 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:09:33.667270 5088 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9162f37e426872971fc72411af9413abb14758a4fd7f0a4977b158942bc274f\": container with ID starting with d9162f37e426872971fc72411af9413abb14758a4fd7f0a4977b158942bc274f not found: ID does not exist" containerID="d9162f37e426872971fc72411af9413abb14758a4fd7f0a4977b158942bc274f" Oct 30 20:09:33.667381 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.667305 5088 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9162f37e426872971fc72411af9413abb14758a4fd7f0a4977b158942bc274f"} err="failed to get container status \"d9162f37e426872971fc72411af9413abb14758a4fd7f0a4977b158942bc274f\": rpc error: code = NotFound desc = could not find container \"d9162f37e426872971fc72411af9413abb14758a4fd7f0a4977b158942bc274f\": container with ID starting with d9162f37e426872971fc72411af9413abb14758a4fd7f0a4977b158942bc274f not found: ID does not exist" Oct 30 20:09:33.667381 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.667330 5088 scope.go:122] "RemoveContainer" containerID="daeffcde17c313caceb0a68272537ab468c4b529d8d60af9e473a9c6ffd1ffd3" Oct 30 20:09:33.676658 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.676608 5088 scope.go:122] "RemoveContainer" containerID="daeffcde17c313caceb0a68272537ab468c4b529d8d60af9e473a9c6ffd1ffd3" Oct 30 20:09:33.677001 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:09:33.676983 5088 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daeffcde17c313caceb0a68272537ab468c4b529d8d60af9e473a9c6ffd1ffd3\": container with ID starting with daeffcde17c313caceb0a68272537ab468c4b529d8d60af9e473a9c6ffd1ffd3 not found: ID does not exist" containerID="daeffcde17c313caceb0a68272537ab468c4b529d8d60af9e473a9c6ffd1ffd3" Oct 30 20:09:33.677074 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.677010 5088 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daeffcde17c313caceb0a68272537ab468c4b529d8d60af9e473a9c6ffd1ffd3"} err="failed to get container status \"daeffcde17c313caceb0a68272537ab468c4b529d8d60af9e473a9c6ffd1ffd3\": rpc error: code = NotFound desc = could not find container \"daeffcde17c313caceb0a68272537ab468c4b529d8d60af9e473a9c6ffd1ffd3\": container with ID starting with daeffcde17c313caceb0a68272537ab468c4b529d8d60af9e473a9c6ffd1ffd3 not found: ID does not exist" Oct 30 20:09:33.677074 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.677034 5088 scope.go:122] "RemoveContainer" containerID="4c13990224bbb25058dad815e340b413c6171b14b68a186e923c3ecf17a7c4fa" Oct 30 20:09:33.682202 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.682180 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98"] Oct 30 20:09:33.693449 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.693089 5088 scope.go:122] "RemoveContainer" containerID="4c13990224bbb25058dad815e340b413c6171b14b68a186e923c3ecf17a7c4fa" Oct 30 20:09:33.694195 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:09:33.694108 5088 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c13990224bbb25058dad815e340b413c6171b14b68a186e923c3ecf17a7c4fa\": container with ID starting with 4c13990224bbb25058dad815e340b413c6171b14b68a186e923c3ecf17a7c4fa not found: ID does not exist" containerID="4c13990224bbb25058dad815e340b413c6171b14b68a186e923c3ecf17a7c4fa" Oct 30 20:09:33.694195 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.694144 5088 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c13990224bbb25058dad815e340b413c6171b14b68a186e923c3ecf17a7c4fa"} err="failed to get container status \"4c13990224bbb25058dad815e340b413c6171b14b68a186e923c3ecf17a7c4fa\": rpc error: code = NotFound desc = could not find container \"4c13990224bbb25058dad815e340b413c6171b14b68a186e923c3ecf17a7c4fa\": container with ID starting with 4c13990224bbb25058dad815e340b413c6171b14b68a186e923c3ecf17a7c4fa not found: ID does not exist" Oct 30 20:09:33.699239 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.699213 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-22d98"] Oct 30 20:09:33.743298 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.743262 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5"] Oct 30 20:09:33.748983 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.748950 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-w29v5"] Oct 30 20:09:33.819790 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.819744 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2"] Oct 30 20:09:33.819971 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.819813 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-tlcl2"] Oct 30 20:09:33.854527 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.854484 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf"] Oct 30 20:09:33.857441 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:33.857419 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-efbac3-openshift-default-844cb4689c-8ctbf"] Oct 30 20:09:34.045700 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:34.044695 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55f6e9a1-znm9t"] Oct 30 20:09:34.051284 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:34.051250 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55f6e9a1-znm9t"] Oct 30 20:09:34.324033 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:34.323869 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-c2tb9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:09:34.324033 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:34.323965 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" podUID="ce0a1543-79bc-440d-ad18-c045ac745786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:34.755775 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:34.755728 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55d65e3f-e3a8-4836-9d57-50c3dfee5a0a" path="/var/lib/kubelet/pods/55d65e3f-e3a8-4836-9d57-50c3dfee5a0a/volumes" Oct 30 20:09:34.756511 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:34.756484 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="811db7c8-11d5-4e9d-8c8b-254c17906e84" path="/var/lib/kubelet/pods/811db7c8-11d5-4e9d-8c8b-254c17906e84/volumes" Oct 30 20:09:34.757295 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:34.757272 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a51889ad-a7b6-41f2-9857-679ac04b7bea" path="/var/lib/kubelet/pods/a51889ad-a7b6-41f2-9857-679ac04b7bea/volumes" Oct 30 20:09:34.758454 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:34.758430 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce337cd9-bd40-432b-998d-4357f13b46d1" path="/var/lib/kubelet/pods/ce337cd9-bd40-432b-998d-4357f13b46d1/volumes" Oct 30 20:09:34.759166 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:34.759143 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf9a8162-01f0-4226-b8f4-783125c277a2" path="/var/lib/kubelet/pods/cf9a8162-01f0-4226-b8f4-783125c277a2/volumes" Oct 30 20:09:35.201672 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.201580 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-774477d6c-kt5g5"] Oct 30 20:09:35.202222 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.202204 5088 cpu_manager.go:414] "containerMap: removing container" podUID="fec493c7-3f5a-459e-99e0-a4c816d5a907" containerName="main" Oct 30 20:09:35.202269 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.202224 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fec493c7-3f5a-459e-99e0-a4c816d5a907" containerName="main" Oct 30 20:09:35.202269 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.202245 5088 cpu_manager.go:414] "containerMap: removing container" podUID="811db7c8-11d5-4e9d-8c8b-254c17906e84" containerName="istio-proxy" Oct 30 20:09:35.202269 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.202250 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="811db7c8-11d5-4e9d-8c8b-254c17906e84" containerName="istio-proxy" Oct 30 20:09:35.202269 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.202258 5088 cpu_manager.go:414] "containerMap: removing container" podUID="ce337cd9-bd40-432b-998d-4357f13b46d1" containerName="istio-proxy" Oct 30 20:09:35.202269 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.202264 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce337cd9-bd40-432b-998d-4357f13b46d1" containerName="istio-proxy" Oct 30 20:09:35.202441 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.202275 5088 cpu_manager.go:414] "containerMap: removing container" podUID="c0748781-63b9-4af1-8eab-5d0cef276da6" containerName="istio-proxy" Oct 30 20:09:35.202441 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.202280 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0748781-63b9-4af1-8eab-5d0cef276da6" containerName="istio-proxy" Oct 30 20:09:35.202441 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.202295 5088 cpu_manager.go:414] "containerMap: removing container" podUID="4c27da71-bbe3-4855-b6aa-d46fb82e42c7" containerName="revoke-keys" Oct 30 20:09:35.202441 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.202302 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c27da71-bbe3-4855-b6aa-d46fb82e42c7" containerName="revoke-keys" Oct 30 20:09:35.202441 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.202319 5088 cpu_manager.go:414] "containerMap: removing container" podUID="0ee0e296-2cf3-4cdc-9340-d202e75c6a14" containerName="payload-pre-processing" Oct 30 20:09:35.202441 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.202327 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ee0e296-2cf3-4cdc-9340-d202e75c6a14" containerName="payload-pre-processing" Oct 30 20:09:35.202441 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.202341 5088 cpu_manager.go:414] "containerMap: removing container" podUID="ce337cd9-bd40-432b-998d-4357f13b46d1" containerName="istio-proxy" Oct 30 20:09:35.202441 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.202348 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce337cd9-bd40-432b-998d-4357f13b46d1" containerName="istio-proxy" Oct 30 20:09:35.202441 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.202360 5088 cpu_manager.go:414] "containerMap: removing container" podUID="e402d580-e9a6-4e35-a2ff-e5ef2d78eb90" containerName="istio-proxy" Oct 30 20:09:35.202441 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.202368 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e402d580-e9a6-4e35-a2ff-e5ef2d78eb90" containerName="istio-proxy" Oct 30 20:09:35.202441 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.202404 5088 cpu_manager.go:414] "containerMap: removing container" podUID="cf9a8162-01f0-4226-b8f4-783125c277a2" containerName="istio-proxy" Oct 30 20:09:35.202441 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.202412 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf9a8162-01f0-4226-b8f4-783125c277a2" containerName="istio-proxy" Oct 30 20:09:35.202441 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.202424 5088 cpu_manager.go:414] "containerMap: removing container" podUID="1f2b960b-8db8-4b84-b6d1-9ac12823e278" containerName="istio-proxy" Oct 30 20:09:35.202441 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.202431 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f2b960b-8db8-4b84-b6d1-9ac12823e278" containerName="istio-proxy" Oct 30 20:09:35.202874 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.202447 5088 cpu_manager.go:414] "containerMap: removing container" podUID="a51889ad-a7b6-41f2-9857-679ac04b7bea" containerName="istio-proxy" Oct 30 20:09:35.202874 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.202462 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a51889ad-a7b6-41f2-9857-679ac04b7bea" containerName="istio-proxy" Oct 30 20:09:35.202874 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.202478 5088 cpu_manager.go:414] "containerMap: removing container" podUID="2f2223b9-aebe-4aab-9437-af457ece8c69" containerName="payload-processing" Oct 30 20:09:35.202874 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.202484 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f2223b9-aebe-4aab-9437-af457ece8c69" containerName="payload-processing" Oct 30 20:09:35.202874 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.202497 5088 cpu_manager.go:414] "containerMap: removing container" podUID="c0748781-63b9-4af1-8eab-5d0cef276da6" containerName="istio-proxy" Oct 30 20:09:35.202874 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.202503 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0748781-63b9-4af1-8eab-5d0cef276da6" containerName="istio-proxy" Oct 30 20:09:35.202874 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.202520 5088 cpu_manager.go:414] "containerMap: removing container" podUID="be2ffe6a-d355-4d95-a7ae-46adeb952736" containerName="maas-api" Oct 30 20:09:35.202874 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.202546 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be2ffe6a-d355-4d95-a7ae-46adeb952736" containerName="maas-api" Oct 30 20:09:35.202874 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.202565 5088 cpu_manager.go:414] "containerMap: removing container" podUID="ab7bcd6d-197b-4c1e-9ce1-4aa111e96e54" containerName="payload-pre-processing" Oct 30 20:09:35.202874 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.202571 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab7bcd6d-197b-4c1e-9ce1-4aa111e96e54" containerName="payload-pre-processing" Oct 30 20:09:35.202874 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.202684 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="e402d580-e9a6-4e35-a2ff-e5ef2d78eb90" containerName="istio-proxy" Oct 30 20:09:35.202874 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.202695 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf9a8162-01f0-4226-b8f4-783125c277a2" containerName="istio-proxy" Oct 30 20:09:35.202874 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.202703 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f2b960b-8db8-4b84-b6d1-9ac12823e278" containerName="istio-proxy" Oct 30 20:09:35.202874 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.202712 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="a51889ad-a7b6-41f2-9857-679ac04b7bea" containerName="istio-proxy" Oct 30 20:09:35.202874 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.202722 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f2223b9-aebe-4aab-9437-af457ece8c69" containerName="payload-processing" Oct 30 20:09:35.202874 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.202729 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0748781-63b9-4af1-8eab-5d0cef276da6" containerName="istio-proxy" Oct 30 20:09:35.202874 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.202735 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="be2ffe6a-d355-4d95-a7ae-46adeb952736" containerName="maas-api" Oct 30 20:09:35.202874 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.202743 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab7bcd6d-197b-4c1e-9ce1-4aa111e96e54" containerName="payload-pre-processing" Oct 30 20:09:35.202874 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.202751 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="fec493c7-3f5a-459e-99e0-a4c816d5a907" containerName="main" Oct 30 20:09:35.202874 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.202757 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="811db7c8-11d5-4e9d-8c8b-254c17906e84" containerName="istio-proxy" Oct 30 20:09:35.202874 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.202763 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce337cd9-bd40-432b-998d-4357f13b46d1" containerName="istio-proxy" Oct 30 20:09:35.202874 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.202771 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c27da71-bbe3-4855-b6aa-d46fb82e42c7" containerName="revoke-keys" Oct 30 20:09:35.202874 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.202779 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ee0e296-2cf3-4cdc-9340-d202e75c6a14" containerName="payload-pre-processing" Oct 30 20:09:35.202874 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.202788 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce337cd9-bd40-432b-998d-4357f13b46d1" containerName="istio-proxy" Oct 30 20:09:35.207759 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.207738 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-774477d6c-kt5g5" Oct 30 20:09:35.238798 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.238766 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 20:09:35.238798 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.238766 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wcnv4\"" Oct 30 20:09:35.238798 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.238777 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 20:09:35.244323 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.244293 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-774477d6c-kt5g5"] Oct 30 20:09:35.252937 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.252916 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 20:09:35.281914 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.281879 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/326e09b7-8038-4806-9589-e991b674cbad-metrics-certs\") pod \"maas-controller-774477d6c-kt5g5\" (UID: \"326e09b7-8038-4806-9589-e991b674cbad\") " pod="opendatahub/maas-controller-774477d6c-kt5g5" Oct 30 20:09:35.281914 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.281919 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqgft\" (UniqueName: \"kubernetes.io/projected/326e09b7-8038-4806-9589-e991b674cbad-kube-api-access-kqgft\") pod \"maas-controller-774477d6c-kt5g5\" (UID: \"326e09b7-8038-4806-9589-e991b674cbad\") " pod="opendatahub/maas-controller-774477d6c-kt5g5" Oct 30 20:09:35.282111 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.282070 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/326e09b7-8038-4806-9589-e991b674cbad-webhook-cert\") pod \"maas-controller-774477d6c-kt5g5\" (UID: \"326e09b7-8038-4806-9589-e991b674cbad\") " pod="opendatahub/maas-controller-774477d6c-kt5g5" Oct 30 20:09:35.383364 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.383312 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/326e09b7-8038-4806-9589-e991b674cbad-webhook-cert\") pod \"maas-controller-774477d6c-kt5g5\" (UID: \"326e09b7-8038-4806-9589-e991b674cbad\") " pod="opendatahub/maas-controller-774477d6c-kt5g5" Oct 30 20:09:35.383364 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.383368 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/326e09b7-8038-4806-9589-e991b674cbad-metrics-certs\") pod \"maas-controller-774477d6c-kt5g5\" (UID: \"326e09b7-8038-4806-9589-e991b674cbad\") " pod="opendatahub/maas-controller-774477d6c-kt5g5" Oct 30 20:09:35.383881 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.383390 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqgft\" (UniqueName: \"kubernetes.io/projected/326e09b7-8038-4806-9589-e991b674cbad-kube-api-access-kqgft\") pod \"maas-controller-774477d6c-kt5g5\" (UID: \"326e09b7-8038-4806-9589-e991b674cbad\") " pod="opendatahub/maas-controller-774477d6c-kt5g5" Oct 30 20:09:35.386206 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.386168 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/326e09b7-8038-4806-9589-e991b674cbad-webhook-cert\") pod \"maas-controller-774477d6c-kt5g5\" (UID: \"326e09b7-8038-4806-9589-e991b674cbad\") " pod="opendatahub/maas-controller-774477d6c-kt5g5" Oct 30 20:09:35.386336 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.386291 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/326e09b7-8038-4806-9589-e991b674cbad-metrics-certs\") pod \"maas-controller-774477d6c-kt5g5\" (UID: \"326e09b7-8038-4806-9589-e991b674cbad\") " pod="opendatahub/maas-controller-774477d6c-kt5g5" Oct 30 20:09:35.410282 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.410227 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqgft\" (UniqueName: \"kubernetes.io/projected/326e09b7-8038-4806-9589-e991b674cbad-kube-api-access-kqgft\") pod \"maas-controller-774477d6c-kt5g5\" (UID: \"326e09b7-8038-4806-9589-e991b674cbad\") " pod="opendatahub/maas-controller-774477d6c-kt5g5" Oct 30 20:09:35.521341 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.521304 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-774477d6c-kt5g5" Oct 30 20:09:35.832519 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.832495 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-774477d6c-kt5g5"] Oct 30 20:09:35.833482 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:09:35.833455 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod326e09b7_8038_4806_9589_e991b674cbad.slice/crio-2441fb083266bfee27d5bc239781e55ffd68419b7f520747fedb7fec03b76f8f WatchSource:0}: Error finding container 2441fb083266bfee27d5bc239781e55ffd68419b7f520747fedb7fec03b76f8f: Status 404 returned error can't find the container with id 2441fb083266bfee27d5bc239781e55ffd68419b7f520747fedb7fec03b76f8f Oct 30 20:09:35.834691 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:35.834671 5088 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:09:36.611084 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:36.611047 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-774477d6c-kt5g5" event={"ID":"326e09b7-8038-4806-9589-e991b674cbad","Type":"ContainerStarted","Data":"2441fb083266bfee27d5bc239781e55ffd68419b7f520747fedb7fec03b76f8f"} Oct 30 20:09:37.618515 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:37.618477 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-774477d6c-kt5g5" event={"ID":"326e09b7-8038-4806-9589-e991b674cbad","Type":"ContainerStarted","Data":"b6ce99d6e55a4de61d508984971ed79872ade90540225bc42093df154d9a2e68"} Oct 30 20:09:37.619052 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:37.618545 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-774477d6c-kt5g5" Oct 30 20:09:37.666031 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:37.665971 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-774477d6c-kt5g5" podStartSLOduration=1.050903946 podStartE2EDuration="2.665952821s" totalImagesPullingTime="1.615048875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:09:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:37.665285395 +0000 UTC m=+1339.077999195" watchObservedRunningTime="2026-10-30 20:09:37.665952821 +0000 UTC m=+1339.078666600" Oct 30 20:09:39.050130 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:39.050098 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5148e465-4pb95"] Oct 30 20:09:39.062447 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:39.062416 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5148e465-4pb95"] Oct 30 20:09:40.037690 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:40.037647 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2757fd15-pbhr2"] Oct 30 20:09:40.051025 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:40.050980 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2757fd15-pbhr2"] Oct 30 20:09:40.739039 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:40.739001 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="784739f5-87b2-40e2-829e-95e3cab8148d" path="/var/lib/kubelet/pods/784739f5-87b2-40e2-829e-95e3cab8148d/volumes" Oct 30 20:09:40.739428 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:40.739413 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bae93c7f-9009-40d5-8c0f-1157e00a46dc" path="/var/lib/kubelet/pods/bae93c7f-9009-40d5-8c0f-1157e00a46dc/volumes" Oct 30 20:09:45.039542 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:45.039508 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-320efee7-2tctv"] Oct 30 20:09:45.052573 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:45.052539 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-320efee7-2tctv"] Oct 30 20:09:46.738877 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:46.738838 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3c58e20-4219-4650-af7e-bdf6795c8cf9" path="/var/lib/kubelet/pods/b3c58e20-4219-4650-af7e-bdf6795c8cf9/volumes" Oct 30 20:09:48.627331 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:48.627302 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-774477d6c-kt5g5" Oct 30 20:09:52.437840 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:09:52.437801 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:09:56.064185 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:56.064144 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-381315a4-w9c7h"] Oct 30 20:09:56.083674 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:56.083640 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-381315a4-w9c7h"] Oct 30 20:09:56.738212 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:56.738180 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ded4b273-9ae0-4ddf-bc70-154f4a67a31c" path="/var/lib/kubelet/pods/ded4b273-9ae0-4ddf-bc70-154f4a67a31c/volumes" Oct 30 20:09:56.822125 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:56.822092 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th"] Oct 30 20:09:56.822694 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:56.822667 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0748781-63b9-4af1-8eab-5d0cef276da6" containerName="istio-proxy" Oct 30 20:09:56.827349 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:56.827326 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" Oct 30 20:09:56.848627 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:56.848591 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq"] Oct 30 20:09:56.852548 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:56.852525 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" Oct 30 20:09:56.864352 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:56.864322 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th"] Oct 30 20:09:56.879799 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:56.879763 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq"] Oct 30 20:09:56.897300 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:56.897271 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4045e4a8-b80c-4b45-96b7-e9e4203ebe5e-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ms5th\" (UID: \"4045e4a8-b80c-4b45-96b7-e9e4203ebe5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" Oct 30 20:09:56.897524 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:56.897501 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4045e4a8-b80c-4b45-96b7-e9e4203ebe5e-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ms5th\" (UID: \"4045e4a8-b80c-4b45-96b7-e9e4203ebe5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" Oct 30 20:09:56.897637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:56.897538 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4045e4a8-b80c-4b45-96b7-e9e4203ebe5e-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ms5th\" (UID: \"4045e4a8-b80c-4b45-96b7-e9e4203ebe5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" Oct 30 20:09:56.897637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:56.897570 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dsr6\" (UniqueName: \"kubernetes.io/projected/4045e4a8-b80c-4b45-96b7-e9e4203ebe5e-kube-api-access-7dsr6\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ms5th\" (UID: \"4045e4a8-b80c-4b45-96b7-e9e4203ebe5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" Oct 30 20:09:56.897819 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:56.897791 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4045e4a8-b80c-4b45-96b7-e9e4203ebe5e-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ms5th\" (UID: \"4045e4a8-b80c-4b45-96b7-e9e4203ebe5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" Oct 30 20:09:56.897901 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:56.897843 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4045e4a8-b80c-4b45-96b7-e9e4203ebe5e-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ms5th\" (UID: \"4045e4a8-b80c-4b45-96b7-e9e4203ebe5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" Oct 30 20:09:56.898014 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:56.897991 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4045e4a8-b80c-4b45-96b7-e9e4203ebe5e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ms5th\" (UID: \"4045e4a8-b80c-4b45-96b7-e9e4203ebe5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" Oct 30 20:09:56.898075 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:56.898058 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4045e4a8-b80c-4b45-96b7-e9e4203ebe5e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ms5th\" (UID: \"4045e4a8-b80c-4b45-96b7-e9e4203ebe5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" Oct 30 20:09:56.898124 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:56.898110 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4045e4a8-b80c-4b45-96b7-e9e4203ebe5e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ms5th\" (UID: \"4045e4a8-b80c-4b45-96b7-e9e4203ebe5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" Oct 30 20:09:56.999087 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:56.998993 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4045e4a8-b80c-4b45-96b7-e9e4203ebe5e-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ms5th\" (UID: \"4045e4a8-b80c-4b45-96b7-e9e4203ebe5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" Oct 30 20:09:56.999087 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:56.999031 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4045e4a8-b80c-4b45-96b7-e9e4203ebe5e-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ms5th\" (UID: \"4045e4a8-b80c-4b45-96b7-e9e4203ebe5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" Oct 30 20:09:56.999087 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:56.999050 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4045e4a8-b80c-4b45-96b7-e9e4203ebe5e-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ms5th\" (UID: \"4045e4a8-b80c-4b45-96b7-e9e4203ebe5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" Oct 30 20:09:56.999376 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:56.999181 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dsr6\" (UniqueName: \"kubernetes.io/projected/4045e4a8-b80c-4b45-96b7-e9e4203ebe5e-kube-api-access-7dsr6\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ms5th\" (UID: \"4045e4a8-b80c-4b45-96b7-e9e4203ebe5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" Oct 30 20:09:56.999376 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:56.999246 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4045e4a8-b80c-4b45-96b7-e9e4203ebe5e-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ms5th\" (UID: \"4045e4a8-b80c-4b45-96b7-e9e4203ebe5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" Oct 30 20:09:56.999376 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:56.999279 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7ff5ff7-5866-473f-9055-e8c74f9380bc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lnxpq\" (UID: \"f7ff5ff7-5866-473f-9055-e8c74f9380bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" Oct 30 20:09:56.999376 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:56.999301 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4045e4a8-b80c-4b45-96b7-e9e4203ebe5e-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ms5th\" (UID: \"4045e4a8-b80c-4b45-96b7-e9e4203ebe5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" Oct 30 20:09:56.999376 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:56.999322 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7ff5ff7-5866-473f-9055-e8c74f9380bc-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lnxpq\" (UID: \"f7ff5ff7-5866-473f-9055-e8c74f9380bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" Oct 30 20:09:56.999672 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:56.999390 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7ff5ff7-5866-473f-9055-e8c74f9380bc-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lnxpq\" (UID: \"f7ff5ff7-5866-473f-9055-e8c74f9380bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" Oct 30 20:09:56.999672 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:56.999480 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7ff5ff7-5866-473f-9055-e8c74f9380bc-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lnxpq\" (UID: \"f7ff5ff7-5866-473f-9055-e8c74f9380bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" Oct 30 20:09:56.999672 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:56.999498 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4045e4a8-b80c-4b45-96b7-e9e4203ebe5e-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ms5th\" (UID: \"4045e4a8-b80c-4b45-96b7-e9e4203ebe5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" Oct 30 20:09:56.999672 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:56.999517 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjrkf\" (UniqueName: \"kubernetes.io/projected/f7ff5ff7-5866-473f-9055-e8c74f9380bc-kube-api-access-bjrkf\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lnxpq\" (UID: \"f7ff5ff7-5866-473f-9055-e8c74f9380bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" Oct 30 20:09:56.999672 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:56.999639 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4045e4a8-b80c-4b45-96b7-e9e4203ebe5e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ms5th\" (UID: \"4045e4a8-b80c-4b45-96b7-e9e4203ebe5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" Oct 30 20:09:56.999930 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:56.999701 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4045e4a8-b80c-4b45-96b7-e9e4203ebe5e-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ms5th\" (UID: \"4045e4a8-b80c-4b45-96b7-e9e4203ebe5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" Oct 30 20:09:56.999930 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:56.999708 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4045e4a8-b80c-4b45-96b7-e9e4203ebe5e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ms5th\" (UID: \"4045e4a8-b80c-4b45-96b7-e9e4203ebe5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" Oct 30 20:09:56.999930 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:56.999763 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7ff5ff7-5866-473f-9055-e8c74f9380bc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lnxpq\" (UID: \"f7ff5ff7-5866-473f-9055-e8c74f9380bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" Oct 30 20:09:56.999930 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:56.999765 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4045e4a8-b80c-4b45-96b7-e9e4203ebe5e-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ms5th\" (UID: \"4045e4a8-b80c-4b45-96b7-e9e4203ebe5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" Oct 30 20:09:56.999930 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:56.999801 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4045e4a8-b80c-4b45-96b7-e9e4203ebe5e-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ms5th\" (UID: \"4045e4a8-b80c-4b45-96b7-e9e4203ebe5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" Oct 30 20:09:56.999930 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:56.999810 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7ff5ff7-5866-473f-9055-e8c74f9380bc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lnxpq\" (UID: \"f7ff5ff7-5866-473f-9055-e8c74f9380bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" Oct 30 20:09:56.999930 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:56.999865 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4045e4a8-b80c-4b45-96b7-e9e4203ebe5e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ms5th\" (UID: \"4045e4a8-b80c-4b45-96b7-e9e4203ebe5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" Oct 30 20:09:56.999930 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:56.999906 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7ff5ff7-5866-473f-9055-e8c74f9380bc-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lnxpq\" (UID: \"f7ff5ff7-5866-473f-9055-e8c74f9380bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" Oct 30 20:09:57.000328 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:56.999969 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7ff5ff7-5866-473f-9055-e8c74f9380bc-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lnxpq\" (UID: \"f7ff5ff7-5866-473f-9055-e8c74f9380bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" Oct 30 20:09:57.000328 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.000252 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4045e4a8-b80c-4b45-96b7-e9e4203ebe5e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ms5th\" (UID: \"4045e4a8-b80c-4b45-96b7-e9e4203ebe5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" Oct 30 20:09:57.002051 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.002018 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4045e4a8-b80c-4b45-96b7-e9e4203ebe5e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ms5th\" (UID: \"4045e4a8-b80c-4b45-96b7-e9e4203ebe5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" Oct 30 20:09:57.002306 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.002286 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4045e4a8-b80c-4b45-96b7-e9e4203ebe5e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ms5th\" (UID: \"4045e4a8-b80c-4b45-96b7-e9e4203ebe5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" Oct 30 20:09:57.025683 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.025656 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4045e4a8-b80c-4b45-96b7-e9e4203ebe5e-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ms5th\" (UID: \"4045e4a8-b80c-4b45-96b7-e9e4203ebe5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" Oct 30 20:09:57.025854 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.025743 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dsr6\" (UniqueName: \"kubernetes.io/projected/4045e4a8-b80c-4b45-96b7-e9e4203ebe5e-kube-api-access-7dsr6\") pod \"maas-default-gateway-openshift-default-7f5667f86c-ms5th\" (UID: \"4045e4a8-b80c-4b45-96b7-e9e4203ebe5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" Oct 30 20:09:57.101028 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.100983 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7ff5ff7-5866-473f-9055-e8c74f9380bc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lnxpq\" (UID: \"f7ff5ff7-5866-473f-9055-e8c74f9380bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" Oct 30 20:09:57.101535 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.101042 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7ff5ff7-5866-473f-9055-e8c74f9380bc-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lnxpq\" (UID: \"f7ff5ff7-5866-473f-9055-e8c74f9380bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" Oct 30 20:09:57.101535 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.101075 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7ff5ff7-5866-473f-9055-e8c74f9380bc-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lnxpq\" (UID: \"f7ff5ff7-5866-473f-9055-e8c74f9380bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" Oct 30 20:09:57.101535 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.101109 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7ff5ff7-5866-473f-9055-e8c74f9380bc-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lnxpq\" (UID: \"f7ff5ff7-5866-473f-9055-e8c74f9380bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" Oct 30 20:09:57.101535 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.101143 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjrkf\" (UniqueName: \"kubernetes.io/projected/f7ff5ff7-5866-473f-9055-e8c74f9380bc-kube-api-access-bjrkf\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lnxpq\" (UID: \"f7ff5ff7-5866-473f-9055-e8c74f9380bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" Oct 30 20:09:57.101535 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.101218 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7ff5ff7-5866-473f-9055-e8c74f9380bc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lnxpq\" (UID: \"f7ff5ff7-5866-473f-9055-e8c74f9380bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" Oct 30 20:09:57.101535 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.101239 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7ff5ff7-5866-473f-9055-e8c74f9380bc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lnxpq\" (UID: \"f7ff5ff7-5866-473f-9055-e8c74f9380bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" Oct 30 20:09:57.101535 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.101280 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7ff5ff7-5866-473f-9055-e8c74f9380bc-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lnxpq\" (UID: \"f7ff5ff7-5866-473f-9055-e8c74f9380bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" Oct 30 20:09:57.101535 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.101346 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7ff5ff7-5866-473f-9055-e8c74f9380bc-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lnxpq\" (UID: \"f7ff5ff7-5866-473f-9055-e8c74f9380bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" Oct 30 20:09:57.101957 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.101541 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7ff5ff7-5866-473f-9055-e8c74f9380bc-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lnxpq\" (UID: \"f7ff5ff7-5866-473f-9055-e8c74f9380bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" Oct 30 20:09:57.101957 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.101674 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7ff5ff7-5866-473f-9055-e8c74f9380bc-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lnxpq\" (UID: \"f7ff5ff7-5866-473f-9055-e8c74f9380bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" Oct 30 20:09:57.101957 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.101746 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7ff5ff7-5866-473f-9055-e8c74f9380bc-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lnxpq\" (UID: \"f7ff5ff7-5866-473f-9055-e8c74f9380bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" Oct 30 20:09:57.101957 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.101812 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7ff5ff7-5866-473f-9055-e8c74f9380bc-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lnxpq\" (UID: \"f7ff5ff7-5866-473f-9055-e8c74f9380bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" Oct 30 20:09:57.101957 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.101931 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7ff5ff7-5866-473f-9055-e8c74f9380bc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lnxpq\" (UID: \"f7ff5ff7-5866-473f-9055-e8c74f9380bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" Oct 30 20:09:57.103575 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.103557 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7ff5ff7-5866-473f-9055-e8c74f9380bc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lnxpq\" (UID: \"f7ff5ff7-5866-473f-9055-e8c74f9380bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" Oct 30 20:09:57.103717 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.103701 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7ff5ff7-5866-473f-9055-e8c74f9380bc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lnxpq\" (UID: \"f7ff5ff7-5866-473f-9055-e8c74f9380bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" Oct 30 20:09:57.128439 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.128397 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7ff5ff7-5866-473f-9055-e8c74f9380bc-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lnxpq\" (UID: \"f7ff5ff7-5866-473f-9055-e8c74f9380bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" Oct 30 20:09:57.128639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.128579 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjrkf\" (UniqueName: \"kubernetes.io/projected/f7ff5ff7-5866-473f-9055-e8c74f9380bc-kube-api-access-bjrkf\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lnxpq\" (UID: \"f7ff5ff7-5866-473f-9055-e8c74f9380bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" Oct 30 20:09:57.141233 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.141195 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" Oct 30 20:09:57.168174 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.168136 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" Oct 30 20:09:57.168915 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.168877 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fb6dw"] Oct 30 20:09:57.173268 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.173236 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fb6dw" Oct 30 20:09:57.191778 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.191474 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:09:57.191778 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.191686 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:09:57.191778 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.191714 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:09:57.191778 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.191751 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:09:57.192091 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.192047 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fb6dw"] Oct 30 20:09:57.303396 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.303362 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1be427d5-7034-4669-bc89-4986e9fe3ceb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-fb6dw\" (UID: \"1be427d5-7034-4669-bc89-4986e9fe3ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fb6dw" Oct 30 20:09:57.303576 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.303448 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx2kq\" (UniqueName: \"kubernetes.io/projected/1be427d5-7034-4669-bc89-4986e9fe3ceb-kube-api-access-bx2kq\") pod \"maas-api-revoke-sub-85237e45-fb6dw\" (UID: \"1be427d5-7034-4669-bc89-4986e9fe3ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fb6dw" Oct 30 20:09:57.303576 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.303485 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1be427d5-7034-4669-bc89-4986e9fe3ceb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-fb6dw\" (UID: \"1be427d5-7034-4669-bc89-4986e9fe3ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fb6dw" Oct 30 20:09:57.404678 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.404642 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx2kq\" (UniqueName: \"kubernetes.io/projected/1be427d5-7034-4669-bc89-4986e9fe3ceb-kube-api-access-bx2kq\") pod \"maas-api-revoke-sub-85237e45-fb6dw\" (UID: \"1be427d5-7034-4669-bc89-4986e9fe3ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fb6dw" Oct 30 20:09:57.404678 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.404687 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1be427d5-7034-4669-bc89-4986e9fe3ceb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-fb6dw\" (UID: \"1be427d5-7034-4669-bc89-4986e9fe3ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fb6dw" Oct 30 20:09:57.405014 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.404761 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1be427d5-7034-4669-bc89-4986e9fe3ceb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-fb6dw\" (UID: \"1be427d5-7034-4669-bc89-4986e9fe3ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fb6dw" Oct 30 20:09:57.405409 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.405386 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1be427d5-7034-4669-bc89-4986e9fe3ceb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-fb6dw\" (UID: \"1be427d5-7034-4669-bc89-4986e9fe3ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fb6dw" Oct 30 20:09:57.407215 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.407190 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1be427d5-7034-4669-bc89-4986e9fe3ceb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-fb6dw\" (UID: \"1be427d5-7034-4669-bc89-4986e9fe3ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fb6dw" Oct 30 20:09:57.431568 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.431538 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx2kq\" (UniqueName: \"kubernetes.io/projected/1be427d5-7034-4669-bc89-4986e9fe3ceb-kube-api-access-bx2kq\") pod \"maas-api-revoke-sub-85237e45-fb6dw\" (UID: \"1be427d5-7034-4669-bc89-4986e9fe3ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fb6dw" Oct 30 20:09:57.482551 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.482518 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th"] Oct 30 20:09:57.482945 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:09:57.482924 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4045e4a8_b80c_4b45_96b7_e9e4203ebe5e.slice/crio-c4c9d79a110cdd5fc9212156daa571666019c747333538154e04f2bc79f340f1 WatchSource:0}: Error finding container c4c9d79a110cdd5fc9212156daa571666019c747333538154e04f2bc79f340f1: Status 404 returned error can't find the container with id c4c9d79a110cdd5fc9212156daa571666019c747333538154e04f2bc79f340f1 Oct 30 20:09:57.485185 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.485152 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:09:57.485274 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.485219 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:09:57.485274 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.485249 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:09:57.493972 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.493950 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fb6dw" Oct 30 20:09:57.511373 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.511343 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq"] Oct 30 20:09:57.514220 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:09:57.514194 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7ff5ff7_5866_473f_9055_e8c74f9380bc.slice/crio-8c0823c697c72643d067a3a01ae20cb94ea386a8ca9a13484e1576bdc4716dbb WatchSource:0}: Error finding container 8c0823c697c72643d067a3a01ae20cb94ea386a8ca9a13484e1576bdc4716dbb: Status 404 returned error can't find the container with id 8c0823c697c72643d067a3a01ae20cb94ea386a8ca9a13484e1576bdc4716dbb Oct 30 20:09:57.523647 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.523537 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:09:57.523768 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.523718 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:09:57.523820 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.523776 5088 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:09:57.757243 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.756561 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" event={"ID":"4045e4a8-b80c-4b45-96b7-e9e4203ebe5e","Type":"ContainerStarted","Data":"798f80ea2b54eabb1152b3e245bf49aa86d15d618eb63aaeb58f8f0f0abb7a75"} Oct 30 20:09:57.757243 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.756647 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" event={"ID":"4045e4a8-b80c-4b45-96b7-e9e4203ebe5e","Type":"ContainerStarted","Data":"c4c9d79a110cdd5fc9212156daa571666019c747333538154e04f2bc79f340f1"} Oct 30 20:09:57.758492 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.758419 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" event={"ID":"f7ff5ff7-5866-473f-9055-e8c74f9380bc","Type":"ContainerStarted","Data":"8448b85f796c5699b1bb898302a4ccbb92bccfccecfa7956b9b12353c7610d98"} Oct 30 20:09:57.758492 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.758465 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" event={"ID":"f7ff5ff7-5866-473f-9055-e8c74f9380bc","Type":"ContainerStarted","Data":"8c0823c697c72643d067a3a01ae20cb94ea386a8ca9a13484e1576bdc4716dbb"} Oct 30 20:09:57.807767 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.807696 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" podStartSLOduration=1.807673257 podStartE2EDuration="1.807673257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:09:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:57.806301811 +0000 UTC m=+1359.219015615" watchObservedRunningTime="2026-10-30 20:09:57.807673257 +0000 UTC m=+1359.220387037" Oct 30 20:09:57.817710 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:09:57.817669 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1be427d5_7034_4669_bc89_4986e9fe3ceb.slice/crio-3848ff2e4692a0d1a837312e0019759def1ab9fa9c2d9ee452014318d152695c WatchSource:0}: Error finding container 3848ff2e4692a0d1a837312e0019759def1ab9fa9c2d9ee452014318d152695c: Status 404 returned error can't find the container with id 3848ff2e4692a0d1a837312e0019759def1ab9fa9c2d9ee452014318d152695c Oct 30 20:09:57.822796 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.822717 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fb6dw"] Oct 30 20:09:57.858839 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:57.858779 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" podStartSLOduration=1.858758485 podStartE2EDuration="1.858758485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:09:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:57.855189422 +0000 UTC m=+1359.267903201" watchObservedRunningTime="2026-10-30 20:09:57.858758485 +0000 UTC m=+1359.271472270" Oct 30 20:09:58.142074 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:58.141978 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" Oct 30 20:09:58.143080 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:58.143056 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-ms5th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 20:09:58.143133 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:58.143103 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" podUID="4045e4a8-b80c-4b45-96b7-e9e4203ebe5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 20:09:58.170226 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:58.170199 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" Oct 30 20:09:58.171642 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:58.171606 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lnxpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 20:09:58.171760 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:58.171674 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" podUID="f7ff5ff7-5866-473f-9055-e8c74f9380bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 20:09:58.821637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:58.820842 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fb6dw" event={"ID":"1be427d5-7034-4669-bc89-4986e9fe3ceb","Type":"ContainerStarted","Data":"c302df3e92694e010371046019929f6c93910cedebc8f040354bdc056abace48"} Oct 30 20:09:58.821637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:58.820930 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fb6dw" event={"ID":"1be427d5-7034-4669-bc89-4986e9fe3ceb","Type":"ContainerStarted","Data":"3848ff2e4692a0d1a837312e0019759def1ab9fa9c2d9ee452014318d152695c"} Oct 30 20:09:58.880069 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:58.879999 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fb6dw" podStartSLOduration=1.879978034 podStartE2EDuration="1.879978034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:09:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:58.878708866 +0000 UTC m=+1360.291422647" watchObservedRunningTime="2026-10-30 20:09:58.879978034 +0000 UTC m=+1360.292691816" Oct 30 20:09:59.142472 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:59.142373 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-ms5th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 20:09:59.142472 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:59.142440 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" podUID="4045e4a8-b80c-4b45-96b7-e9e4203ebe5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 20:09:59.170965 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:59.170926 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lnxpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 20:09:59.171116 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:59.170979 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" podUID="f7ff5ff7-5866-473f-9055-e8c74f9380bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 20:09:59.838957 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:59.838924 5088 generic.go:356] "Generic (PLEG): container finished" podID="1be427d5-7034-4669-bc89-4986e9fe3ceb" containerID="c302df3e92694e010371046019929f6c93910cedebc8f040354bdc056abace48" exitCode=0 Oct 30 20:09:59.839591 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:09:59.839098 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fb6dw" event={"ID":"1be427d5-7034-4669-bc89-4986e9fe3ceb","Type":"ContainerDied","Data":"c302df3e92694e010371046019929f6c93910cedebc8f040354bdc056abace48"} Oct 30 20:10:00.142236 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:00.142148 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-ms5th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 20:10:00.142236 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:00.142211 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" podUID="4045e4a8-b80c-4b45-96b7-e9e4203ebe5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 20:10:00.170961 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:00.170930 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lnxpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 20:10:00.171403 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:00.171381 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" podUID="f7ff5ff7-5866-473f-9055-e8c74f9380bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 20:10:01.072869 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:01.072844 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fb6dw" Oct 30 20:10:01.142366 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:01.142290 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-ms5th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 20:10:01.142657 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:01.142605 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" podUID="4045e4a8-b80c-4b45-96b7-e9e4203ebe5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 20:10:01.170819 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:01.170784 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lnxpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 20:10:01.170998 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:01.170838 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" podUID="f7ff5ff7-5866-473f-9055-e8c74f9380bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 20:10:01.172179 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:01.172148 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1be427d5-7034-4669-bc89-4986e9fe3ceb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1be427d5-7034-4669-bc89-4986e9fe3ceb-maas-api-openshift-ca\") pod \"1be427d5-7034-4669-bc89-4986e9fe3ceb\" (UID: \"1be427d5-7034-4669-bc89-4986e9fe3ceb\") " Oct 30 20:10:01.172303 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:01.172264 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1be427d5-7034-4669-bc89-4986e9fe3ceb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1be427d5-7034-4669-bc89-4986e9fe3ceb-maas-api-xks-ca\") pod \"1be427d5-7034-4669-bc89-4986e9fe3ceb\" (UID: \"1be427d5-7034-4669-bc89-4986e9fe3ceb\") " Oct 30 20:10:01.172393 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:01.172359 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1be427d5-7034-4669-bc89-4986e9fe3ceb-kube-api-access-bx2kq\" (UniqueName: \"kubernetes.io/projected/1be427d5-7034-4669-bc89-4986e9fe3ceb-kube-api-access-bx2kq\") pod \"1be427d5-7034-4669-bc89-4986e9fe3ceb\" (UID: \"1be427d5-7034-4669-bc89-4986e9fe3ceb\") " Oct 30 20:10:01.173124 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:01.173092 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1be427d5-7034-4669-bc89-4986e9fe3ceb-maas-api-openshift-ca" pod "1be427d5-7034-4669-bc89-4986e9fe3ceb" (UID: "1be427d5-7034-4669-bc89-4986e9fe3ceb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:10:01.174412 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:01.174385 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1be427d5-7034-4669-bc89-4986e9fe3ceb-kube-api-access-bx2kq" pod "1be427d5-7034-4669-bc89-4986e9fe3ceb" (UID: "1be427d5-7034-4669-bc89-4986e9fe3ceb"). InnerVolumeSpecName "kube-api-access-bx2kq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:10:01.175107 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:01.175084 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1be427d5-7034-4669-bc89-4986e9fe3ceb-maas-api-xks-ca" pod "1be427d5-7034-4669-bc89-4986e9fe3ceb" (UID: "1be427d5-7034-4669-bc89-4986e9fe3ceb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:10:01.273450 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:01.273409 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1be427d5-7034-4669-bc89-4986e9fe3ceb-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:10:01.273450 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:01.273441 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx2kq\" (UniqueName: \"kubernetes.io/projected/1be427d5-7034-4669-bc89-4986e9fe3ceb-kube-api-access-bx2kq\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:10:01.273450 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:01.273450 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1be427d5-7034-4669-bc89-4986e9fe3ceb-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:10:01.853837 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:01.853809 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fb6dw" Oct 30 20:10:01.854296 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:01.853803 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fb6dw" event={"ID":"1be427d5-7034-4669-bc89-4986e9fe3ceb","Type":"ContainerDied","Data":"3848ff2e4692a0d1a837312e0019759def1ab9fa9c2d9ee452014318d152695c"} Oct 30 20:10:01.854418 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:01.854405 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3848ff2e4692a0d1a837312e0019759def1ab9fa9c2d9ee452014318d152695c" Oct 30 20:10:02.142504 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:02.142414 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-ms5th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 20:10:02.142504 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:02.142469 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" podUID="4045e4a8-b80c-4b45-96b7-e9e4203ebe5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 20:10:02.170660 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:02.170626 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lnxpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 20:10:02.170824 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:02.170681 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" podUID="f7ff5ff7-5866-473f-9055-e8c74f9380bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 20:10:03.141930 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:03.141889 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-ms5th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 20:10:03.142353 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:03.141947 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" podUID="4045e4a8-b80c-4b45-96b7-e9e4203ebe5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 20:10:03.170257 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:03.170214 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lnxpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 20:10:03.170447 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:03.170271 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" podUID="f7ff5ff7-5866-473f-9055-e8c74f9380bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 20:10:04.141718 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:04.141682 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-ms5th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 20:10:04.141888 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:04.141739 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" podUID="4045e4a8-b80c-4b45-96b7-e9e4203ebe5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 20:10:04.170750 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:04.170712 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lnxpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 20:10:04.171147 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:04.170783 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" podUID="f7ff5ff7-5866-473f-9055-e8c74f9380bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 20:10:05.042506 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:05.042462 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6035ee0c-wphdw"] Oct 30 20:10:05.055808 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:05.055771 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6035ee0c-wphdw"] Oct 30 20:10:05.141672 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:05.141635 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-ms5th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 20:10:05.141844 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:05.141698 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" podUID="4045e4a8-b80c-4b45-96b7-e9e4203ebe5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 20:10:05.170808 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:05.170771 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lnxpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 20:10:05.171217 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:05.170833 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" podUID="f7ff5ff7-5866-473f-9055-e8c74f9380bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 20:10:06.739699 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:06.739665 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb4bf3de-ea87-4425-80d2-e76a3354d23d" path="/var/lib/kubelet/pods/fb4bf3de-ea87-4425-80d2-e76a3354d23d/volumes" Oct 30 20:10:07.183889 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:07.183795 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-ms5th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:07.184175 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:07.184136 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" podUID="4045e4a8-b80c-4b45-96b7-e9e4203ebe5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:07.211910 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:07.211862 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lnxpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:07.212148 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:07.211937 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" podUID="f7ff5ff7-5866-473f-9055-e8c74f9380bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:08.225923 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:08.225863 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-ms5th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:08.226461 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:08.225961 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" podUID="4045e4a8-b80c-4b45-96b7-e9e4203ebe5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:08.252829 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:08.252788 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lnxpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:08.253017 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:08.252858 5088 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" podUID="f7ff5ff7-5866-473f-9055-e8c74f9380bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:08.489339 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:08.489303 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" Oct 30 20:10:08.489664 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:08.489643 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" Oct 30 20:10:08.490454 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:08.490434 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" Oct 30 20:10:08.516343 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:08.516308 5088 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" Oct 30 20:10:08.516665 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:08.516637 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" Oct 30 20:10:08.517442 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:08.517423 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" Oct 30 20:10:08.938630 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:08.938530 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-774477d6c-kt5g5"] Oct 30 20:10:08.938823 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:08.938737 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-774477d6c-kt5g5" podUID="326e09b7-8038-4806-9589-e991b674cbad" containerName="manager" containerID="cri-o://b6ce99d6e55a4de61d508984971ed79872ade90540225bc42093df154d9a2e68" gracePeriod=30 Oct 30 20:10:09.190712 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:09.190645 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-774477d6c-kt5g5" Oct 30 20:10:09.292320 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:09.292283 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/326e09b7-8038-4806-9589-e991b674cbad-metrics-certs\" (UniqueName: \"kubernetes.io/secret/326e09b7-8038-4806-9589-e991b674cbad-metrics-certs\") pod \"326e09b7-8038-4806-9589-e991b674cbad\" (UID: \"326e09b7-8038-4806-9589-e991b674cbad\") " Oct 30 20:10:09.292776 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:09.292392 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/326e09b7-8038-4806-9589-e991b674cbad-webhook-cert\" (UniqueName: \"kubernetes.io/secret/326e09b7-8038-4806-9589-e991b674cbad-webhook-cert\") pod \"326e09b7-8038-4806-9589-e991b674cbad\" (UID: \"326e09b7-8038-4806-9589-e991b674cbad\") " Oct 30 20:10:09.292776 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:09.292469 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/326e09b7-8038-4806-9589-e991b674cbad-kube-api-access-kqgft\" (UniqueName: \"kubernetes.io/projected/326e09b7-8038-4806-9589-e991b674cbad-kube-api-access-kqgft\") pod \"326e09b7-8038-4806-9589-e991b674cbad\" (UID: \"326e09b7-8038-4806-9589-e991b674cbad\") " Oct 30 20:10:09.294749 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:09.294710 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/326e09b7-8038-4806-9589-e991b674cbad-webhook-cert" pod "326e09b7-8038-4806-9589-e991b674cbad" (UID: "326e09b7-8038-4806-9589-e991b674cbad"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:10:09.294900 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:09.294846 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/326e09b7-8038-4806-9589-e991b674cbad-kube-api-access-kqgft" pod "326e09b7-8038-4806-9589-e991b674cbad" (UID: "326e09b7-8038-4806-9589-e991b674cbad"). InnerVolumeSpecName "kube-api-access-kqgft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:10:09.295215 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:09.295189 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/326e09b7-8038-4806-9589-e991b674cbad-metrics-certs" pod "326e09b7-8038-4806-9589-e991b674cbad" (UID: "326e09b7-8038-4806-9589-e991b674cbad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:10:09.393712 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:09.393674 5088 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/326e09b7-8038-4806-9589-e991b674cbad-webhook-cert\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:10:09.393712 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:09.393707 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqgft\" (UniqueName: \"kubernetes.io/projected/326e09b7-8038-4806-9589-e991b674cbad-kube-api-access-kqgft\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:10:09.393712 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:09.393718 5088 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/326e09b7-8038-4806-9589-e991b674cbad-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:10:09.934554 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:09.934518 5088 generic.go:356] "Generic (PLEG): container finished" podID="326e09b7-8038-4806-9589-e991b674cbad" containerID="b6ce99d6e55a4de61d508984971ed79872ade90540225bc42093df154d9a2e68" exitCode=0 Oct 30 20:10:09.934763 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:09.934579 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-774477d6c-kt5g5" Oct 30 20:10:09.934763 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:09.934641 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-774477d6c-kt5g5" event={"ID":"326e09b7-8038-4806-9589-e991b674cbad","Type":"ContainerDied","Data":"b6ce99d6e55a4de61d508984971ed79872ade90540225bc42093df154d9a2e68"} Oct 30 20:10:09.934763 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:09.934703 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-774477d6c-kt5g5" event={"ID":"326e09b7-8038-4806-9589-e991b674cbad","Type":"ContainerDied","Data":"2441fb083266bfee27d5bc239781e55ffd68419b7f520747fedb7fec03b76f8f"} Oct 30 20:10:09.934763 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:09.934737 5088 scope.go:122] "RemoveContainer" containerID="b6ce99d6e55a4de61d508984971ed79872ade90540225bc42093df154d9a2e68" Oct 30 20:10:09.945344 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:09.945228 5088 scope.go:122] "RemoveContainer" containerID="b6ce99d6e55a4de61d508984971ed79872ade90540225bc42093df154d9a2e68" Oct 30 20:10:09.945577 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:10:09.945555 5088 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6ce99d6e55a4de61d508984971ed79872ade90540225bc42093df154d9a2e68\": container with ID starting with b6ce99d6e55a4de61d508984971ed79872ade90540225bc42093df154d9a2e68 not found: ID does not exist" containerID="b6ce99d6e55a4de61d508984971ed79872ade90540225bc42093df154d9a2e68" Oct 30 20:10:09.945670 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:09.945587 5088 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6ce99d6e55a4de61d508984971ed79872ade90540225bc42093df154d9a2e68"} err="failed to get container status \"b6ce99d6e55a4de61d508984971ed79872ade90540225bc42093df154d9a2e68\": rpc error: code = NotFound desc = could not find container \"b6ce99d6e55a4de61d508984971ed79872ade90540225bc42093df154d9a2e68\": container with ID starting with b6ce99d6e55a4de61d508984971ed79872ade90540225bc42093df154d9a2e68 not found: ID does not exist" Oct 30 20:10:09.995915 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:09.995880 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-774477d6c-kt5g5"] Oct 30 20:10:10.009282 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:10.009252 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-774477d6c-kt5g5"] Oct 30 20:10:10.739679 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:10.739641 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="326e09b7-8038-4806-9589-e991b674cbad" path="/var/lib/kubelet/pods/326e09b7-8038-4806-9589-e991b674cbad/volumes" Oct 30 20:10:11.268823 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:11.268784 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-774477d6c-wsdl5"] Oct 30 20:10:11.269274 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:11.269259 5088 cpu_manager.go:414] "containerMap: removing container" podUID="326e09b7-8038-4806-9589-e991b674cbad" containerName="manager" Oct 30 20:10:11.269328 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:11.269275 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="326e09b7-8038-4806-9589-e991b674cbad" containerName="manager" Oct 30 20:10:11.269328 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:11.269306 5088 cpu_manager.go:414] "containerMap: removing container" podUID="1be427d5-7034-4669-bc89-4986e9fe3ceb" containerName="revoke-keys" Oct 30 20:10:11.269328 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:11.269312 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1be427d5-7034-4669-bc89-4986e9fe3ceb" containerName="revoke-keys" Oct 30 20:10:11.269421 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:11.269386 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="326e09b7-8038-4806-9589-e991b674cbad" containerName="manager" Oct 30 20:10:11.269421 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:11.269400 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="1be427d5-7034-4669-bc89-4986e9fe3ceb" containerName="revoke-keys" Oct 30 20:10:11.272915 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:11.272898 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-774477d6c-wsdl5" Oct 30 20:10:11.307208 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:11.307175 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wcnv4\"" Oct 30 20:10:11.307395 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:11.307236 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 20:10:11.307395 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:11.307300 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 20:10:11.307395 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:11.307354 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 20:10:11.310375 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:11.310349 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-774477d6c-wsdl5"] Oct 30 20:10:11.314783 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:11.314763 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4132b74d-b894-4c19-9333-201adb0ef1ad-webhook-cert\") pod \"maas-controller-774477d6c-wsdl5\" (UID: \"4132b74d-b894-4c19-9333-201adb0ef1ad\") " pod="opendatahub/maas-controller-774477d6c-wsdl5" Oct 30 20:10:11.314881 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:11.314827 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4132b74d-b894-4c19-9333-201adb0ef1ad-metrics-certs\") pod \"maas-controller-774477d6c-wsdl5\" (UID: \"4132b74d-b894-4c19-9333-201adb0ef1ad\") " pod="opendatahub/maas-controller-774477d6c-wsdl5" Oct 30 20:10:11.314944 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:11.314927 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qmvp\" (UniqueName: \"kubernetes.io/projected/4132b74d-b894-4c19-9333-201adb0ef1ad-kube-api-access-5qmvp\") pod \"maas-controller-774477d6c-wsdl5\" (UID: \"4132b74d-b894-4c19-9333-201adb0ef1ad\") " pod="opendatahub/maas-controller-774477d6c-wsdl5" Oct 30 20:10:11.415582 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:11.415545 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4132b74d-b894-4c19-9333-201adb0ef1ad-webhook-cert\") pod \"maas-controller-774477d6c-wsdl5\" (UID: \"4132b74d-b894-4c19-9333-201adb0ef1ad\") " pod="opendatahub/maas-controller-774477d6c-wsdl5" Oct 30 20:10:11.415768 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:11.415688 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4132b74d-b894-4c19-9333-201adb0ef1ad-metrics-certs\") pod \"maas-controller-774477d6c-wsdl5\" (UID: \"4132b74d-b894-4c19-9333-201adb0ef1ad\") " pod="opendatahub/maas-controller-774477d6c-wsdl5" Oct 30 20:10:11.415768 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:11.415736 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qmvp\" (UniqueName: \"kubernetes.io/projected/4132b74d-b894-4c19-9333-201adb0ef1ad-kube-api-access-5qmvp\") pod \"maas-controller-774477d6c-wsdl5\" (UID: \"4132b74d-b894-4c19-9333-201adb0ef1ad\") " pod="opendatahub/maas-controller-774477d6c-wsdl5" Oct 30 20:10:11.418214 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:11.418187 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4132b74d-b894-4c19-9333-201adb0ef1ad-metrics-certs\") pod \"maas-controller-774477d6c-wsdl5\" (UID: \"4132b74d-b894-4c19-9333-201adb0ef1ad\") " pod="opendatahub/maas-controller-774477d6c-wsdl5" Oct 30 20:10:11.418214 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:11.418207 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4132b74d-b894-4c19-9333-201adb0ef1ad-webhook-cert\") pod \"maas-controller-774477d6c-wsdl5\" (UID: \"4132b74d-b894-4c19-9333-201adb0ef1ad\") " pod="opendatahub/maas-controller-774477d6c-wsdl5" Oct 30 20:10:11.442903 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:11.442868 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qmvp\" (UniqueName: \"kubernetes.io/projected/4132b74d-b894-4c19-9333-201adb0ef1ad-kube-api-access-5qmvp\") pod \"maas-controller-774477d6c-wsdl5\" (UID: \"4132b74d-b894-4c19-9333-201adb0ef1ad\") " pod="opendatahub/maas-controller-774477d6c-wsdl5" Oct 30 20:10:11.587022 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:11.586934 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-774477d6c-wsdl5" Oct 30 20:10:11.893893 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:11.893863 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-774477d6c-wsdl5"] Oct 30 20:10:11.894261 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:10:11.894114 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4132b74d_b894_4c19_9333_201adb0ef1ad.slice/crio-0773212ea86edb8fc9cc0654264b9e16ff5d6d1e36a0ff38b884703cd426f146 WatchSource:0}: Error finding container 0773212ea86edb8fc9cc0654264b9e16ff5d6d1e36a0ff38b884703cd426f146: Status 404 returned error can't find the container with id 0773212ea86edb8fc9cc0654264b9e16ff5d6d1e36a0ff38b884703cd426f146 Oct 30 20:10:11.951329 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:11.951289 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-774477d6c-wsdl5" event={"ID":"4132b74d-b894-4c19-9333-201adb0ef1ad","Type":"ContainerStarted","Data":"0773212ea86edb8fc9cc0654264b9e16ff5d6d1e36a0ff38b884703cd426f146"} Oct 30 20:10:12.959935 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:12.959888 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-774477d6c-wsdl5" event={"ID":"4132b74d-b894-4c19-9333-201adb0ef1ad","Type":"ContainerStarted","Data":"ab44d829b250a58598371bdaf2ead10c4d8e5d1aca7b5ae7aaab537e25a3ce7d"} Oct 30 20:10:12.960379 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:12.960009 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-774477d6c-wsdl5" Oct 30 20:10:13.010409 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:13.010361 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-774477d6c-wsdl5" podStartSLOduration=1.651805633 podStartE2EDuration="2.010346107s" totalImagesPullingTime="358.540474ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:10:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:10:13.010043276 +0000 UTC m=+1374.422757077" watchObservedRunningTime="2026-10-30 20:10:13.010346107 +0000 UTC m=+1374.423059886" Oct 30 20:10:22.438471 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:10:22.438429 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:10:23.968988 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:23.968954 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-774477d6c-wsdl5" Oct 30 20:10:27.034454 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:27.034413 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7db4898-8jr6z"] Oct 30 20:10:27.047908 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:27.047871 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7db4898-8jr6z"] Oct 30 20:10:27.213773 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:27.213741 5088 scope.go:122] "RemoveContainer" containerID="a8b9dc33d1d1d99b8da8acb0302238282ffdfe5573ce59c8ffd52380eedfab24" Oct 30 20:10:27.223844 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:27.223819 5088 scope.go:122] "RemoveContainer" containerID="ec0589382cf960ccf68133b378f1efc8178b0d114b31e0ab49aabf9a1e03f11a" Oct 30 20:10:27.232771 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:27.232747 5088 scope.go:122] "RemoveContainer" containerID="7b3f67c30439751b9f4993f2dcacdba42057ed92629c3944acdce861e40a79eb" Oct 30 20:10:28.738347 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:28.738309 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c1f9b70-c9a5-4f10-9fee-8c75f815378e" path="/var/lib/kubelet/pods/8c1f9b70-c9a5-4f10-9fee-8c75f815378e/volumes" Oct 30 20:10:40.240778 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:40.240694 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:10:41.040790 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:41.040740 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0d7df91-4crlv"] Oct 30 20:10:41.052023 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:41.051978 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0d7df91-4crlv"] Oct 30 20:10:42.763045 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:42.763003 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb63c45f-e1d2-41b8-8723-55f67fa0f5a2" path="/var/lib/kubelet/pods/bb63c45f-e1d2-41b8-8723-55f67fa0f5a2/volumes" Oct 30 20:10:46.077961 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:46.077922 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sjpgp"] Oct 30 20:10:46.082871 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:46.082851 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sjpgp" Oct 30 20:10:46.102572 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:46.102536 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sjpgp"] Oct 30 20:10:46.112194 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:46.112167 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:10:46.112194 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:46.112167 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:10:46.112194 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:46.112181 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:10:46.114545 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:46.114524 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:10:46.174119 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:46.174081 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b887acf-189b-4980-92c2-6deed24c9e29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-sjpgp\" (UID: \"5b887acf-189b-4980-92c2-6deed24c9e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sjpgp" Oct 30 20:10:46.174315 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:46.174215 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b887acf-189b-4980-92c2-6deed24c9e29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-sjpgp\" (UID: \"5b887acf-189b-4980-92c2-6deed24c9e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sjpgp" Oct 30 20:10:46.174315 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:46.174274 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bfmg\" (UniqueName: \"kubernetes.io/projected/5b887acf-189b-4980-92c2-6deed24c9e29-kube-api-access-7bfmg\") pod \"maas-api-revoke-sub-c1e271a9-sjpgp\" (UID: \"5b887acf-189b-4980-92c2-6deed24c9e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sjpgp" Oct 30 20:10:46.275669 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:46.275589 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b887acf-189b-4980-92c2-6deed24c9e29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-sjpgp\" (UID: \"5b887acf-189b-4980-92c2-6deed24c9e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sjpgp" Oct 30 20:10:46.275860 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:46.275684 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bfmg\" (UniqueName: \"kubernetes.io/projected/5b887acf-189b-4980-92c2-6deed24c9e29-kube-api-access-7bfmg\") pod \"maas-api-revoke-sub-c1e271a9-sjpgp\" (UID: \"5b887acf-189b-4980-92c2-6deed24c9e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sjpgp" Oct 30 20:10:46.275860 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:46.275805 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b887acf-189b-4980-92c2-6deed24c9e29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-sjpgp\" (UID: \"5b887acf-189b-4980-92c2-6deed24c9e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sjpgp" Oct 30 20:10:46.276334 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:46.276311 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b887acf-189b-4980-92c2-6deed24c9e29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-sjpgp\" (UID: \"5b887acf-189b-4980-92c2-6deed24c9e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sjpgp" Oct 30 20:10:46.278253 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:46.278233 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b887acf-189b-4980-92c2-6deed24c9e29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-sjpgp\" (UID: \"5b887acf-189b-4980-92c2-6deed24c9e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sjpgp" Oct 30 20:10:46.302417 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:46.302391 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bfmg\" (UniqueName: \"kubernetes.io/projected/5b887acf-189b-4980-92c2-6deed24c9e29-kube-api-access-7bfmg\") pod \"maas-api-revoke-sub-c1e271a9-sjpgp\" (UID: \"5b887acf-189b-4980-92c2-6deed24c9e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sjpgp" Oct 30 20:10:46.396279 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:46.396196 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sjpgp" Oct 30 20:10:46.697981 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:46.697951 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sjpgp"] Oct 30 20:10:46.700346 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:10:46.700318 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b887acf_189b_4980_92c2_6deed24c9e29.slice/crio-12bebf753a9f628a639ef5e89129257b868abe68fa6f8eb45f9b8fb102e0f5bd WatchSource:0}: Error finding container 12bebf753a9f628a639ef5e89129257b868abe68fa6f8eb45f9b8fb102e0f5bd: Status 404 returned error can't find the container with id 12bebf753a9f628a639ef5e89129257b868abe68fa6f8eb45f9b8fb102e0f5bd Oct 30 20:10:47.269446 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:47.269344 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sjpgp" event={"ID":"5b887acf-189b-4980-92c2-6deed24c9e29","Type":"ContainerStarted","Data":"010705871fc21f2262e38b7113638d2c3e0def8585d3e51e88400c932cfa7b38"} Oct 30 20:10:47.269446 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:47.269411 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sjpgp" event={"ID":"5b887acf-189b-4980-92c2-6deed24c9e29","Type":"ContainerStarted","Data":"12bebf753a9f628a639ef5e89129257b868abe68fa6f8eb45f9b8fb102e0f5bd"} Oct 30 20:10:47.334403 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:47.334344 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sjpgp" podStartSLOduration=1.334329575 podStartE2EDuration="1.334329575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:10:47.33339015 +0000 UTC m=+1408.746103943" watchObservedRunningTime="2026-10-30 20:10:47.334329575 +0000 UTC m=+1408.747043351" Oct 30 20:10:48.277964 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:48.277929 5088 generic.go:356] "Generic (PLEG): container finished" podID="5b887acf-189b-4980-92c2-6deed24c9e29" containerID="010705871fc21f2262e38b7113638d2c3e0def8585d3e51e88400c932cfa7b38" exitCode=0 Oct 30 20:10:48.278341 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:48.278013 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sjpgp" event={"ID":"5b887acf-189b-4980-92c2-6deed24c9e29","Type":"ContainerDied","Data":"010705871fc21f2262e38b7113638d2c3e0def8585d3e51e88400c932cfa7b38"} Oct 30 20:10:49.447148 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:49.447122 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sjpgp" Oct 30 20:10:49.511311 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:49.511275 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b887acf-189b-4980-92c2-6deed24c9e29-kube-api-access-7bfmg\" (UniqueName: \"kubernetes.io/projected/5b887acf-189b-4980-92c2-6deed24c9e29-kube-api-access-7bfmg\") pod \"5b887acf-189b-4980-92c2-6deed24c9e29\" (UID: \"5b887acf-189b-4980-92c2-6deed24c9e29\") " Oct 30 20:10:49.511488 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:49.511408 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b887acf-189b-4980-92c2-6deed24c9e29-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b887acf-189b-4980-92c2-6deed24c9e29-maas-api-xks-ca\") pod \"5b887acf-189b-4980-92c2-6deed24c9e29\" (UID: \"5b887acf-189b-4980-92c2-6deed24c9e29\") " Oct 30 20:10:49.511550 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:49.511498 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b887acf-189b-4980-92c2-6deed24c9e29-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b887acf-189b-4980-92c2-6deed24c9e29-maas-api-openshift-ca\") pod \"5b887acf-189b-4980-92c2-6deed24c9e29\" (UID: \"5b887acf-189b-4980-92c2-6deed24c9e29\") " Oct 30 20:10:49.511878 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:49.511846 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b887acf-189b-4980-92c2-6deed24c9e29-maas-api-openshift-ca" pod "5b887acf-189b-4980-92c2-6deed24c9e29" (UID: "5b887acf-189b-4980-92c2-6deed24c9e29"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:10:49.513492 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:49.513468 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b887acf-189b-4980-92c2-6deed24c9e29-kube-api-access-7bfmg" pod "5b887acf-189b-4980-92c2-6deed24c9e29" (UID: "5b887acf-189b-4980-92c2-6deed24c9e29"). InnerVolumeSpecName "kube-api-access-7bfmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:10:49.513596 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:49.513538 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b887acf-189b-4980-92c2-6deed24c9e29-maas-api-xks-ca" pod "5b887acf-189b-4980-92c2-6deed24c9e29" (UID: "5b887acf-189b-4980-92c2-6deed24c9e29"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:10:49.612859 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:49.612772 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bfmg\" (UniqueName: \"kubernetes.io/projected/5b887acf-189b-4980-92c2-6deed24c9e29-kube-api-access-7bfmg\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:10:49.612859 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:49.612801 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b887acf-189b-4980-92c2-6deed24c9e29-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:10:49.612859 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:49.612810 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b887acf-189b-4980-92c2-6deed24c9e29-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:10:50.292042 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:50.292009 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sjpgp" Oct 30 20:10:50.292042 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:50.292030 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sjpgp" event={"ID":"5b887acf-189b-4980-92c2-6deed24c9e29","Type":"ContainerDied","Data":"12bebf753a9f628a639ef5e89129257b868abe68fa6f8eb45f9b8fb102e0f5bd"} Oct 30 20:10:50.292267 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:50.292107 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12bebf753a9f628a639ef5e89129257b868abe68fa6f8eb45f9b8fb102e0f5bd" Oct 30 20:10:52.438802 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:10:52.438761 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:10:56.037570 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:56.037531 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cad5119-c2c5m"] Oct 30 20:10:56.050894 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:56.050864 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cad5119-c2c5m"] Oct 30 20:10:56.742513 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:56.742470 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92c93d36-4f3f-4670-a8f4-c8d2fd372cff" path="/var/lib/kubelet/pods/92c93d36-4f3f-4670-a8f4-c8d2fd372cff/volumes" Oct 30 20:10:57.336381 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:10:57.336339 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:11:01.524408 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:01.524372 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:11:04.333841 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:04.333762 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-c2tb9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:11:04.334547 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:04.334507 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" podUID="ce0a1543-79bc-440d-ad18-c045ac745786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:11:05.035632 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:05.035577 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acbac306-99gbd"] Oct 30 20:11:05.048536 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:05.048503 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acbac306-99gbd"] Oct 30 20:11:06.166074 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:06.166033 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z9pbl"] Oct 30 20:11:06.166524 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:06.166510 5088 cpu_manager.go:414] "containerMap: removing container" podUID="5b887acf-189b-4980-92c2-6deed24c9e29" containerName="revoke-keys" Oct 30 20:11:06.166569 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:06.166525 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b887acf-189b-4980-92c2-6deed24c9e29" containerName="revoke-keys" Oct 30 20:11:06.166605 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:06.166594 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b887acf-189b-4980-92c2-6deed24c9e29" containerName="revoke-keys" Oct 30 20:11:06.169445 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:06.169428 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z9pbl" Oct 30 20:11:06.191447 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:06.191416 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z9pbl"] Oct 30 20:11:06.201183 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:06.201155 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:11:06.201339 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:06.201190 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:11:06.255353 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:06.255322 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:11:06.255543 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:06.255469 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:11:06.338786 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:06.338745 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdrmq\" (UniqueName: \"kubernetes.io/projected/f32c9d53-2197-4ded-8237-305cdd5528a9-kube-api-access-mdrmq\") pod \"maas-api-revoke-sub-2e1f5b5f-z9pbl\" (UID: \"f32c9d53-2197-4ded-8237-305cdd5528a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z9pbl" Oct 30 20:11:06.338984 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:06.338870 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f32c9d53-2197-4ded-8237-305cdd5528a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-z9pbl\" (UID: \"f32c9d53-2197-4ded-8237-305cdd5528a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z9pbl" Oct 30 20:11:06.338984 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:06.338909 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f32c9d53-2197-4ded-8237-305cdd5528a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-z9pbl\" (UID: \"f32c9d53-2197-4ded-8237-305cdd5528a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z9pbl" Oct 30 20:11:06.440090 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:06.439994 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f32c9d53-2197-4ded-8237-305cdd5528a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-z9pbl\" (UID: \"f32c9d53-2197-4ded-8237-305cdd5528a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z9pbl" Oct 30 20:11:06.440090 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:06.440036 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f32c9d53-2197-4ded-8237-305cdd5528a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-z9pbl\" (UID: \"f32c9d53-2197-4ded-8237-305cdd5528a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z9pbl" Oct 30 20:11:06.440330 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:06.440132 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdrmq\" (UniqueName: \"kubernetes.io/projected/f32c9d53-2197-4ded-8237-305cdd5528a9-kube-api-access-mdrmq\") pod \"maas-api-revoke-sub-2e1f5b5f-z9pbl\" (UID: \"f32c9d53-2197-4ded-8237-305cdd5528a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z9pbl" Oct 30 20:11:06.440893 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:06.440863 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f32c9d53-2197-4ded-8237-305cdd5528a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-z9pbl\" (UID: \"f32c9d53-2197-4ded-8237-305cdd5528a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z9pbl" Oct 30 20:11:06.442427 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:06.442404 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f32c9d53-2197-4ded-8237-305cdd5528a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-z9pbl\" (UID: \"f32c9d53-2197-4ded-8237-305cdd5528a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z9pbl" Oct 30 20:11:06.467218 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:06.467169 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdrmq\" (UniqueName: \"kubernetes.io/projected/f32c9d53-2197-4ded-8237-305cdd5528a9-kube-api-access-mdrmq\") pod \"maas-api-revoke-sub-2e1f5b5f-z9pbl\" (UID: \"f32c9d53-2197-4ded-8237-305cdd5528a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z9pbl" Oct 30 20:11:06.482650 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:06.482587 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z9pbl" Oct 30 20:11:06.738725 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:06.738692 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5984058e-2af6-4a63-9c6d-00827d986a40" path="/var/lib/kubelet/pods/5984058e-2af6-4a63-9c6d-00827d986a40/volumes" Oct 30 20:11:06.798124 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:06.798094 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z9pbl"] Oct 30 20:11:06.798572 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:11:06.798544 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf32c9d53_2197_4ded_8237_305cdd5528a9.slice/crio-fb94cf8d6bb8f88a642914ad75172fd8d1434951d5338101a632a1867197bb3e WatchSource:0}: Error finding container fb94cf8d6bb8f88a642914ad75172fd8d1434951d5338101a632a1867197bb3e: Status 404 returned error can't find the container with id fb94cf8d6bb8f88a642914ad75172fd8d1434951d5338101a632a1867197bb3e Oct 30 20:11:07.033411 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:07.033378 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd332a36-hxcw2"] Oct 30 20:11:07.037543 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:07.037519 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd332a36-hxcw2"] Oct 30 20:11:07.468755 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:07.468657 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z9pbl" event={"ID":"f32c9d53-2197-4ded-8237-305cdd5528a9","Type":"ContainerStarted","Data":"aef673440fb9c4fc235657af681a500ce1247a4ba055dff59744064f1a18950c"} Oct 30 20:11:07.468755 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:07.468719 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z9pbl" event={"ID":"f32c9d53-2197-4ded-8237-305cdd5528a9","Type":"ContainerStarted","Data":"fb94cf8d6bb8f88a642914ad75172fd8d1434951d5338101a632a1867197bb3e"} Oct 30 20:11:07.514882 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:07.514821 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z9pbl" podStartSLOduration=1.5148069240000002 podStartE2EDuration="1.514806924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:11:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:11:07.512998369 +0000 UTC m=+1428.925712148" watchObservedRunningTime="2026-10-30 20:11:07.514806924 +0000 UTC m=+1428.927520914" Oct 30 20:11:08.477047 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:08.477003 5088 generic.go:356] "Generic (PLEG): container finished" podID="f32c9d53-2197-4ded-8237-305cdd5528a9" containerID="aef673440fb9c4fc235657af681a500ce1247a4ba055dff59744064f1a18950c" exitCode=0 Oct 30 20:11:08.477503 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:08.477082 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z9pbl" event={"ID":"f32c9d53-2197-4ded-8237-305cdd5528a9","Type":"ContainerDied","Data":"aef673440fb9c4fc235657af681a500ce1247a4ba055dff59744064f1a18950c"} Oct 30 20:11:08.738496 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:08.738464 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f4eba44-f3cb-4dd7-8557-0310f51ddf26" path="/var/lib/kubelet/pods/5f4eba44-f3cb-4dd7-8557-0310f51ddf26/volumes" Oct 30 20:11:09.637790 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:09.637767 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z9pbl" Oct 30 20:11:09.775040 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:09.774949 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f32c9d53-2197-4ded-8237-305cdd5528a9-kube-api-access-mdrmq\" (UniqueName: \"kubernetes.io/projected/f32c9d53-2197-4ded-8237-305cdd5528a9-kube-api-access-mdrmq\") pod \"f32c9d53-2197-4ded-8237-305cdd5528a9\" (UID: \"f32c9d53-2197-4ded-8237-305cdd5528a9\") " Oct 30 20:11:09.775040 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:09.774986 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f32c9d53-2197-4ded-8237-305cdd5528a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f32c9d53-2197-4ded-8237-305cdd5528a9-maas-api-openshift-ca\") pod \"f32c9d53-2197-4ded-8237-305cdd5528a9\" (UID: \"f32c9d53-2197-4ded-8237-305cdd5528a9\") " Oct 30 20:11:09.775040 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:09.775007 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f32c9d53-2197-4ded-8237-305cdd5528a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f32c9d53-2197-4ded-8237-305cdd5528a9-maas-api-xks-ca\") pod \"f32c9d53-2197-4ded-8237-305cdd5528a9\" (UID: \"f32c9d53-2197-4ded-8237-305cdd5528a9\") " Oct 30 20:11:09.775388 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:09.775365 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f32c9d53-2197-4ded-8237-305cdd5528a9-maas-api-openshift-ca" pod "f32c9d53-2197-4ded-8237-305cdd5528a9" (UID: "f32c9d53-2197-4ded-8237-305cdd5528a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:11:09.777148 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:09.777119 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f32c9d53-2197-4ded-8237-305cdd5528a9-maas-api-xks-ca" pod "f32c9d53-2197-4ded-8237-305cdd5528a9" (UID: "f32c9d53-2197-4ded-8237-305cdd5528a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:11:09.777148 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:09.777139 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f32c9d53-2197-4ded-8237-305cdd5528a9-kube-api-access-mdrmq" pod "f32c9d53-2197-4ded-8237-305cdd5528a9" (UID: "f32c9d53-2197-4ded-8237-305cdd5528a9"). InnerVolumeSpecName "kube-api-access-mdrmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:11:09.876790 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:09.876737 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdrmq\" (UniqueName: \"kubernetes.io/projected/f32c9d53-2197-4ded-8237-305cdd5528a9-kube-api-access-mdrmq\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:11:09.876790 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:09.876789 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f32c9d53-2197-4ded-8237-305cdd5528a9-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:11:09.877090 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:09.876808 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f32c9d53-2197-4ded-8237-305cdd5528a9-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:11:10.492320 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:10.492281 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z9pbl" event={"ID":"f32c9d53-2197-4ded-8237-305cdd5528a9","Type":"ContainerDied","Data":"fb94cf8d6bb8f88a642914ad75172fd8d1434951d5338101a632a1867197bb3e"} Oct 30 20:11:10.492497 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:10.492337 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb94cf8d6bb8f88a642914ad75172fd8d1434951d5338101a632a1867197bb3e" Oct 30 20:11:10.492497 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:10.492294 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z9pbl" Oct 30 20:11:17.438790 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:17.438755 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:11:17.487482 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:17.487445 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rqscv"] Oct 30 20:11:17.487963 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:17.487945 5088 cpu_manager.go:414] "containerMap: removing container" podUID="f32c9d53-2197-4ded-8237-305cdd5528a9" containerName="revoke-keys" Oct 30 20:11:17.487963 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:17.487961 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f32c9d53-2197-4ded-8237-305cdd5528a9" containerName="revoke-keys" Oct 30 20:11:17.488123 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:17.488042 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="f32c9d53-2197-4ded-8237-305cdd5528a9" containerName="revoke-keys" Oct 30 20:11:17.491506 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:17.491490 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rqscv" Oct 30 20:11:17.509894 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:17.509671 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:11:17.509894 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:17.509666 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:11:17.512598 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:17.512571 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rqscv"] Oct 30 20:11:17.521844 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:17.521816 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:11:17.521957 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:17.521912 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:11:17.553805 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:17.553768 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-rqscv\" (UID: \"d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rqscv" Oct 30 20:11:17.553953 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:17.553835 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxf49\" (UniqueName: \"kubernetes.io/projected/d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d-kube-api-access-mxf49\") pod \"maas-api-revoke-sub-ec843018-rqscv\" (UID: \"d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rqscv" Oct 30 20:11:17.553953 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:17.553905 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-rqscv\" (UID: \"d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rqscv" Oct 30 20:11:17.655127 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:17.655087 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-rqscv\" (UID: \"d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rqscv" Oct 30 20:11:17.655324 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:17.655143 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxf49\" (UniqueName: \"kubernetes.io/projected/d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d-kube-api-access-mxf49\") pod \"maas-api-revoke-sub-ec843018-rqscv\" (UID: \"d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rqscv" Oct 30 20:11:17.655324 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:17.655188 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-rqscv\" (UID: \"d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rqscv" Oct 30 20:11:17.656138 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:17.656108 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-rqscv\" (UID: \"d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rqscv" Oct 30 20:11:17.657649 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:17.657631 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-rqscv\" (UID: \"d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rqscv" Oct 30 20:11:17.681947 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:17.681916 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxf49\" (UniqueName: \"kubernetes.io/projected/d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d-kube-api-access-mxf49\") pod \"maas-api-revoke-sub-ec843018-rqscv\" (UID: \"d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rqscv" Oct 30 20:11:17.804765 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:17.804731 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rqscv" Oct 30 20:11:18.129668 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:18.129638 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rqscv"] Oct 30 20:11:18.130810 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:11:18.130783 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd48ebadd_aacd_4e1d_9bc1_1ad174b5f09d.slice/crio-ed998140ff31d56f0edcbac4ba50835e43fced1c7f8b1595741b7a2c08a1d282 WatchSource:0}: Error finding container ed998140ff31d56f0edcbac4ba50835e43fced1c7f8b1595741b7a2c08a1d282: Status 404 returned error can't find the container with id ed998140ff31d56f0edcbac4ba50835e43fced1c7f8b1595741b7a2c08a1d282 Oct 30 20:11:18.547354 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:18.547316 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rqscv" event={"ID":"d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d","Type":"ContainerStarted","Data":"613230f6ff2137f88200daff11ec519423d55d9a968dabbcdfbc8dd763853cfb"} Oct 30 20:11:18.547841 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:18.547372 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rqscv" event={"ID":"d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d","Type":"ContainerStarted","Data":"ed998140ff31d56f0edcbac4ba50835e43fced1c7f8b1595741b7a2c08a1d282"} Oct 30 20:11:18.592884 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:18.592812 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rqscv" podStartSLOduration=1.592793204 podStartE2EDuration="1.592793204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:11:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:11:18.592666552 +0000 UTC m=+1440.005380354" watchObservedRunningTime="2026-10-30 20:11:18.592793204 +0000 UTC m=+1440.005506985" Oct 30 20:11:19.555594 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:19.555556 5088 generic.go:356] "Generic (PLEG): container finished" podID="d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d" containerID="613230f6ff2137f88200daff11ec519423d55d9a968dabbcdfbc8dd763853cfb" exitCode=0 Oct 30 20:11:19.556027 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:19.555643 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rqscv" event={"ID":"d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d","Type":"ContainerDied","Data":"613230f6ff2137f88200daff11ec519423d55d9a968dabbcdfbc8dd763853cfb"} Oct 30 20:11:20.707797 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:20.707771 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rqscv" Oct 30 20:11:20.793272 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:20.793239 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d-maas-api-openshift-ca\") pod \"d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d\" (UID: \"d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d\") " Oct 30 20:11:20.793483 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:20.793303 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d-maas-api-xks-ca\") pod \"d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d\" (UID: \"d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d\") " Oct 30 20:11:20.793483 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:20.793350 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d-kube-api-access-mxf49\" (UniqueName: \"kubernetes.io/projected/d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d-kube-api-access-mxf49\") pod \"d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d\" (UID: \"d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d\") " Oct 30 20:11:20.793737 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:20.793706 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d-maas-api-openshift-ca" pod "d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d" (UID: "d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:11:20.793988 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:20.793952 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:11:20.795436 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:20.795414 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d-maas-api-xks-ca" pod "d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d" (UID: "d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:11:20.795556 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:20.795537 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d-kube-api-access-mxf49" pod "d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d" (UID: "d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d"). InnerVolumeSpecName "kube-api-access-mxf49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:11:20.895029 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:20.894945 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:11:20.895029 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:20.894982 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxf49\" (UniqueName: \"kubernetes.io/projected/d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d-kube-api-access-mxf49\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:11:21.569662 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:21.569603 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rqscv" Oct 30 20:11:21.569662 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:21.569647 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rqscv" event={"ID":"d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d","Type":"ContainerDied","Data":"ed998140ff31d56f0edcbac4ba50835e43fced1c7f8b1595741b7a2c08a1d282"} Oct 30 20:11:21.569887 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:21.569712 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed998140ff31d56f0edcbac4ba50835e43fced1c7f8b1595741b7a2c08a1d282" Oct 30 20:11:22.438121 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:11:22.438080 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:11:27.294520 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:27.294486 5088 scope.go:122] "RemoveContainer" containerID="e2b0d690456a79235aab26cb2f341ef4af7ae03bad18f9d8222163d3b012ead0" Oct 30 20:11:27.304905 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:27.304879 5088 scope.go:122] "RemoveContainer" containerID="c3fb1272968557cba700fced30ca4bd585345d53ef20b3c596965d955230c051" Oct 30 20:11:27.313791 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:27.313769 5088 scope.go:122] "RemoveContainer" containerID="2874047ba764f4224752e90c08cf1915db184a73416b93b8fee618511f18df69" Oct 30 20:11:27.322172 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:27.322148 5088 scope.go:122] "RemoveContainer" containerID="6110fd54edcf054e9561252d08db05b212c4b3292529b849022ef776b6dc56f4" Oct 30 20:11:27.330501 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:27.330480 5088 scope.go:122] "RemoveContainer" containerID="0b117f14dd7c3c281855b300156ecb9442c3c45994511bf02127cb8204f079f0" Oct 30 20:11:27.339013 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:27.338995 5088 scope.go:122] "RemoveContainer" containerID="d5a49f9350f40f63b0f7f4562bc4374c91f2d77dccb5ded6d804cb5a8591faf3" Oct 30 20:11:27.346920 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:27.346897 5088 scope.go:122] "RemoveContainer" containerID="a075bc844ce526b73ea161dbb634c0376055def669926fe19de10d1fae182038" Oct 30 20:11:27.354770 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:27.354751 5088 scope.go:122] "RemoveContainer" containerID="51662691140ba7074b7a7ec3d5cc198d59ee5eef0cae8c18126b5643bce0c917" Oct 30 20:11:27.362857 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:27.362841 5088 scope.go:122] "RemoveContainer" containerID="7c0f447c7ea99fa8edd0712214712a55332134e468229d48911850e098aa43db" Oct 30 20:11:30.044201 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:30.044158 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-470dd631-q8tgv"] Oct 30 20:11:30.056028 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:30.056000 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-470dd631-q8tgv"] Oct 30 20:11:30.738369 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:30.738335 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97f8f58f-a1ca-4489-9b23-d065cf67d9ef" path="/var/lib/kubelet/pods/97f8f58f-a1ca-4489-9b23-d065cf67d9ef/volumes" Oct 30 20:11:32.034784 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:32.034744 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea50d2fa-jxdxf"] Oct 30 20:11:32.048297 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:32.048263 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea50d2fa-jxdxf"] Oct 30 20:11:32.738108 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:32.738073 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67c834be-8e1e-4f8a-a0d4-565fb0f806a6" path="/var/lib/kubelet/pods/67c834be-8e1e-4f8a-a0d4-565fb0f806a6/volumes" Oct 30 20:11:33.062559 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:33.062478 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:11:45.738598 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:45.738557 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6z5jn"] Oct 30 20:11:45.739039 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:45.739031 5088 cpu_manager.go:414] "containerMap: removing container" podUID="d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d" containerName="revoke-keys" Oct 30 20:11:45.739083 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:45.739043 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d" containerName="revoke-keys" Oct 30 20:11:45.739145 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:45.739134 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d" containerName="revoke-keys" Oct 30 20:11:45.742548 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:45.742529 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6z5jn" Oct 30 20:11:45.763181 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:45.763145 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6z5jn"] Oct 30 20:11:45.772705 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:45.772650 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:11:45.772877 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:45.772716 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:11:45.772877 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:45.772723 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:11:45.772877 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:45.772757 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:11:45.892377 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:45.892337 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stx58\" (UniqueName: \"kubernetes.io/projected/f01cb5ca-b494-4248-a645-4b0f0edccae9-kube-api-access-stx58\") pod \"maas-api-revoke-sub-046e1343-6z5jn\" (UID: \"f01cb5ca-b494-4248-a645-4b0f0edccae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6z5jn" Oct 30 20:11:45.892564 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:45.892455 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f01cb5ca-b494-4248-a645-4b0f0edccae9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-6z5jn\" (UID: \"f01cb5ca-b494-4248-a645-4b0f0edccae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6z5jn" Oct 30 20:11:45.892564 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:45.892501 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f01cb5ca-b494-4248-a645-4b0f0edccae9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-6z5jn\" (UID: \"f01cb5ca-b494-4248-a645-4b0f0edccae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6z5jn" Oct 30 20:11:45.993167 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:45.993068 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stx58\" (UniqueName: \"kubernetes.io/projected/f01cb5ca-b494-4248-a645-4b0f0edccae9-kube-api-access-stx58\") pod \"maas-api-revoke-sub-046e1343-6z5jn\" (UID: \"f01cb5ca-b494-4248-a645-4b0f0edccae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6z5jn" Oct 30 20:11:45.993359 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:45.993167 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f01cb5ca-b494-4248-a645-4b0f0edccae9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-6z5jn\" (UID: \"f01cb5ca-b494-4248-a645-4b0f0edccae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6z5jn" Oct 30 20:11:45.993359 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:45.993213 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f01cb5ca-b494-4248-a645-4b0f0edccae9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-6z5jn\" (UID: \"f01cb5ca-b494-4248-a645-4b0f0edccae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6z5jn" Oct 30 20:11:45.993957 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:45.993923 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f01cb5ca-b494-4248-a645-4b0f0edccae9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-6z5jn\" (UID: \"f01cb5ca-b494-4248-a645-4b0f0edccae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6z5jn" Oct 30 20:11:45.995554 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:45.995528 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f01cb5ca-b494-4248-a645-4b0f0edccae9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-6z5jn\" (UID: \"f01cb5ca-b494-4248-a645-4b0f0edccae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6z5jn" Oct 30 20:11:46.019474 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:46.019442 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stx58\" (UniqueName: \"kubernetes.io/projected/f01cb5ca-b494-4248-a645-4b0f0edccae9-kube-api-access-stx58\") pod \"maas-api-revoke-sub-046e1343-6z5jn\" (UID: \"f01cb5ca-b494-4248-a645-4b0f0edccae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6z5jn" Oct 30 20:11:46.035726 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:46.035694 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-308588d2-nl64j"] Oct 30 20:11:46.039790 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:46.039761 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-308588d2-nl64j"] Oct 30 20:11:46.056221 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:46.056189 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6z5jn" Oct 30 20:11:46.361977 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:46.361946 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6z5jn"] Oct 30 20:11:46.363359 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:11:46.363333 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf01cb5ca_b494_4248_a645_4b0f0edccae9.slice/crio-655ed7c579f518e837f4d5f49bef88e644c9d9f1b4d9003428d7da36cdfa7d35 WatchSource:0}: Error finding container 655ed7c579f518e837f4d5f49bef88e644c9d9f1b4d9003428d7da36cdfa7d35: Status 404 returned error can't find the container with id 655ed7c579f518e837f4d5f49bef88e644c9d9f1b4d9003428d7da36cdfa7d35 Oct 30 20:11:46.738932 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:46.738900 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9d900df-2b7f-4e2c-9de7-76d587992dbc" path="/var/lib/kubelet/pods/f9d900df-2b7f-4e2c-9de7-76d587992dbc/volumes" Oct 30 20:11:46.776917 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:46.776881 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6z5jn" event={"ID":"f01cb5ca-b494-4248-a645-4b0f0edccae9","Type":"ContainerStarted","Data":"7f2e875dab008bb0c67acd8f27918547aa6dbd49275345ff73cb7c5969a6ff97"} Oct 30 20:11:46.777116 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:46.776940 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6z5jn" event={"ID":"f01cb5ca-b494-4248-a645-4b0f0edccae9","Type":"ContainerStarted","Data":"655ed7c579f518e837f4d5f49bef88e644c9d9f1b4d9003428d7da36cdfa7d35"} Oct 30 20:11:46.825408 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:46.825344 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6z5jn" podStartSLOduration=1.825329173 podStartE2EDuration="1.825329173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:11:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:11:46.822897516 +0000 UTC m=+1468.235611324" watchObservedRunningTime="2026-10-30 20:11:46.825329173 +0000 UTC m=+1468.238042952" Oct 30 20:11:47.784181 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:47.784134 5088 generic.go:356] "Generic (PLEG): container finished" podID="f01cb5ca-b494-4248-a645-4b0f0edccae9" containerID="7f2e875dab008bb0c67acd8f27918547aa6dbd49275345ff73cb7c5969a6ff97" exitCode=0 Oct 30 20:11:47.784689 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:47.784227 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6z5jn" event={"ID":"f01cb5ca-b494-4248-a645-4b0f0edccae9","Type":"ContainerDied","Data":"7f2e875dab008bb0c67acd8f27918547aa6dbd49275345ff73cb7c5969a6ff97"} Oct 30 20:11:48.938565 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:48.938538 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6z5jn" Oct 30 20:11:49.023511 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:49.023475 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f01cb5ca-b494-4248-a645-4b0f0edccae9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f01cb5ca-b494-4248-a645-4b0f0edccae9-maas-api-openshift-ca\") pod \"f01cb5ca-b494-4248-a645-4b0f0edccae9\" (UID: \"f01cb5ca-b494-4248-a645-4b0f0edccae9\") " Oct 30 20:11:49.023751 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:49.023581 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f01cb5ca-b494-4248-a645-4b0f0edccae9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f01cb5ca-b494-4248-a645-4b0f0edccae9-maas-api-xks-ca\") pod \"f01cb5ca-b494-4248-a645-4b0f0edccae9\" (UID: \"f01cb5ca-b494-4248-a645-4b0f0edccae9\") " Oct 30 20:11:49.023751 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:49.023664 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f01cb5ca-b494-4248-a645-4b0f0edccae9-kube-api-access-stx58\" (UniqueName: \"kubernetes.io/projected/f01cb5ca-b494-4248-a645-4b0f0edccae9-kube-api-access-stx58\") pod \"f01cb5ca-b494-4248-a645-4b0f0edccae9\" (UID: \"f01cb5ca-b494-4248-a645-4b0f0edccae9\") " Oct 30 20:11:49.023934 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:49.023903 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f01cb5ca-b494-4248-a645-4b0f0edccae9-maas-api-openshift-ca" pod "f01cb5ca-b494-4248-a645-4b0f0edccae9" (UID: "f01cb5ca-b494-4248-a645-4b0f0edccae9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:11:49.024080 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:49.024045 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f01cb5ca-b494-4248-a645-4b0f0edccae9-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:11:49.025795 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:49.025771 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f01cb5ca-b494-4248-a645-4b0f0edccae9-maas-api-xks-ca" pod "f01cb5ca-b494-4248-a645-4b0f0edccae9" (UID: "f01cb5ca-b494-4248-a645-4b0f0edccae9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:11:49.025937 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:49.025916 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f01cb5ca-b494-4248-a645-4b0f0edccae9-kube-api-access-stx58" pod "f01cb5ca-b494-4248-a645-4b0f0edccae9" (UID: "f01cb5ca-b494-4248-a645-4b0f0edccae9"). InnerVolumeSpecName "kube-api-access-stx58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:11:49.125762 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:49.125664 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f01cb5ca-b494-4248-a645-4b0f0edccae9-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:11:49.125762 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:49.125706 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stx58\" (UniqueName: \"kubernetes.io/projected/f01cb5ca-b494-4248-a645-4b0f0edccae9-kube-api-access-stx58\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:11:49.798015 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:49.797982 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6z5jn" Oct 30 20:11:49.798235 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:49.797985 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6z5jn" event={"ID":"f01cb5ca-b494-4248-a645-4b0f0edccae9","Type":"ContainerDied","Data":"655ed7c579f518e837f4d5f49bef88e644c9d9f1b4d9003428d7da36cdfa7d35"} Oct 30 20:11:49.798235 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:49.798105 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="655ed7c579f518e837f4d5f49bef88e644c9d9f1b4d9003428d7da36cdfa7d35" Oct 30 20:11:52.437096 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:11:52.437055 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:11:56.037057 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:56.037014 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0a61055-88j9x"] Oct 30 20:11:56.050166 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:56.050134 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0a61055-88j9x"] Oct 30 20:11:56.096701 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:56.096666 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-j4mt8"] Oct 30 20:11:56.097189 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:56.097176 5088 cpu_manager.go:414] "containerMap: removing container" podUID="f01cb5ca-b494-4248-a645-4b0f0edccae9" containerName="revoke-keys" Oct 30 20:11:56.097237 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:56.097190 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f01cb5ca-b494-4248-a645-4b0f0edccae9" containerName="revoke-keys" Oct 30 20:11:56.097307 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:56.097297 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="f01cb5ca-b494-4248-a645-4b0f0edccae9" containerName="revoke-keys" Oct 30 20:11:56.100997 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:56.100977 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-j4mt8" Oct 30 20:11:56.118673 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:56.118642 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:11:56.118900 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:56.118878 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:11:56.118998 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:56.118962 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:11:56.119072 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:56.119002 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:11:56.121210 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:56.121189 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-j4mt8"] Oct 30 20:11:56.199429 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:56.199384 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a9f523-bdde-48a1-9b10-4771a344df4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-j4mt8\" (UID: \"f6a9f523-bdde-48a1-9b10-4771a344df4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-j4mt8" Oct 30 20:11:56.199604 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:56.199492 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a9f523-bdde-48a1-9b10-4771a344df4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-j4mt8\" (UID: \"f6a9f523-bdde-48a1-9b10-4771a344df4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-j4mt8" Oct 30 20:11:56.199604 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:56.199545 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z7r8\" (UniqueName: \"kubernetes.io/projected/f6a9f523-bdde-48a1-9b10-4771a344df4a-kube-api-access-4z7r8\") pod \"maas-api-revoke-sub-d3fba056-j4mt8\" (UID: \"f6a9f523-bdde-48a1-9b10-4771a344df4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-j4mt8" Oct 30 20:11:56.300810 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:56.300722 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a9f523-bdde-48a1-9b10-4771a344df4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-j4mt8\" (UID: \"f6a9f523-bdde-48a1-9b10-4771a344df4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-j4mt8" Oct 30 20:11:56.300810 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:56.300795 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a9f523-bdde-48a1-9b10-4771a344df4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-j4mt8\" (UID: \"f6a9f523-bdde-48a1-9b10-4771a344df4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-j4mt8" Oct 30 20:11:56.301006 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:56.300838 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z7r8\" (UniqueName: \"kubernetes.io/projected/f6a9f523-bdde-48a1-9b10-4771a344df4a-kube-api-access-4z7r8\") pod \"maas-api-revoke-sub-d3fba056-j4mt8\" (UID: \"f6a9f523-bdde-48a1-9b10-4771a344df4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-j4mt8" Oct 30 20:11:56.301443 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:56.301416 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a9f523-bdde-48a1-9b10-4771a344df4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-j4mt8\" (UID: \"f6a9f523-bdde-48a1-9b10-4771a344df4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-j4mt8" Oct 30 20:11:56.303213 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:56.303187 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a9f523-bdde-48a1-9b10-4771a344df4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-j4mt8\" (UID: \"f6a9f523-bdde-48a1-9b10-4771a344df4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-j4mt8" Oct 30 20:11:56.327010 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:56.326982 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z7r8\" (UniqueName: \"kubernetes.io/projected/f6a9f523-bdde-48a1-9b10-4771a344df4a-kube-api-access-4z7r8\") pod \"maas-api-revoke-sub-d3fba056-j4mt8\" (UID: \"f6a9f523-bdde-48a1-9b10-4771a344df4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-j4mt8" Oct 30 20:11:56.414888 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:56.414854 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-j4mt8" Oct 30 20:11:56.721932 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:56.721899 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-j4mt8"] Oct 30 20:11:56.723399 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:11:56.723372 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6a9f523_bdde_48a1_9b10_4771a344df4a.slice/crio-7df422c2e78d9702cdb49cdac4f84a3c3754ad35fa91626f8b95d47afe26f20b WatchSource:0}: Error finding container 7df422c2e78d9702cdb49cdac4f84a3c3754ad35fa91626f8b95d47afe26f20b: Status 404 returned error can't find the container with id 7df422c2e78d9702cdb49cdac4f84a3c3754ad35fa91626f8b95d47afe26f20b Oct 30 20:11:56.739720 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:56.739695 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec80a8b1-509a-4c56-a03c-f5a2a5e3f720" path="/var/lib/kubelet/pods/ec80a8b1-509a-4c56-a03c-f5a2a5e3f720/volumes" Oct 30 20:11:56.844274 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:56.844224 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-j4mt8" event={"ID":"f6a9f523-bdde-48a1-9b10-4771a344df4a","Type":"ContainerStarted","Data":"7df422c2e78d9702cdb49cdac4f84a3c3754ad35fa91626f8b95d47afe26f20b"} Oct 30 20:11:56.876073 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:56.876023 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:11:57.046317 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:57.046282 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fb6fdb9-6wq2f"] Oct 30 20:11:57.050488 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:57.050461 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fb6fdb9-6wq2f"] Oct 30 20:11:57.856453 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:57.856322 5088 generic.go:356] "Generic (PLEG): container finished" podID="f6a9f523-bdde-48a1-9b10-4771a344df4a" containerID="fce48059b335127c85c3e0dcfd88a2cdf5dfdcd0b500ebb4b4f31f4466a8a185" exitCode=0 Oct 30 20:11:57.856453 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:57.856417 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-j4mt8" event={"ID":"f6a9f523-bdde-48a1-9b10-4771a344df4a","Type":"ContainerDied","Data":"fce48059b335127c85c3e0dcfd88a2cdf5dfdcd0b500ebb4b4f31f4466a8a185"} Oct 30 20:11:58.770444 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:58.769970 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f63941f-399d-48c0-8203-0bba97e49e8a" path="/var/lib/kubelet/pods/8f63941f-399d-48c0-8203-0bba97e49e8a/volumes" Oct 30 20:11:59.821809 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:59.821772 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-j4mt8" Oct 30 20:11:59.866287 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:59.866168 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6a9f523-bdde-48a1-9b10-4771a344df4a-kube-api-access-4z7r8\" (UniqueName: \"kubernetes.io/projected/f6a9f523-bdde-48a1-9b10-4771a344df4a-kube-api-access-4z7r8\") pod \"f6a9f523-bdde-48a1-9b10-4771a344df4a\" (UID: \"f6a9f523-bdde-48a1-9b10-4771a344df4a\") " Oct 30 20:11:59.866634 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:59.866591 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6a9f523-bdde-48a1-9b10-4771a344df4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a9f523-bdde-48a1-9b10-4771a344df4a-maas-api-xks-ca\") pod \"f6a9f523-bdde-48a1-9b10-4771a344df4a\" (UID: \"f6a9f523-bdde-48a1-9b10-4771a344df4a\") " Oct 30 20:11:59.866957 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:59.866936 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6a9f523-bdde-48a1-9b10-4771a344df4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a9f523-bdde-48a1-9b10-4771a344df4a-maas-api-openshift-ca\") pod \"f6a9f523-bdde-48a1-9b10-4771a344df4a\" (UID: \"f6a9f523-bdde-48a1-9b10-4771a344df4a\") " Oct 30 20:11:59.868010 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:59.867980 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6a9f523-bdde-48a1-9b10-4771a344df4a-maas-api-openshift-ca" pod "f6a9f523-bdde-48a1-9b10-4771a344df4a" (UID: "f6a9f523-bdde-48a1-9b10-4771a344df4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:11:59.906797 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:59.906751 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6a9f523-bdde-48a1-9b10-4771a344df4a-maas-api-xks-ca" pod "f6a9f523-bdde-48a1-9b10-4771a344df4a" (UID: "f6a9f523-bdde-48a1-9b10-4771a344df4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:11:59.913797 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:59.913757 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6a9f523-bdde-48a1-9b10-4771a344df4a-kube-api-access-4z7r8" pod "f6a9f523-bdde-48a1-9b10-4771a344df4a" (UID: "f6a9f523-bdde-48a1-9b10-4771a344df4a"). InnerVolumeSpecName "kube-api-access-4z7r8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:11:59.956997 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:59.956951 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-j4mt8" event={"ID":"f6a9f523-bdde-48a1-9b10-4771a344df4a","Type":"ContainerDied","Data":"7df422c2e78d9702cdb49cdac4f84a3c3754ad35fa91626f8b95d47afe26f20b"} Oct 30 20:11:59.957362 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:59.957337 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7df422c2e78d9702cdb49cdac4f84a3c3754ad35fa91626f8b95d47afe26f20b" Oct 30 20:11:59.957572 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:59.957556 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-j4mt8" Oct 30 20:11:59.969350 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:59.969316 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a9f523-bdde-48a1-9b10-4771a344df4a-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:11:59.971276 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:59.971255 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z7r8\" (UniqueName: \"kubernetes.io/projected/f6a9f523-bdde-48a1-9b10-4771a344df4a-kube-api-access-4z7r8\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:11:59.971435 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:11:59.971420 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a9f523-bdde-48a1-9b10-4771a344df4a-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:12:07.249471 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:07.249431 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:12:08.034208 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:08.034178 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77119bdd-lg99n"] Oct 30 20:12:08.047756 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:08.047724 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77119bdd-lg99n"] Oct 30 20:12:08.739366 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:08.739334 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cec7595d-0333-45e9-9145-1a5ce41648b8" path="/var/lib/kubelet/pods/cec7595d-0333-45e9-9145-1a5ce41648b8/volumes" Oct 30 20:12:11.555907 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:11.555868 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:12:16.039397 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:16.039359 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1de1a9bb-lqpq5"] Oct 30 20:12:16.052539 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:16.052511 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1de1a9bb-lqpq5"] Oct 30 20:12:16.738156 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:16.738126 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6404ea32-98aa-4ca9-bf40-842e68f949cb" path="/var/lib/kubelet/pods/6404ea32-98aa-4ca9-bf40-842e68f949cb/volumes" Oct 30 20:12:17.742872 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:17.742830 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6fr7c"] Oct 30 20:12:17.743523 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:17.743501 5088 cpu_manager.go:414] "containerMap: removing container" podUID="f6a9f523-bdde-48a1-9b10-4771a344df4a" containerName="revoke-keys" Oct 30 20:12:17.743639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:17.743524 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6a9f523-bdde-48a1-9b10-4771a344df4a" containerName="revoke-keys" Oct 30 20:12:17.743708 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:17.743648 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6a9f523-bdde-48a1-9b10-4771a344df4a" containerName="revoke-keys" Oct 30 20:12:17.747249 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:17.747224 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6fr7c" Oct 30 20:12:17.767546 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:17.767509 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6fr7c"] Oct 30 20:12:17.777624 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:17.777584 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:12:17.777746 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:17.777667 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:12:17.777794 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:17.777755 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:12:17.778212 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:17.778198 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:12:17.779663 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:17.779644 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62ceedbd-7e7e-4e87-99f0-88e6485f65f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-6fr7c\" (UID: \"62ceedbd-7e7e-4e87-99f0-88e6485f65f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6fr7c" Oct 30 20:12:17.779744 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:17.779729 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddblf\" (UniqueName: \"kubernetes.io/projected/62ceedbd-7e7e-4e87-99f0-88e6485f65f2-kube-api-access-ddblf\") pod \"maas-api-revoke-sub-ae594979-6fr7c\" (UID: \"62ceedbd-7e7e-4e87-99f0-88e6485f65f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6fr7c" Oct 30 20:12:17.779821 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:17.779805 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62ceedbd-7e7e-4e87-99f0-88e6485f65f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-6fr7c\" (UID: \"62ceedbd-7e7e-4e87-99f0-88e6485f65f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6fr7c" Oct 30 20:12:17.880929 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:17.880897 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62ceedbd-7e7e-4e87-99f0-88e6485f65f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-6fr7c\" (UID: \"62ceedbd-7e7e-4e87-99f0-88e6485f65f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6fr7c" Oct 30 20:12:17.881142 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:17.880967 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddblf\" (UniqueName: \"kubernetes.io/projected/62ceedbd-7e7e-4e87-99f0-88e6485f65f2-kube-api-access-ddblf\") pod \"maas-api-revoke-sub-ae594979-6fr7c\" (UID: \"62ceedbd-7e7e-4e87-99f0-88e6485f65f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6fr7c" Oct 30 20:12:17.881142 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:17.880996 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62ceedbd-7e7e-4e87-99f0-88e6485f65f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-6fr7c\" (UID: \"62ceedbd-7e7e-4e87-99f0-88e6485f65f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6fr7c" Oct 30 20:12:17.881773 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:17.881748 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62ceedbd-7e7e-4e87-99f0-88e6485f65f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-6fr7c\" (UID: \"62ceedbd-7e7e-4e87-99f0-88e6485f65f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6fr7c" Oct 30 20:12:17.883470 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:17.883444 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62ceedbd-7e7e-4e87-99f0-88e6485f65f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-6fr7c\" (UID: \"62ceedbd-7e7e-4e87-99f0-88e6485f65f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6fr7c" Oct 30 20:12:17.909892 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:17.909862 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddblf\" (UniqueName: \"kubernetes.io/projected/62ceedbd-7e7e-4e87-99f0-88e6485f65f2-kube-api-access-ddblf\") pod \"maas-api-revoke-sub-ae594979-6fr7c\" (UID: \"62ceedbd-7e7e-4e87-99f0-88e6485f65f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6fr7c" Oct 30 20:12:18.060379 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:18.060296 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6fr7c" Oct 30 20:12:18.370144 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:18.370120 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6fr7c"] Oct 30 20:12:18.371159 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:12:18.371130 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62ceedbd_7e7e_4e87_99f0_88e6485f65f2.slice/crio-6a120804390915bb811f14e758cca839318ab339f76e9f72aa1980335786751a WatchSource:0}: Error finding container 6a120804390915bb811f14e758cca839318ab339f76e9f72aa1980335786751a: Status 404 returned error can't find the container with id 6a120804390915bb811f14e758cca839318ab339f76e9f72aa1980335786751a Oct 30 20:12:18.921627 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:18.921592 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qdj4h_da850825-59ab-480f-9047-afd7d2bdc19f/console-operator/1.log" Oct 30 20:12:18.924392 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:18.924356 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qdj4h_da850825-59ab-480f-9047-afd7d2bdc19f/console-operator/1.log" Oct 30 20:12:18.926509 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:18.926488 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xpjpn_688d079a-932d-4f63-97da-5cd6222a03e1/ovn-acl-logging/0.log" Oct 30 20:12:18.928927 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:18.928901 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xpjpn_688d079a-932d-4f63-97da-5cd6222a03e1/ovn-acl-logging/0.log" Oct 30 20:12:19.113086 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:19.113041 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6fr7c" event={"ID":"62ceedbd-7e7e-4e87-99f0-88e6485f65f2","Type":"ContainerStarted","Data":"a3707ef386fbf3e61e087256f81c7811e90e357c4c4ed9a6ebca88e3155c1c5e"} Oct 30 20:12:19.113264 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:19.113104 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6fr7c" event={"ID":"62ceedbd-7e7e-4e87-99f0-88e6485f65f2","Type":"ContainerStarted","Data":"6a120804390915bb811f14e758cca839318ab339f76e9f72aa1980335786751a"} Oct 30 20:12:19.161075 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:19.161026 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6fr7c" podStartSLOduration=2.161007579 podStartE2EDuration="2.161007579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:12:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:12:19.157998121 +0000 UTC m=+1500.570711915" watchObservedRunningTime="2026-10-30 20:12:19.161007579 +0000 UTC m=+1500.573721399" Oct 30 20:12:20.121537 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:20.121499 5088 generic.go:356] "Generic (PLEG): container finished" podID="62ceedbd-7e7e-4e87-99f0-88e6485f65f2" containerID="a3707ef386fbf3e61e087256f81c7811e90e357c4c4ed9a6ebca88e3155c1c5e" exitCode=0 Oct 30 20:12:20.121947 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:20.121585 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6fr7c" event={"ID":"62ceedbd-7e7e-4e87-99f0-88e6485f65f2","Type":"ContainerDied","Data":"a3707ef386fbf3e61e087256f81c7811e90e357c4c4ed9a6ebca88e3155c1c5e"} Oct 30 20:12:21.261701 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:21.261676 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6fr7c" Oct 30 20:12:21.315367 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:21.315330 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62ceedbd-7e7e-4e87-99f0-88e6485f65f2-kube-api-access-ddblf\" (UniqueName: \"kubernetes.io/projected/62ceedbd-7e7e-4e87-99f0-88e6485f65f2-kube-api-access-ddblf\") pod \"62ceedbd-7e7e-4e87-99f0-88e6485f65f2\" (UID: \"62ceedbd-7e7e-4e87-99f0-88e6485f65f2\") " Oct 30 20:12:21.316114 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:21.315405 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62ceedbd-7e7e-4e87-99f0-88e6485f65f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62ceedbd-7e7e-4e87-99f0-88e6485f65f2-maas-api-openshift-ca\") pod \"62ceedbd-7e7e-4e87-99f0-88e6485f65f2\" (UID: \"62ceedbd-7e7e-4e87-99f0-88e6485f65f2\") " Oct 30 20:12:21.316187 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:21.315859 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62ceedbd-7e7e-4e87-99f0-88e6485f65f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62ceedbd-7e7e-4e87-99f0-88e6485f65f2-maas-api-xks-ca\") pod \"62ceedbd-7e7e-4e87-99f0-88e6485f65f2\" (UID: \"62ceedbd-7e7e-4e87-99f0-88e6485f65f2\") " Oct 30 20:12:21.316263 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:21.315991 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62ceedbd-7e7e-4e87-99f0-88e6485f65f2-maas-api-openshift-ca" pod "62ceedbd-7e7e-4e87-99f0-88e6485f65f2" (UID: "62ceedbd-7e7e-4e87-99f0-88e6485f65f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:12:21.320249 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:21.317414 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62ceedbd-7e7e-4e87-99f0-88e6485f65f2-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:12:21.320705 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:21.320667 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62ceedbd-7e7e-4e87-99f0-88e6485f65f2-maas-api-xks-ca" pod "62ceedbd-7e7e-4e87-99f0-88e6485f65f2" (UID: "62ceedbd-7e7e-4e87-99f0-88e6485f65f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:12:21.331462 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:21.331428 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62ceedbd-7e7e-4e87-99f0-88e6485f65f2-kube-api-access-ddblf" pod "62ceedbd-7e7e-4e87-99f0-88e6485f65f2" (UID: "62ceedbd-7e7e-4e87-99f0-88e6485f65f2"). InnerVolumeSpecName "kube-api-access-ddblf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:21.418860 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:21.418772 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddblf\" (UniqueName: \"kubernetes.io/projected/62ceedbd-7e7e-4e87-99f0-88e6485f65f2-kube-api-access-ddblf\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:12:21.418860 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:21.418807 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62ceedbd-7e7e-4e87-99f0-88e6485f65f2-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:12:22.135232 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:22.135200 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6fr7c" Oct 30 20:12:22.135232 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:22.135224 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6fr7c" event={"ID":"62ceedbd-7e7e-4e87-99f0-88e6485f65f2","Type":"ContainerDied","Data":"6a120804390915bb811f14e758cca839318ab339f76e9f72aa1980335786751a"} Oct 30 20:12:22.135481 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:22.135276 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a120804390915bb811f14e758cca839318ab339f76e9f72aa1980335786751a" Oct 30 20:12:22.438652 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:12:22.438547 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:12:27.047182 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:27.047147 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9aac2627-nz2jn"] Oct 30 20:12:27.060294 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:27.060264 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2891b522-8gd2t"] Oct 30 20:12:27.062844 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:27.062816 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9aac2627-nz2jn"] Oct 30 20:12:27.070199 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:27.070165 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2891b522-8gd2t"] Oct 30 20:12:27.421714 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:27.421632 5088 scope.go:122] "RemoveContainer" containerID="9780e31b2f0eb4868cc069ebc55fda95e3ce12500c33763138c5cf7e02729dc9" Oct 30 20:12:27.430876 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:27.430854 5088 scope.go:122] "RemoveContainer" containerID="e177dca68f51f73c541f0e1faea273dc955ae463601e62ed7eacabe2e9d777e1" Oct 30 20:12:27.439075 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:27.439057 5088 scope.go:122] "RemoveContainer" containerID="0f24a00bc89f1b48287b00ea3bbd8a19162402791b3e7d1763abefe344dd0d6c" Oct 30 20:12:27.447672 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:27.447650 5088 scope.go:122] "RemoveContainer" containerID="148ed002f2b11c8794736f4511c4466813ee332e1411e6af730235e536896102" Oct 30 20:12:27.455565 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:27.455548 5088 scope.go:122] "RemoveContainer" containerID="34cec33ae3f1c9111edc95f0c34fbd8067eb728565379b352b63a49205dfa7b9" Oct 30 20:12:28.034758 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:28.034727 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe47f907-snhsh"] Oct 30 20:12:28.048248 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:28.048220 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe47f907-snhsh"] Oct 30 20:12:28.738480 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:28.738446 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b45a2b75-a359-49fd-8f33-3086717b7c39" path="/var/lib/kubelet/pods/b45a2b75-a359-49fd-8f33-3086717b7c39/volumes" Oct 30 20:12:28.738932 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:28.738916 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea2533f2-6a01-44f1-a528-7b5e3eb8dde4" path="/var/lib/kubelet/pods/ea2533f2-6a01-44f1-a528-7b5e3eb8dde4/volumes" Oct 30 20:12:28.739300 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:28.739287 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7754459-8e35-4d81-a0e8-325f39cd7c8b" path="/var/lib/kubelet/pods/f7754459-8e35-4d81-a0e8-325f39cd7c8b/volumes" Oct 30 20:12:28.938097 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:28.938061 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:12:33.974115 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:33.974073 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:12:34.341639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:34.329969 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-c2tb9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:12:34.341639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:34.330071 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" podUID="ce0a1543-79bc-440d-ad18-c045ac745786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:12:39.386766 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:39.386730 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p4wm7"] Oct 30 20:12:39.387189 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:39.387176 5088 cpu_manager.go:414] "containerMap: removing container" podUID="62ceedbd-7e7e-4e87-99f0-88e6485f65f2" containerName="revoke-keys" Oct 30 20:12:39.387189 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:39.387189 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62ceedbd-7e7e-4e87-99f0-88e6485f65f2" containerName="revoke-keys" Oct 30 20:12:39.387282 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:39.387273 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="62ceedbd-7e7e-4e87-99f0-88e6485f65f2" containerName="revoke-keys" Oct 30 20:12:39.390542 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:39.390521 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p4wm7" Oct 30 20:12:39.412267 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:39.412240 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p4wm7"] Oct 30 20:12:39.422557 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:39.422523 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:12:39.422718 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:39.422573 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:12:39.422718 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:39.422586 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:12:39.422718 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:39.422686 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:12:39.464510 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:39.464480 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e46d7707-4ed5-44ae-90ac-73116de6403e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-p4wm7\" (UID: \"e46d7707-4ed5-44ae-90ac-73116de6403e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p4wm7" Oct 30 20:12:39.464685 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:39.464534 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e46d7707-4ed5-44ae-90ac-73116de6403e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-p4wm7\" (UID: \"e46d7707-4ed5-44ae-90ac-73116de6403e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p4wm7" Oct 30 20:12:39.464685 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:39.464642 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgh7w\" (UniqueName: \"kubernetes.io/projected/e46d7707-4ed5-44ae-90ac-73116de6403e-kube-api-access-jgh7w\") pod \"maas-api-revoke-sub-a4aef7d7-p4wm7\" (UID: \"e46d7707-4ed5-44ae-90ac-73116de6403e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p4wm7" Oct 30 20:12:39.565536 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:39.565502 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e46d7707-4ed5-44ae-90ac-73116de6403e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-p4wm7\" (UID: \"e46d7707-4ed5-44ae-90ac-73116de6403e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p4wm7" Oct 30 20:12:39.565726 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:39.565554 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e46d7707-4ed5-44ae-90ac-73116de6403e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-p4wm7\" (UID: \"e46d7707-4ed5-44ae-90ac-73116de6403e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p4wm7" Oct 30 20:12:39.565726 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:39.565573 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgh7w\" (UniqueName: \"kubernetes.io/projected/e46d7707-4ed5-44ae-90ac-73116de6403e-kube-api-access-jgh7w\") pod \"maas-api-revoke-sub-a4aef7d7-p4wm7\" (UID: \"e46d7707-4ed5-44ae-90ac-73116de6403e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p4wm7" Oct 30 20:12:39.566202 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:39.566176 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e46d7707-4ed5-44ae-90ac-73116de6403e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-p4wm7\" (UID: \"e46d7707-4ed5-44ae-90ac-73116de6403e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p4wm7" Oct 30 20:12:39.568011 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:39.567990 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e46d7707-4ed5-44ae-90ac-73116de6403e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-p4wm7\" (UID: \"e46d7707-4ed5-44ae-90ac-73116de6403e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p4wm7" Oct 30 20:12:39.594435 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:39.594409 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgh7w\" (UniqueName: \"kubernetes.io/projected/e46d7707-4ed5-44ae-90ac-73116de6403e-kube-api-access-jgh7w\") pod \"maas-api-revoke-sub-a4aef7d7-p4wm7\" (UID: \"e46d7707-4ed5-44ae-90ac-73116de6403e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p4wm7" Oct 30 20:12:39.703993 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:39.703912 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p4wm7" Oct 30 20:12:40.007931 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:40.007904 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p4wm7"] Oct 30 20:12:40.009452 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:12:40.009415 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode46d7707_4ed5_44ae_90ac_73116de6403e.slice/crio-d45198770a810aa2948285ed213ffe72b20b132737cf69a16fdb6f7d4e1f6b21 WatchSource:0}: Error finding container d45198770a810aa2948285ed213ffe72b20b132737cf69a16fdb6f7d4e1f6b21: Status 404 returned error can't find the container with id d45198770a810aa2948285ed213ffe72b20b132737cf69a16fdb6f7d4e1f6b21 Oct 30 20:12:40.295316 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:40.295287 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p4wm7" event={"ID":"e46d7707-4ed5-44ae-90ac-73116de6403e","Type":"ContainerStarted","Data":"d45198770a810aa2948285ed213ffe72b20b132737cf69a16fdb6f7d4e1f6b21"} Oct 30 20:12:41.036684 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:41.036604 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6e06370-nwczs"] Oct 30 20:12:41.050175 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:41.050143 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6e06370-nwczs"] Oct 30 20:12:41.304018 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:41.303979 5088 generic.go:356] "Generic (PLEG): container finished" podID="e46d7707-4ed5-44ae-90ac-73116de6403e" containerID="3877d83d30d5a9021cd8b7ffad1cb3919c6890ee875f441fa16e8ee09a66a495" exitCode=0 Oct 30 20:12:41.304208 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:41.304039 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p4wm7" event={"ID":"e46d7707-4ed5-44ae-90ac-73116de6403e","Type":"ContainerDied","Data":"3877d83d30d5a9021cd8b7ffad1cb3919c6890ee875f441fa16e8ee09a66a495"} Oct 30 20:12:42.464863 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:42.464833 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p4wm7" Oct 30 20:12:42.599148 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:42.599045 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e46d7707-4ed5-44ae-90ac-73116de6403e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e46d7707-4ed5-44ae-90ac-73116de6403e-maas-api-openshift-ca\") pod \"e46d7707-4ed5-44ae-90ac-73116de6403e\" (UID: \"e46d7707-4ed5-44ae-90ac-73116de6403e\") " Oct 30 20:12:42.599148 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:42.599125 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e46d7707-4ed5-44ae-90ac-73116de6403e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e46d7707-4ed5-44ae-90ac-73116de6403e-maas-api-xks-ca\") pod \"e46d7707-4ed5-44ae-90ac-73116de6403e\" (UID: \"e46d7707-4ed5-44ae-90ac-73116de6403e\") " Oct 30 20:12:42.599413 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:42.599264 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e46d7707-4ed5-44ae-90ac-73116de6403e-kube-api-access-jgh7w\" (UniqueName: \"kubernetes.io/projected/e46d7707-4ed5-44ae-90ac-73116de6403e-kube-api-access-jgh7w\") pod \"e46d7707-4ed5-44ae-90ac-73116de6403e\" (UID: \"e46d7707-4ed5-44ae-90ac-73116de6403e\") " Oct 30 20:12:42.599532 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:42.599507 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e46d7707-4ed5-44ae-90ac-73116de6403e-maas-api-openshift-ca" pod "e46d7707-4ed5-44ae-90ac-73116de6403e" (UID: "e46d7707-4ed5-44ae-90ac-73116de6403e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:12:42.599682 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:42.599665 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e46d7707-4ed5-44ae-90ac-73116de6403e-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:12:42.601400 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:42.601368 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e46d7707-4ed5-44ae-90ac-73116de6403e-maas-api-xks-ca" pod "e46d7707-4ed5-44ae-90ac-73116de6403e" (UID: "e46d7707-4ed5-44ae-90ac-73116de6403e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:12:42.601507 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:42.601425 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e46d7707-4ed5-44ae-90ac-73116de6403e-kube-api-access-jgh7w" pod "e46d7707-4ed5-44ae-90ac-73116de6403e" (UID: "e46d7707-4ed5-44ae-90ac-73116de6403e"). InnerVolumeSpecName "kube-api-access-jgh7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:42.701217 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:42.701179 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgh7w\" (UniqueName: \"kubernetes.io/projected/e46d7707-4ed5-44ae-90ac-73116de6403e-kube-api-access-jgh7w\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:12:42.701217 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:42.701210 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e46d7707-4ed5-44ae-90ac-73116de6403e-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:12:42.738742 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:42.738705 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5faeded-3cb2-4229-9710-4579376f59cc" path="/var/lib/kubelet/pods/e5faeded-3cb2-4229-9710-4579376f59cc/volumes" Oct 30 20:12:43.317671 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:43.317638 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p4wm7" Oct 30 20:12:43.317671 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:43.317648 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p4wm7" event={"ID":"e46d7707-4ed5-44ae-90ac-73116de6403e","Type":"ContainerDied","Data":"d45198770a810aa2948285ed213ffe72b20b132737cf69a16fdb6f7d4e1f6b21"} Oct 30 20:12:43.317907 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:43.317701 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d45198770a810aa2948285ed213ffe72b20b132737cf69a16fdb6f7d4e1f6b21" Oct 30 20:12:50.632378 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:50.632346 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:12:52.436502 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:12:52.436467 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:12:54.944660 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:54.944626 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:12:58.042648 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:58.041688 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0350007e-v6mjg"] Oct 30 20:12:58.063632 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:58.056824 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0350007e-v6mjg"] Oct 30 20:12:58.745502 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:12:58.745464 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b07cf7ca-5ebb-49c6-a757-1243524af40f" path="/var/lib/kubelet/pods/b07cf7ca-5ebb-49c6-a757-1243524af40f/volumes" Oct 30 20:13:00.589693 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:00.589660 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nqnxl"] Oct 30 20:13:00.590140 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:00.590126 5088 cpu_manager.go:414] "containerMap: removing container" podUID="e46d7707-4ed5-44ae-90ac-73116de6403e" containerName="revoke-keys" Oct 30 20:13:00.590185 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:00.590141 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e46d7707-4ed5-44ae-90ac-73116de6403e" containerName="revoke-keys" Oct 30 20:13:00.590247 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:00.590237 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="e46d7707-4ed5-44ae-90ac-73116de6403e" containerName="revoke-keys" Oct 30 20:13:00.594901 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:00.594884 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nqnxl" Oct 30 20:13:00.607730 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:00.607707 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mllv5\" (UniqueName: \"kubernetes.io/projected/0fbcac15-3eac-4838-b1db-ddd8a11e9df1-kube-api-access-mllv5\") pod \"maas-api-revoke-sub-79805ca9-nqnxl\" (UID: \"0fbcac15-3eac-4838-b1db-ddd8a11e9df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nqnxl" Oct 30 20:13:00.607877 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:00.607862 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fbcac15-3eac-4838-b1db-ddd8a11e9df1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-nqnxl\" (UID: \"0fbcac15-3eac-4838-b1db-ddd8a11e9df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nqnxl" Oct 30 20:13:00.607945 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:00.607906 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fbcac15-3eac-4838-b1db-ddd8a11e9df1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-nqnxl\" (UID: \"0fbcac15-3eac-4838-b1db-ddd8a11e9df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nqnxl" Oct 30 20:13:00.614645 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:00.614603 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nqnxl"] Oct 30 20:13:00.625279 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:00.625260 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:13:00.625387 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:00.625345 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:13:00.625456 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:00.625436 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:13:00.625496 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:00.625466 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:13:00.708842 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:00.708810 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fbcac15-3eac-4838-b1db-ddd8a11e9df1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-nqnxl\" (UID: \"0fbcac15-3eac-4838-b1db-ddd8a11e9df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nqnxl" Oct 30 20:13:00.709032 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:00.708855 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fbcac15-3eac-4838-b1db-ddd8a11e9df1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-nqnxl\" (UID: \"0fbcac15-3eac-4838-b1db-ddd8a11e9df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nqnxl" Oct 30 20:13:00.709032 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:00.708909 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mllv5\" (UniqueName: \"kubernetes.io/projected/0fbcac15-3eac-4838-b1db-ddd8a11e9df1-kube-api-access-mllv5\") pod \"maas-api-revoke-sub-79805ca9-nqnxl\" (UID: \"0fbcac15-3eac-4838-b1db-ddd8a11e9df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nqnxl" Oct 30 20:13:00.709598 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:00.709578 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fbcac15-3eac-4838-b1db-ddd8a11e9df1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-nqnxl\" (UID: \"0fbcac15-3eac-4838-b1db-ddd8a11e9df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nqnxl" Oct 30 20:13:00.711199 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:00.711177 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fbcac15-3eac-4838-b1db-ddd8a11e9df1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-nqnxl\" (UID: \"0fbcac15-3eac-4838-b1db-ddd8a11e9df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nqnxl" Oct 30 20:13:00.735512 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:00.735487 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mllv5\" (UniqueName: \"kubernetes.io/projected/0fbcac15-3eac-4838-b1db-ddd8a11e9df1-kube-api-access-mllv5\") pod \"maas-api-revoke-sub-79805ca9-nqnxl\" (UID: \"0fbcac15-3eac-4838-b1db-ddd8a11e9df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nqnxl" Oct 30 20:13:00.907200 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:00.907126 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nqnxl" Oct 30 20:13:01.215421 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:01.215391 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nqnxl"] Oct 30 20:13:01.217248 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:13:01.217218 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fbcac15_3eac_4838_b1db_ddd8a11e9df1.slice/crio-e52fcea763af5835ed50999f1714231f0bc6b1e3b0aafa7ee7197a4de346bfc7 WatchSource:0}: Error finding container e52fcea763af5835ed50999f1714231f0bc6b1e3b0aafa7ee7197a4de346bfc7: Status 404 returned error can't find the container with id e52fcea763af5835ed50999f1714231f0bc6b1e3b0aafa7ee7197a4de346bfc7 Oct 30 20:13:01.469033 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:01.468944 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nqnxl" event={"ID":"0fbcac15-3eac-4838-b1db-ddd8a11e9df1","Type":"ContainerStarted","Data":"e52fcea763af5835ed50999f1714231f0bc6b1e3b0aafa7ee7197a4de346bfc7"} Oct 30 20:13:02.476760 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:02.476677 5088 generic.go:356] "Generic (PLEG): container finished" podID="0fbcac15-3eac-4838-b1db-ddd8a11e9df1" containerID="e9948dadfed2fb320f63aad60b5e068574ffb6adfe45cb22b4533bf77d9ccd2b" exitCode=0 Oct 30 20:13:02.477125 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:02.476756 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nqnxl" event={"ID":"0fbcac15-3eac-4838-b1db-ddd8a11e9df1","Type":"ContainerDied","Data":"e9948dadfed2fb320f63aad60b5e068574ffb6adfe45cb22b4533bf77d9ccd2b"} Oct 30 20:13:03.628008 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:03.627984 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nqnxl" Oct 30 20:13:03.635095 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:03.635076 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fbcac15-3eac-4838-b1db-ddd8a11e9df1-kube-api-access-mllv5\" (UniqueName: \"kubernetes.io/projected/0fbcac15-3eac-4838-b1db-ddd8a11e9df1-kube-api-access-mllv5\") pod \"0fbcac15-3eac-4838-b1db-ddd8a11e9df1\" (UID: \"0fbcac15-3eac-4838-b1db-ddd8a11e9df1\") " Oct 30 20:13:03.635162 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:03.635107 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fbcac15-3eac-4838-b1db-ddd8a11e9df1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fbcac15-3eac-4838-b1db-ddd8a11e9df1-maas-api-openshift-ca\") pod \"0fbcac15-3eac-4838-b1db-ddd8a11e9df1\" (UID: \"0fbcac15-3eac-4838-b1db-ddd8a11e9df1\") " Oct 30 20:13:03.635230 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:03.635215 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fbcac15-3eac-4838-b1db-ddd8a11e9df1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fbcac15-3eac-4838-b1db-ddd8a11e9df1-maas-api-xks-ca\") pod \"0fbcac15-3eac-4838-b1db-ddd8a11e9df1\" (UID: \"0fbcac15-3eac-4838-b1db-ddd8a11e9df1\") " Oct 30 20:13:03.635553 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:03.635520 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fbcac15-3eac-4838-b1db-ddd8a11e9df1-maas-api-openshift-ca" pod "0fbcac15-3eac-4838-b1db-ddd8a11e9df1" (UID: "0fbcac15-3eac-4838-b1db-ddd8a11e9df1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:13:03.637115 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:03.637097 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fbcac15-3eac-4838-b1db-ddd8a11e9df1-maas-api-xks-ca" pod "0fbcac15-3eac-4838-b1db-ddd8a11e9df1" (UID: "0fbcac15-3eac-4838-b1db-ddd8a11e9df1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:13:03.637188 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:03.637121 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fbcac15-3eac-4838-b1db-ddd8a11e9df1-kube-api-access-mllv5" pod "0fbcac15-3eac-4838-b1db-ddd8a11e9df1" (UID: "0fbcac15-3eac-4838-b1db-ddd8a11e9df1"). InnerVolumeSpecName "kube-api-access-mllv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:03.736678 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:03.736577 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mllv5\" (UniqueName: \"kubernetes.io/projected/0fbcac15-3eac-4838-b1db-ddd8a11e9df1-kube-api-access-mllv5\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:13:03.736678 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:03.736606 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fbcac15-3eac-4838-b1db-ddd8a11e9df1-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:13:03.736678 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:03.736644 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fbcac15-3eac-4838-b1db-ddd8a11e9df1-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:13:04.489659 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:04.489604 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nqnxl" Oct 30 20:13:04.489659 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:04.489650 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nqnxl" event={"ID":"0fbcac15-3eac-4838-b1db-ddd8a11e9df1","Type":"ContainerDied","Data":"e52fcea763af5835ed50999f1714231f0bc6b1e3b0aafa7ee7197a4de346bfc7"} Oct 30 20:13:04.489847 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:04.489696 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e52fcea763af5835ed50999f1714231f0bc6b1e3b0aafa7ee7197a4de346bfc7" Oct 30 20:13:05.038801 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:05.038771 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bebf4566-g9fwj"] Oct 30 20:13:05.052099 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:05.052072 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bebf4566-g9fwj"] Oct 30 20:13:06.738672 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:06.738640 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad2f9107-20fe-4ae3-ae9a-e92c315e5a45" path="/var/lib/kubelet/pods/ad2f9107-20fe-4ae3-ae9a-e92c315e5a45/volumes" Oct 30 20:13:12.665603 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:12.665568 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:13:17.465634 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:17.465576 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-29tcl"] Oct 30 20:13:17.466044 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:17.466029 5088 cpu_manager.go:414] "containerMap: removing container" podUID="0fbcac15-3eac-4838-b1db-ddd8a11e9df1" containerName="revoke-keys" Oct 30 20:13:17.466094 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:17.466044 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fbcac15-3eac-4838-b1db-ddd8a11e9df1" containerName="revoke-keys" Oct 30 20:13:17.466137 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:17.466127 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fbcac15-3eac-4838-b1db-ddd8a11e9df1" containerName="revoke-keys" Oct 30 20:13:17.469402 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:17.469385 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-29tcl" Oct 30 20:13:17.492208 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:17.492185 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-29tcl"] Oct 30 20:13:17.502712 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:17.502684 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:13:17.502891 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:17.502873 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:13:17.502955 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:17.502893 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:13:17.502955 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:17.502876 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:13:17.523695 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:17.523669 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhfqz\" (UniqueName: \"kubernetes.io/projected/c71e4333-f91b-4cad-864c-a5859918847a-kube-api-access-mhfqz\") pod \"maas-api-revoke-sub-0b7e8edb-29tcl\" (UID: \"c71e4333-f91b-4cad-864c-a5859918847a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-29tcl" Oct 30 20:13:17.523825 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:17.523805 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c71e4333-f91b-4cad-864c-a5859918847a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-29tcl\" (UID: \"c71e4333-f91b-4cad-864c-a5859918847a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-29tcl" Oct 30 20:13:17.523927 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:17.523901 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c71e4333-f91b-4cad-864c-a5859918847a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-29tcl\" (UID: \"c71e4333-f91b-4cad-864c-a5859918847a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-29tcl" Oct 30 20:13:17.625030 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:17.624981 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhfqz\" (UniqueName: \"kubernetes.io/projected/c71e4333-f91b-4cad-864c-a5859918847a-kube-api-access-mhfqz\") pod \"maas-api-revoke-sub-0b7e8edb-29tcl\" (UID: \"c71e4333-f91b-4cad-864c-a5859918847a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-29tcl" Oct 30 20:13:17.625179 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:17.625103 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c71e4333-f91b-4cad-864c-a5859918847a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-29tcl\" (UID: \"c71e4333-f91b-4cad-864c-a5859918847a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-29tcl" Oct 30 20:13:17.625179 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:17.625146 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c71e4333-f91b-4cad-864c-a5859918847a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-29tcl\" (UID: \"c71e4333-f91b-4cad-864c-a5859918847a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-29tcl" Oct 30 20:13:17.625752 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:17.625734 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c71e4333-f91b-4cad-864c-a5859918847a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-29tcl\" (UID: \"c71e4333-f91b-4cad-864c-a5859918847a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-29tcl" Oct 30 20:13:17.627495 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:17.627472 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c71e4333-f91b-4cad-864c-a5859918847a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-29tcl\" (UID: \"c71e4333-f91b-4cad-864c-a5859918847a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-29tcl" Oct 30 20:13:17.651513 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:17.651484 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhfqz\" (UniqueName: \"kubernetes.io/projected/c71e4333-f91b-4cad-864c-a5859918847a-kube-api-access-mhfqz\") pod \"maas-api-revoke-sub-0b7e8edb-29tcl\" (UID: \"c71e4333-f91b-4cad-864c-a5859918847a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-29tcl" Oct 30 20:13:17.782681 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:17.782656 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-29tcl" Oct 30 20:13:18.086026 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:18.086002 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-29tcl"] Oct 30 20:13:18.087811 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:13:18.087782 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc71e4333_f91b_4cad_864c_a5859918847a.slice/crio-cafec2085b3a1a149188277de1c3b2e00aace51960165d5cbf9f6e856ab68a0a WatchSource:0}: Error finding container cafec2085b3a1a149188277de1c3b2e00aace51960165d5cbf9f6e856ab68a0a: Status 404 returned error can't find the container with id cafec2085b3a1a149188277de1c3b2e00aace51960165d5cbf9f6e856ab68a0a Oct 30 20:13:18.649573 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:18.649499 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-29tcl" event={"ID":"c71e4333-f91b-4cad-864c-a5859918847a","Type":"ContainerStarted","Data":"f2649853951573801a308505fa3af0008c16ff2a07bccd78437e8d549e45e5c9"} Oct 30 20:13:18.649573 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:18.649549 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-29tcl" event={"ID":"c71e4333-f91b-4cad-864c-a5859918847a","Type":"ContainerStarted","Data":"cafec2085b3a1a149188277de1c3b2e00aace51960165d5cbf9f6e856ab68a0a"} Oct 30 20:13:18.696056 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:18.696005 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-29tcl" podStartSLOduration=1.695991728 podStartE2EDuration="1.695991728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:13:18.693716311 +0000 UTC m=+1560.106430104" watchObservedRunningTime="2026-10-30 20:13:18.695991728 +0000 UTC m=+1560.108705506" Oct 30 20:13:19.656768 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:19.656729 5088 generic.go:356] "Generic (PLEG): container finished" podID="c71e4333-f91b-4cad-864c-a5859918847a" containerID="f2649853951573801a308505fa3af0008c16ff2a07bccd78437e8d549e45e5c9" exitCode=0 Oct 30 20:13:19.657164 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:19.656784 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-29tcl" event={"ID":"c71e4333-f91b-4cad-864c-a5859918847a","Type":"ContainerDied","Data":"f2649853951573801a308505fa3af0008c16ff2a07bccd78437e8d549e45e5c9"} Oct 30 20:13:20.821576 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:20.821550 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-29tcl" Oct 30 20:13:20.957571 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:20.957476 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c71e4333-f91b-4cad-864c-a5859918847a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c71e4333-f91b-4cad-864c-a5859918847a-maas-api-xks-ca\") pod \"c71e4333-f91b-4cad-864c-a5859918847a\" (UID: \"c71e4333-f91b-4cad-864c-a5859918847a\") " Oct 30 20:13:20.957571 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:20.957536 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c71e4333-f91b-4cad-864c-a5859918847a-kube-api-access-mhfqz\" (UniqueName: \"kubernetes.io/projected/c71e4333-f91b-4cad-864c-a5859918847a-kube-api-access-mhfqz\") pod \"c71e4333-f91b-4cad-864c-a5859918847a\" (UID: \"c71e4333-f91b-4cad-864c-a5859918847a\") " Oct 30 20:13:20.957848 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:20.957574 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c71e4333-f91b-4cad-864c-a5859918847a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c71e4333-f91b-4cad-864c-a5859918847a-maas-api-openshift-ca\") pod \"c71e4333-f91b-4cad-864c-a5859918847a\" (UID: \"c71e4333-f91b-4cad-864c-a5859918847a\") " Oct 30 20:13:20.957976 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:20.957952 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c71e4333-f91b-4cad-864c-a5859918847a-maas-api-openshift-ca" pod "c71e4333-f91b-4cad-864c-a5859918847a" (UID: "c71e4333-f91b-4cad-864c-a5859918847a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:13:20.958052 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:20.958038 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c71e4333-f91b-4cad-864c-a5859918847a-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:13:20.959730 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:20.959707 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c71e4333-f91b-4cad-864c-a5859918847a-maas-api-xks-ca" pod "c71e4333-f91b-4cad-864c-a5859918847a" (UID: "c71e4333-f91b-4cad-864c-a5859918847a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:13:20.959819 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:20.959787 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c71e4333-f91b-4cad-864c-a5859918847a-kube-api-access-mhfqz" pod "c71e4333-f91b-4cad-864c-a5859918847a" (UID: "c71e4333-f91b-4cad-864c-a5859918847a"). InnerVolumeSpecName "kube-api-access-mhfqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:21.059220 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:21.059185 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c71e4333-f91b-4cad-864c-a5859918847a-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:13:21.059220 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:21.059214 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhfqz\" (UniqueName: \"kubernetes.io/projected/c71e4333-f91b-4cad-864c-a5859918847a-kube-api-access-mhfqz\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:13:21.670308 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:21.670273 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-29tcl" Oct 30 20:13:21.670510 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:21.670307 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-29tcl" event={"ID":"c71e4333-f91b-4cad-864c-a5859918847a","Type":"ContainerDied","Data":"cafec2085b3a1a149188277de1c3b2e00aace51960165d5cbf9f6e856ab68a0a"} Oct 30 20:13:21.670510 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:21.670396 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cafec2085b3a1a149188277de1c3b2e00aace51960165d5cbf9f6e856ab68a0a" Oct 30 20:13:22.438638 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:13:22.438583 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:13:27.495865 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:27.495836 5088 scope.go:122] "RemoveContainer" containerID="519936d87a96adac40aefbf4d5be3677a34523091745ae557b3e019d8b318ed4" Oct 30 20:13:27.505978 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:27.505958 5088 scope.go:122] "RemoveContainer" containerID="eadcbf3ad4695c28b102286c1fd0ef0502be02b365d937b01045691fb090b939" Oct 30 20:13:27.517145 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:27.517127 5088 scope.go:122] "RemoveContainer" containerID="c767e8356d00fca7d9b63b67a3a4b969852fd55132a8603436807b992e3d36a7" Oct 30 20:13:27.525707 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:27.525689 5088 scope.go:122] "RemoveContainer" containerID="6cae5d08d673b861727974c76117397e49a5786137f44879ea6594942e5ba40d" Oct 30 20:13:27.533789 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:27.533772 5088 scope.go:122] "RemoveContainer" containerID="0b23069896ab0c14ad059cb94401b5fb1dd48892d85bc6369d2142399d570068" Oct 30 20:13:27.542040 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:27.542020 5088 scope.go:122] "RemoveContainer" containerID="e79a786e6a101b0a665f50bd8978730341ad793ba9a856a525178087cdc5b306" Oct 30 20:13:27.549554 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:27.549538 5088 scope.go:122] "RemoveContainer" containerID="c89fbb0ffd35658bc753eb97cbe3d5cba0665a9f3a0b67a978964abfccab9945" Oct 30 20:13:27.556989 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:27.556973 5088 scope.go:122] "RemoveContainer" containerID="223cc417b99fed109810a38316ebd90205fca5c70e153c5074619d86494663be" Oct 30 20:13:27.564534 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:27.564517 5088 scope.go:122] "RemoveContainer" containerID="853c52cf37bf5e2f25d8c97df798ca299b49823df9a42d575e23789e1eb7fba9" Oct 30 20:13:27.573097 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:27.573074 5088 scope.go:122] "RemoveContainer" containerID="a46e16e144f3398c61b6b3e965e037d426be69a1363383e46b2bd2c012054c9d" Oct 30 20:13:28.248921 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:28.248886 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-455qf"] Oct 30 20:13:28.249339 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:28.249326 5088 cpu_manager.go:414] "containerMap: removing container" podUID="c71e4333-f91b-4cad-864c-a5859918847a" containerName="revoke-keys" Oct 30 20:13:28.249395 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:28.249341 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c71e4333-f91b-4cad-864c-a5859918847a" containerName="revoke-keys" Oct 30 20:13:28.249440 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:28.249431 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="c71e4333-f91b-4cad-864c-a5859918847a" containerName="revoke-keys" Oct 30 20:13:28.254573 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:28.254550 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-455qf" Oct 30 20:13:28.273701 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:28.273675 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-455qf"] Oct 30 20:13:28.283471 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:28.283340 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:13:28.283471 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:28.283434 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:13:28.315349 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:28.315319 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:13:28.315509 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:28.315374 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:13:28.324678 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:28.324646 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d35f4ba-9ca1-4981-8b05-e838331242d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-455qf\" (UID: \"5d35f4ba-9ca1-4981-8b05-e838331242d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-455qf" Oct 30 20:13:28.324819 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:28.324698 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcwb5\" (UniqueName: \"kubernetes.io/projected/5d35f4ba-9ca1-4981-8b05-e838331242d2-kube-api-access-lcwb5\") pod \"maas-api-revoke-sub-3d4156f1-455qf\" (UID: \"5d35f4ba-9ca1-4981-8b05-e838331242d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-455qf" Oct 30 20:13:28.324819 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:28.324804 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d35f4ba-9ca1-4981-8b05-e838331242d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-455qf\" (UID: \"5d35f4ba-9ca1-4981-8b05-e838331242d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-455qf" Oct 30 20:13:28.425559 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:28.425531 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d35f4ba-9ca1-4981-8b05-e838331242d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-455qf\" (UID: \"5d35f4ba-9ca1-4981-8b05-e838331242d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-455qf" Oct 30 20:13:28.425713 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:28.425565 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcwb5\" (UniqueName: \"kubernetes.io/projected/5d35f4ba-9ca1-4981-8b05-e838331242d2-kube-api-access-lcwb5\") pod \"maas-api-revoke-sub-3d4156f1-455qf\" (UID: \"5d35f4ba-9ca1-4981-8b05-e838331242d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-455qf" Oct 30 20:13:28.425713 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:28.425592 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d35f4ba-9ca1-4981-8b05-e838331242d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-455qf\" (UID: \"5d35f4ba-9ca1-4981-8b05-e838331242d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-455qf" Oct 30 20:13:28.426212 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:28.426193 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d35f4ba-9ca1-4981-8b05-e838331242d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-455qf\" (UID: \"5d35f4ba-9ca1-4981-8b05-e838331242d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-455qf" Oct 30 20:13:28.427852 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:28.427835 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d35f4ba-9ca1-4981-8b05-e838331242d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-455qf\" (UID: \"5d35f4ba-9ca1-4981-8b05-e838331242d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-455qf" Oct 30 20:13:28.452554 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:28.452527 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcwb5\" (UniqueName: \"kubernetes.io/projected/5d35f4ba-9ca1-4981-8b05-e838331242d2-kube-api-access-lcwb5\") pod \"maas-api-revoke-sub-3d4156f1-455qf\" (UID: \"5d35f4ba-9ca1-4981-8b05-e838331242d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-455qf" Oct 30 20:13:28.571597 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:28.571542 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-455qf" Oct 30 20:13:28.664628 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:28.663981 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:13:28.874759 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:28.874729 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-455qf"] Oct 30 20:13:28.875325 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:13:28.875303 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d35f4ba_9ca1_4981_8b05_e838331242d2.slice/crio-07494fc88775ca7eca52a03670c884a85e04a248eb021bf4b637e63e97570204 WatchSource:0}: Error finding container 07494fc88775ca7eca52a03670c884a85e04a248eb021bf4b637e63e97570204: Status 404 returned error can't find the container with id 07494fc88775ca7eca52a03670c884a85e04a248eb021bf4b637e63e97570204 Oct 30 20:13:29.721302 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:29.721269 5088 generic.go:356] "Generic (PLEG): container finished" podID="5d35f4ba-9ca1-4981-8b05-e838331242d2" containerID="3e5a5f548c077e1f54c95c455d71db3b59337570cbd571b5e8dcf04d38c0791c" exitCode=0 Oct 30 20:13:29.721595 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:29.721351 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-455qf" event={"ID":"5d35f4ba-9ca1-4981-8b05-e838331242d2","Type":"ContainerDied","Data":"3e5a5f548c077e1f54c95c455d71db3b59337570cbd571b5e8dcf04d38c0791c"} Oct 30 20:13:29.721595 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:29.721401 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-455qf" event={"ID":"5d35f4ba-9ca1-4981-8b05-e838331242d2","Type":"ContainerStarted","Data":"07494fc88775ca7eca52a03670c884a85e04a248eb021bf4b637e63e97570204"} Oct 30 20:13:30.867380 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:30.867355 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-455qf" Oct 30 20:13:30.950117 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:30.950056 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d35f4ba-9ca1-4981-8b05-e838331242d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d35f4ba-9ca1-4981-8b05-e838331242d2-maas-api-xks-ca\") pod \"5d35f4ba-9ca1-4981-8b05-e838331242d2\" (UID: \"5d35f4ba-9ca1-4981-8b05-e838331242d2\") " Oct 30 20:13:30.950260 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:30.950126 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d35f4ba-9ca1-4981-8b05-e838331242d2-kube-api-access-lcwb5\" (UniqueName: \"kubernetes.io/projected/5d35f4ba-9ca1-4981-8b05-e838331242d2-kube-api-access-lcwb5\") pod \"5d35f4ba-9ca1-4981-8b05-e838331242d2\" (UID: \"5d35f4ba-9ca1-4981-8b05-e838331242d2\") " Oct 30 20:13:30.950260 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:30.950164 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d35f4ba-9ca1-4981-8b05-e838331242d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d35f4ba-9ca1-4981-8b05-e838331242d2-maas-api-openshift-ca\") pod \"5d35f4ba-9ca1-4981-8b05-e838331242d2\" (UID: \"5d35f4ba-9ca1-4981-8b05-e838331242d2\") " Oct 30 20:13:30.950570 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:30.950533 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d35f4ba-9ca1-4981-8b05-e838331242d2-maas-api-openshift-ca" pod "5d35f4ba-9ca1-4981-8b05-e838331242d2" (UID: "5d35f4ba-9ca1-4981-8b05-e838331242d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:13:30.952183 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:30.952158 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d35f4ba-9ca1-4981-8b05-e838331242d2-maas-api-xks-ca" pod "5d35f4ba-9ca1-4981-8b05-e838331242d2" (UID: "5d35f4ba-9ca1-4981-8b05-e838331242d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:13:30.952289 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:30.952198 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d35f4ba-9ca1-4981-8b05-e838331242d2-kube-api-access-lcwb5" pod "5d35f4ba-9ca1-4981-8b05-e838331242d2" (UID: "5d35f4ba-9ca1-4981-8b05-e838331242d2"). InnerVolumeSpecName "kube-api-access-lcwb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:31.051871 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:31.051844 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d35f4ba-9ca1-4981-8b05-e838331242d2-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:13:31.051871 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:31.051870 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d35f4ba-9ca1-4981-8b05-e838331242d2-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:13:31.052034 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:31.051880 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcwb5\" (UniqueName: \"kubernetes.io/projected/5d35f4ba-9ca1-4981-8b05-e838331242d2-kube-api-access-lcwb5\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:13:31.733284 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:31.733253 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-455qf" Oct 30 20:13:31.733488 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:31.733288 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-455qf" event={"ID":"5d35f4ba-9ca1-4981-8b05-e838331242d2","Type":"ContainerDied","Data":"07494fc88775ca7eca52a03670c884a85e04a248eb021bf4b637e63e97570204"} Oct 30 20:13:31.733488 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:31.733328 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07494fc88775ca7eca52a03670c884a85e04a248eb021bf4b637e63e97570204" Oct 30 20:13:32.041627 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:32.041546 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcafb519-gmmj2"] Oct 30 20:13:32.046645 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:32.046592 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcafb519-gmmj2"] Oct 30 20:13:32.739723 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:32.739685 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="442a352a-40cd-4152-97a5-88a598908ac1" path="/var/lib/kubelet/pods/442a352a-40cd-4152-97a5-88a598908ac1/volumes" Oct 30 20:13:39.475796 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:39.475758 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:13:44.655442 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:44.655408 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:13:48.658252 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:48.658219 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-455qf"] Oct 30 20:13:48.671181 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:48.671154 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-455qf"] Oct 30 20:13:48.712341 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:48.712316 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2m9ng"] Oct 30 20:13:48.712777 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:48.712763 5088 cpu_manager.go:414] "containerMap: removing container" podUID="5d35f4ba-9ca1-4981-8b05-e838331242d2" containerName="revoke-keys" Oct 30 20:13:48.712828 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:48.712778 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d35f4ba-9ca1-4981-8b05-e838331242d2" containerName="revoke-keys" Oct 30 20:13:48.712868 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:48.712864 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d35f4ba-9ca1-4981-8b05-e838331242d2" containerName="revoke-keys" Oct 30 20:13:48.716045 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:48.716026 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2m9ng" Oct 30 20:13:48.734607 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:48.734585 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:13:48.734805 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:48.734789 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:13:48.734935 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:48.734862 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:13:48.735021 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:48.734947 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:13:48.739810 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:48.739782 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d35f4ba-9ca1-4981-8b05-e838331242d2" path="/var/lib/kubelet/pods/5d35f4ba-9ca1-4981-8b05-e838331242d2/volumes" Oct 30 20:13:48.740136 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:48.740113 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2m9ng"] Oct 30 20:13:48.831233 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:48.831208 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8bh5\" (UniqueName: \"kubernetes.io/projected/c701b842-c3d6-4861-bfc2-a4196cd78c01-kube-api-access-l8bh5\") pod \"maas-api-revoke-sub-3d4156f1-2m9ng\" (UID: \"c701b842-c3d6-4861-bfc2-a4196cd78c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2m9ng" Oct 30 20:13:48.831376 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:48.831277 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c701b842-c3d6-4861-bfc2-a4196cd78c01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-2m9ng\" (UID: \"c701b842-c3d6-4861-bfc2-a4196cd78c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2m9ng" Oct 30 20:13:48.831465 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:48.831445 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c701b842-c3d6-4861-bfc2-a4196cd78c01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-2m9ng\" (UID: \"c701b842-c3d6-4861-bfc2-a4196cd78c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2m9ng" Oct 30 20:13:48.933114 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:48.933054 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8bh5\" (UniqueName: \"kubernetes.io/projected/c701b842-c3d6-4861-bfc2-a4196cd78c01-kube-api-access-l8bh5\") pod \"maas-api-revoke-sub-3d4156f1-2m9ng\" (UID: \"c701b842-c3d6-4861-bfc2-a4196cd78c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2m9ng" Oct 30 20:13:48.933114 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:48.933102 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c701b842-c3d6-4861-bfc2-a4196cd78c01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-2m9ng\" (UID: \"c701b842-c3d6-4861-bfc2-a4196cd78c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2m9ng" Oct 30 20:13:48.933316 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:48.933152 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c701b842-c3d6-4861-bfc2-a4196cd78c01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-2m9ng\" (UID: \"c701b842-c3d6-4861-bfc2-a4196cd78c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2m9ng" Oct 30 20:13:48.933826 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:48.933806 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c701b842-c3d6-4861-bfc2-a4196cd78c01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-2m9ng\" (UID: \"c701b842-c3d6-4861-bfc2-a4196cd78c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2m9ng" Oct 30 20:13:48.935345 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:48.935325 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c701b842-c3d6-4861-bfc2-a4196cd78c01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-2m9ng\" (UID: \"c701b842-c3d6-4861-bfc2-a4196cd78c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2m9ng" Oct 30 20:13:48.958591 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:48.958570 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8bh5\" (UniqueName: \"kubernetes.io/projected/c701b842-c3d6-4861-bfc2-a4196cd78c01-kube-api-access-l8bh5\") pod \"maas-api-revoke-sub-3d4156f1-2m9ng\" (UID: \"c701b842-c3d6-4861-bfc2-a4196cd78c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2m9ng" Oct 30 20:13:49.028429 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:49.028410 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2m9ng" Oct 30 20:13:49.329649 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:49.329625 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2m9ng"] Oct 30 20:13:49.331455 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:13:49.331429 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc701b842_c3d6_4861_bfc2_a4196cd78c01.slice/crio-2de485965be91ac84507f69eef4111bc908ee2bc354cb49fedfdbca4040db3c0 WatchSource:0}: Error finding container 2de485965be91ac84507f69eef4111bc908ee2bc354cb49fedfdbca4040db3c0: Status 404 returned error can't find the container with id 2de485965be91ac84507f69eef4111bc908ee2bc354cb49fedfdbca4040db3c0 Oct 30 20:13:49.869001 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:49.868912 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2m9ng" event={"ID":"c701b842-c3d6-4861-bfc2-a4196cd78c01","Type":"ContainerStarted","Data":"55f6becc4110005ba46b39f027f66437a9d5f58a3b1577cf3944209150e73a09"} Oct 30 20:13:49.869001 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:49.868970 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2m9ng" event={"ID":"c701b842-c3d6-4861-bfc2-a4196cd78c01","Type":"ContainerStarted","Data":"2de485965be91ac84507f69eef4111bc908ee2bc354cb49fedfdbca4040db3c0"} Oct 30 20:13:49.914596 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:49.914546 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2m9ng" podStartSLOduration=1.9145316829999999 podStartE2EDuration="1.914531683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:13:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:13:49.91330062 +0000 UTC m=+1591.326014419" watchObservedRunningTime="2026-10-30 20:13:49.914531683 +0000 UTC m=+1591.327245461" Oct 30 20:13:50.875388 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:50.875351 5088 generic.go:356] "Generic (PLEG): container finished" podID="c701b842-c3d6-4861-bfc2-a4196cd78c01" containerID="55f6becc4110005ba46b39f027f66437a9d5f58a3b1577cf3944209150e73a09" exitCode=0 Oct 30 20:13:50.875786 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:50.875436 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2m9ng" event={"ID":"c701b842-c3d6-4861-bfc2-a4196cd78c01","Type":"ContainerDied","Data":"55f6becc4110005ba46b39f027f66437a9d5f58a3b1577cf3944209150e73a09"} Oct 30 20:13:52.028369 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:52.028344 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2m9ng" Oct 30 20:13:52.163834 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:52.163765 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c701b842-c3d6-4861-bfc2-a4196cd78c01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c701b842-c3d6-4861-bfc2-a4196cd78c01-maas-api-openshift-ca\") pod \"c701b842-c3d6-4861-bfc2-a4196cd78c01\" (UID: \"c701b842-c3d6-4861-bfc2-a4196cd78c01\") " Oct 30 20:13:52.163972 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:52.163855 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c701b842-c3d6-4861-bfc2-a4196cd78c01-kube-api-access-l8bh5\" (UniqueName: \"kubernetes.io/projected/c701b842-c3d6-4861-bfc2-a4196cd78c01-kube-api-access-l8bh5\") pod \"c701b842-c3d6-4861-bfc2-a4196cd78c01\" (UID: \"c701b842-c3d6-4861-bfc2-a4196cd78c01\") " Oct 30 20:13:52.163972 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:52.163902 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c701b842-c3d6-4861-bfc2-a4196cd78c01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c701b842-c3d6-4861-bfc2-a4196cd78c01-maas-api-xks-ca\") pod \"c701b842-c3d6-4861-bfc2-a4196cd78c01\" (UID: \"c701b842-c3d6-4861-bfc2-a4196cd78c01\") " Oct 30 20:13:52.164147 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:52.164121 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c701b842-c3d6-4861-bfc2-a4196cd78c01-maas-api-openshift-ca" pod "c701b842-c3d6-4861-bfc2-a4196cd78c01" (UID: "c701b842-c3d6-4861-bfc2-a4196cd78c01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:13:52.165988 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:52.165958 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c701b842-c3d6-4861-bfc2-a4196cd78c01-kube-api-access-l8bh5" pod "c701b842-c3d6-4861-bfc2-a4196cd78c01" (UID: "c701b842-c3d6-4861-bfc2-a4196cd78c01"). InnerVolumeSpecName "kube-api-access-l8bh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:52.166102 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:52.166075 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c701b842-c3d6-4861-bfc2-a4196cd78c01-maas-api-xks-ca" pod "c701b842-c3d6-4861-bfc2-a4196cd78c01" (UID: "c701b842-c3d6-4861-bfc2-a4196cd78c01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:13:52.264959 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:52.264936 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c701b842-c3d6-4861-bfc2-a4196cd78c01-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:13:52.264959 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:52.264958 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c701b842-c3d6-4861-bfc2-a4196cd78c01-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:13:52.265116 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:52.264967 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8bh5\" (UniqueName: \"kubernetes.io/projected/c701b842-c3d6-4861-bfc2-a4196cd78c01-kube-api-access-l8bh5\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:13:52.436797 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:13:52.436714 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:13:52.889367 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:52.889339 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2m9ng" Oct 30 20:13:52.889538 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:52.889376 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2m9ng" event={"ID":"c701b842-c3d6-4861-bfc2-a4196cd78c01","Type":"ContainerDied","Data":"2de485965be91ac84507f69eef4111bc908ee2bc354cb49fedfdbca4040db3c0"} Oct 30 20:13:52.889538 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:52.889424 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2de485965be91ac84507f69eef4111bc908ee2bc354cb49fedfdbca4040db3c0" Oct 30 20:13:58.060678 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:58.060647 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-475b131d-vbdxt"] Oct 30 20:13:58.075700 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:58.075673 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-475b131d-vbdxt"] Oct 30 20:13:58.739349 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:13:58.739314 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c157f2b-3789-4333-ba93-4920242f6093" path="/var/lib/kubelet/pods/6c157f2b-3789-4333-ba93-4920242f6093/volumes" Oct 30 20:14:00.839480 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:00.839449 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:14:05.729557 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:05.729521 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:14:07.050560 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:07.050529 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd04ef0c-lt96r"] Oct 30 20:14:07.064090 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:07.064064 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd04ef0c-lt96r"] Oct 30 20:14:08.740396 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:08.740366 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c27da71-bbe3-4855-b6aa-d46fb82e42c7" path="/var/lib/kubelet/pods/4c27da71-bbe3-4855-b6aa-d46fb82e42c7/volumes" Oct 30 20:14:12.595961 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:12.595928 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2m9ng"] Oct 30 20:14:12.609530 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:12.609502 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2m9ng"] Oct 30 20:14:12.645462 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:12.645437 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zklw6"] Oct 30 20:14:12.645891 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:12.645876 5088 cpu_manager.go:414] "containerMap: removing container" podUID="c701b842-c3d6-4861-bfc2-a4196cd78c01" containerName="revoke-keys" Oct 30 20:14:12.645941 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:12.645892 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c701b842-c3d6-4861-bfc2-a4196cd78c01" containerName="revoke-keys" Oct 30 20:14:12.646002 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:12.645993 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="c701b842-c3d6-4861-bfc2-a4196cd78c01" containerName="revoke-keys" Oct 30 20:14:12.648889 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:12.648872 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zklw6" Oct 30 20:14:12.667812 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:12.667783 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:14:12.667941 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:12.667801 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:14:12.667941 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:12.667881 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:14:12.667941 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:12.667884 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:14:12.670446 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:12.670425 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zklw6"] Oct 30 20:14:12.738311 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:12.738285 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c701b842-c3d6-4861-bfc2-a4196cd78c01" path="/var/lib/kubelet/pods/c701b842-c3d6-4861-bfc2-a4196cd78c01/volumes" Oct 30 20:14:12.756901 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:12.756876 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73d4afb1-deda-4dfb-b9f0-8958e8a6b913-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zklw6\" (UID: \"73d4afb1-deda-4dfb-b9f0-8958e8a6b913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zklw6" Oct 30 20:14:12.757034 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:12.756999 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73d4afb1-deda-4dfb-b9f0-8958e8a6b913-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zklw6\" (UID: \"73d4afb1-deda-4dfb-b9f0-8958e8a6b913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zklw6" Oct 30 20:14:12.757091 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:12.757069 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n797q\" (UniqueName: \"kubernetes.io/projected/73d4afb1-deda-4dfb-b9f0-8958e8a6b913-kube-api-access-n797q\") pod \"maas-api-revoke-sub-3d4156f1-zklw6\" (UID: \"73d4afb1-deda-4dfb-b9f0-8958e8a6b913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zklw6" Oct 30 20:14:12.857850 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:12.857791 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73d4afb1-deda-4dfb-b9f0-8958e8a6b913-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zklw6\" (UID: \"73d4afb1-deda-4dfb-b9f0-8958e8a6b913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zklw6" Oct 30 20:14:12.857850 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:12.857845 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73d4afb1-deda-4dfb-b9f0-8958e8a6b913-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zklw6\" (UID: \"73d4afb1-deda-4dfb-b9f0-8958e8a6b913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zklw6" Oct 30 20:14:12.858002 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:12.857866 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n797q\" (UniqueName: \"kubernetes.io/projected/73d4afb1-deda-4dfb-b9f0-8958e8a6b913-kube-api-access-n797q\") pod \"maas-api-revoke-sub-3d4156f1-zklw6\" (UID: \"73d4afb1-deda-4dfb-b9f0-8958e8a6b913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zklw6" Oct 30 20:14:12.858515 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:12.858489 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73d4afb1-deda-4dfb-b9f0-8958e8a6b913-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zklw6\" (UID: \"73d4afb1-deda-4dfb-b9f0-8958e8a6b913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zklw6" Oct 30 20:14:12.860113 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:12.860088 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73d4afb1-deda-4dfb-b9f0-8958e8a6b913-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zklw6\" (UID: \"73d4afb1-deda-4dfb-b9f0-8958e8a6b913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zklw6" Oct 30 20:14:12.885705 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:12.885684 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n797q\" (UniqueName: \"kubernetes.io/projected/73d4afb1-deda-4dfb-b9f0-8958e8a6b913-kube-api-access-n797q\") pod \"maas-api-revoke-sub-3d4156f1-zklw6\" (UID: \"73d4afb1-deda-4dfb-b9f0-8958e8a6b913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zklw6" Oct 30 20:14:12.962502 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:12.962480 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zklw6" Oct 30 20:14:13.266165 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:13.266136 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zklw6"] Oct 30 20:14:13.268336 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:14:13.268303 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73d4afb1_deda_4dfb_b9f0_8958e8a6b913.slice/crio-59afdc1ec499410f2a044a98c8caa5f4a3c4496f775fe227dc856e69017f4fcc WatchSource:0}: Error finding container 59afdc1ec499410f2a044a98c8caa5f4a3c4496f775fe227dc856e69017f4fcc: Status 404 returned error can't find the container with id 59afdc1ec499410f2a044a98c8caa5f4a3c4496f775fe227dc856e69017f4fcc Oct 30 20:14:14.032135 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:14.032108 5088 generic.go:356] "Generic (PLEG): container finished" podID="73d4afb1-deda-4dfb-b9f0-8958e8a6b913" containerID="398fb7cfc4e31f8a363dc1d552c0ab7097fd7b3d43225b784fcdc30d40535840" exitCode=0 Oct 30 20:14:14.032472 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:14.032190 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zklw6" event={"ID":"73d4afb1-deda-4dfb-b9f0-8958e8a6b913","Type":"ContainerDied","Data":"398fb7cfc4e31f8a363dc1d552c0ab7097fd7b3d43225b784fcdc30d40535840"} Oct 30 20:14:14.032472 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:14.032254 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zklw6" event={"ID":"73d4afb1-deda-4dfb-b9f0-8958e8a6b913","Type":"ContainerStarted","Data":"59afdc1ec499410f2a044a98c8caa5f4a3c4496f775fe227dc856e69017f4fcc"} Oct 30 20:14:15.186564 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:15.186527 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zklw6" Oct 30 20:14:15.278814 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:15.278784 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73d4afb1-deda-4dfb-b9f0-8958e8a6b913-kube-api-access-n797q\" (UniqueName: \"kubernetes.io/projected/73d4afb1-deda-4dfb-b9f0-8958e8a6b913-kube-api-access-n797q\") pod \"73d4afb1-deda-4dfb-b9f0-8958e8a6b913\" (UID: \"73d4afb1-deda-4dfb-b9f0-8958e8a6b913\") " Oct 30 20:14:15.278978 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:15.278868 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73d4afb1-deda-4dfb-b9f0-8958e8a6b913-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73d4afb1-deda-4dfb-b9f0-8958e8a6b913-maas-api-openshift-ca\") pod \"73d4afb1-deda-4dfb-b9f0-8958e8a6b913\" (UID: \"73d4afb1-deda-4dfb-b9f0-8958e8a6b913\") " Oct 30 20:14:15.278978 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:15.278895 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73d4afb1-deda-4dfb-b9f0-8958e8a6b913-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73d4afb1-deda-4dfb-b9f0-8958e8a6b913-maas-api-xks-ca\") pod \"73d4afb1-deda-4dfb-b9f0-8958e8a6b913\" (UID: \"73d4afb1-deda-4dfb-b9f0-8958e8a6b913\") " Oct 30 20:14:15.279286 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:15.279262 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73d4afb1-deda-4dfb-b9f0-8958e8a6b913-maas-api-openshift-ca" pod "73d4afb1-deda-4dfb-b9f0-8958e8a6b913" (UID: "73d4afb1-deda-4dfb-b9f0-8958e8a6b913"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:14:15.280852 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:15.280824 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73d4afb1-deda-4dfb-b9f0-8958e8a6b913-maas-api-xks-ca" pod "73d4afb1-deda-4dfb-b9f0-8958e8a6b913" (UID: "73d4afb1-deda-4dfb-b9f0-8958e8a6b913"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:14:15.280947 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:15.280882 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73d4afb1-deda-4dfb-b9f0-8958e8a6b913-kube-api-access-n797q" pod "73d4afb1-deda-4dfb-b9f0-8958e8a6b913" (UID: "73d4afb1-deda-4dfb-b9f0-8958e8a6b913"). InnerVolumeSpecName "kube-api-access-n797q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:15.379741 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:15.379683 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73d4afb1-deda-4dfb-b9f0-8958e8a6b913-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:14:15.379741 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:15.379707 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73d4afb1-deda-4dfb-b9f0-8958e8a6b913-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:14:15.379741 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:15.379718 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n797q\" (UniqueName: \"kubernetes.io/projected/73d4afb1-deda-4dfb-b9f0-8958e8a6b913-kube-api-access-n797q\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:14:16.045094 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:16.045067 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zklw6" Oct 30 20:14:16.045283 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:16.045056 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zklw6" event={"ID":"73d4afb1-deda-4dfb-b9f0-8958e8a6b913","Type":"ContainerDied","Data":"59afdc1ec499410f2a044a98c8caa5f4a3c4496f775fe227dc856e69017f4fcc"} Oct 30 20:14:16.045283 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:16.045188 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59afdc1ec499410f2a044a98c8caa5f4a3c4496f775fe227dc856e69017f4fcc" Oct 30 20:14:22.436685 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:14:22.436649 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:14:24.566087 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:24.566053 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:14:27.630008 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:27.629978 5088 scope.go:122] "RemoveContainer" containerID="725d8881240d39c401290b472f11172ec5422a72d839bd432b4511d02f60a0ba" Oct 30 20:14:27.641742 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:27.641721 5088 scope.go:122] "RemoveContainer" containerID="ab73a40d0355bf1339ca35b0625d6f80b3b92822a256bbae2f71117d46d72940" Oct 30 20:14:27.650368 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:27.650343 5088 scope.go:122] "RemoveContainer" containerID="1fa04071a7cfb0edc7dcfaee6d0d27db5d0ee935ee8812a7b7dc5d0d4f1612ed" Oct 30 20:14:27.658333 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:27.658317 5088 scope.go:122] "RemoveContainer" containerID="f68c3a10ea537704b7cf0aed2ccae53e6226800a25be8dd807b3f3ddc7330cbb" Oct 30 20:14:29.546535 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:29.546500 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:14:39.856508 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:39.856475 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a989da-6658db8qkp"] Oct 30 20:14:39.856999 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:39.856981 5088 cpu_manager.go:414] "containerMap: removing container" podUID="73d4afb1-deda-4dfb-b9f0-8958e8a6b913" containerName="revoke-keys" Oct 30 20:14:39.857049 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:39.857001 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73d4afb1-deda-4dfb-b9f0-8958e8a6b913" containerName="revoke-keys" Oct 30 20:14:39.857087 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:39.857077 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="73d4afb1-deda-4dfb-b9f0-8958e8a6b913" containerName="revoke-keys" Oct 30 20:14:39.862155 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:39.862136 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a989da-6658db8qkp" Oct 30 20:14:39.882263 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:39.882234 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a989da-6658db8qkp"] Oct 30 20:14:39.892381 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:39.892354 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-a989da\"" Oct 30 20:14:39.893780 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:39.893755 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-7582cb13-dockercfg-pgdt5\"" Oct 30 20:14:39.968895 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:39.968855 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a989da-67c48bfc7lvnq6"] Oct 30 20:14:39.980983 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:39.980951 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a989da-67c48bfc7lvnq6" Oct 30 20:14:39.990364 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:39.987920 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a989da-67c48bfc7lvnq6"] Oct 30 20:14:40.019368 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:40.019333 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5416142f-771f-438e-910b-56163ea335ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-a989da-6658db8qkp\" (UID: \"5416142f-771f-438e-910b-56163ea335ae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a989da-6658db8qkp" Oct 30 20:14:40.019560 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:40.019535 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57kn8\" (UniqueName: \"kubernetes.io/projected/5416142f-771f-438e-910b-56163ea335ae-kube-api-access-57kn8\") pod \"payload-pre-processing-e2e-worker-single-main-a989da-6658db8qkp\" (UID: \"5416142f-771f-438e-910b-56163ea335ae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a989da-6658db8qkp" Oct 30 20:14:40.121167 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:40.121088 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57kn8\" (UniqueName: \"kubernetes.io/projected/5416142f-771f-438e-910b-56163ea335ae-kube-api-access-57kn8\") pod \"payload-pre-processing-e2e-worker-single-main-a989da-6658db8qkp\" (UID: \"5416142f-771f-438e-910b-56163ea335ae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a989da-6658db8qkp" Oct 30 20:14:40.121167 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:40.121162 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5416142f-771f-438e-910b-56163ea335ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-a989da-6658db8qkp\" (UID: \"5416142f-771f-438e-910b-56163ea335ae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a989da-6658db8qkp" Oct 30 20:14:40.121393 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:40.121240 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q6s7\" (UniqueName: \"kubernetes.io/projected/1fc1f63b-1880-484a-85c9-8b94119e81c6-kube-api-access-6q6s7\") pod \"payload-processing-e2e-worker-single-main-a989da-67c48bfc7lvnq6\" (UID: \"1fc1f63b-1880-484a-85c9-8b94119e81c6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a989da-67c48bfc7lvnq6" Oct 30 20:14:40.121393 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:40.121269 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fc1f63b-1880-484a-85c9-8b94119e81c6-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-a989da-67c48bfc7lvnq6\" (UID: \"1fc1f63b-1880-484a-85c9-8b94119e81c6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a989da-67c48bfc7lvnq6" Oct 30 20:14:40.121891 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:40.121871 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5416142f-771f-438e-910b-56163ea335ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-a989da-6658db8qkp\" (UID: \"5416142f-771f-438e-910b-56163ea335ae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a989da-6658db8qkp" Oct 30 20:14:40.147336 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:40.147308 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57kn8\" (UniqueName: \"kubernetes.io/projected/5416142f-771f-438e-910b-56163ea335ae-kube-api-access-57kn8\") pod \"payload-pre-processing-e2e-worker-single-main-a989da-6658db8qkp\" (UID: \"5416142f-771f-438e-910b-56163ea335ae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a989da-6658db8qkp" Oct 30 20:14:40.175399 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:40.175379 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a989da-6658db8qkp" Oct 30 20:14:40.222233 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:40.222194 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q6s7\" (UniqueName: \"kubernetes.io/projected/1fc1f63b-1880-484a-85c9-8b94119e81c6-kube-api-access-6q6s7\") pod \"payload-processing-e2e-worker-single-main-a989da-67c48bfc7lvnq6\" (UID: \"1fc1f63b-1880-484a-85c9-8b94119e81c6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a989da-67c48bfc7lvnq6" Oct 30 20:14:40.222800 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:40.222248 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fc1f63b-1880-484a-85c9-8b94119e81c6-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-a989da-67c48bfc7lvnq6\" (UID: \"1fc1f63b-1880-484a-85c9-8b94119e81c6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a989da-67c48bfc7lvnq6" Oct 30 20:14:40.223297 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:40.223268 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fc1f63b-1880-484a-85c9-8b94119e81c6-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-a989da-67c48bfc7lvnq6\" (UID: \"1fc1f63b-1880-484a-85c9-8b94119e81c6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a989da-67c48bfc7lvnq6" Oct 30 20:14:40.250517 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:40.250489 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q6s7\" (UniqueName: \"kubernetes.io/projected/1fc1f63b-1880-484a-85c9-8b94119e81c6-kube-api-access-6q6s7\") pod \"payload-processing-e2e-worker-single-main-a989da-67c48bfc7lvnq6\" (UID: \"1fc1f63b-1880-484a-85c9-8b94119e81c6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a989da-67c48bfc7lvnq6" Oct 30 20:14:40.305079 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:40.305052 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a989da-67c48bfc7lvnq6" Oct 30 20:14:40.479168 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:40.479136 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a989da-6658db8qkp"] Oct 30 20:14:40.479753 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:14:40.479729 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5416142f_771f_438e_910b_56163ea335ae.slice/crio-d8a6880142b16b17181d8a1f1832287954668eb9c6897a7b1a022ed766d303f2 WatchSource:0}: Error finding container d8a6880142b16b17181d8a1f1832287954668eb9c6897a7b1a022ed766d303f2: Status 404 returned error can't find the container with id d8a6880142b16b17181d8a1f1832287954668eb9c6897a7b1a022ed766d303f2 Oct 30 20:14:40.605585 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:40.605558 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a989da-67c48bfc7lvnq6"] Oct 30 20:14:40.606391 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:14:40.606368 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fc1f63b_1880_484a_85c9_8b94119e81c6.slice/crio-1bc2c7b8db82b66fb48cf5a5861f7450438776c68828f77aeb309e416f2a1325 WatchSource:0}: Error finding container 1bc2c7b8db82b66fb48cf5a5861f7450438776c68828f77aeb309e416f2a1325: Status 404 returned error can't find the container with id 1bc2c7b8db82b66fb48cf5a5861f7450438776c68828f77aeb309e416f2a1325 Oct 30 20:14:41.196514 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:41.196473 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a989da-6658db8qkp" event={"ID":"5416142f-771f-438e-910b-56163ea335ae","Type":"ContainerStarted","Data":"bc6c68ce0e1c96259c155924fad5cd98b2535a24e1e82576fd792ae8a3130970"} Oct 30 20:14:41.197030 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:41.196529 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a989da-6658db8qkp" event={"ID":"5416142f-771f-438e-910b-56163ea335ae","Type":"ContainerStarted","Data":"d8a6880142b16b17181d8a1f1832287954668eb9c6897a7b1a022ed766d303f2"} Oct 30 20:14:41.197030 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:41.196581 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a989da-6658db8qkp" Oct 30 20:14:41.198320 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:41.198295 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a989da-67c48bfc7lvnq6" event={"ID":"1fc1f63b-1880-484a-85c9-8b94119e81c6","Type":"ContainerStarted","Data":"831ab5225d82bce2ab4703da906d266aa4d153b9cf355420d8c59fc068185629"} Oct 30 20:14:41.198437 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:41.198350 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a989da-67c48bfc7lvnq6" event={"ID":"1fc1f63b-1880-484a-85c9-8b94119e81c6","Type":"ContainerStarted","Data":"1bc2c7b8db82b66fb48cf5a5861f7450438776c68828f77aeb309e416f2a1325"} Oct 30 20:14:41.198437 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:41.198381 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a989da-67c48bfc7lvnq6" Oct 30 20:14:41.244545 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:41.244484 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a989da-6658db8qkp" podStartSLOduration=2.244469603 podStartE2EDuration="2.244469603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:41.242706735 +0000 UTC m=+1642.655420537" watchObservedRunningTime="2026-10-30 20:14:41.244469603 +0000 UTC m=+1642.657183468" Oct 30 20:14:41.294029 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:41.293976 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a989da-67c48bfc7lvnq6" podStartSLOduration=2.293962515 podStartE2EDuration="2.293962515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:41.293068211 +0000 UTC m=+1642.705782005" watchObservedRunningTime="2026-10-30 20:14:41.293962515 +0000 UTC m=+1642.706676293" Oct 30 20:14:43.172122 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:43.172085 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:14:52.207245 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:52.207210 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a989da-6658db8qkp" Oct 30 20:14:52.207799 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:52.207535 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a989da-67c48bfc7lvnq6" Oct 30 20:14:52.441013 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:14:52.440977 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:14:58.280807 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:14:58.280770 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:15:00.148863 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:00.148825 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-rhtch"] Oct 30 20:15:00.153862 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:00.153841 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-rhtch" Oct 30 20:15:00.181966 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:00.181927 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:15:00.194322 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:00.194286 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:15:00.194526 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:00.194390 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:15:00.198762 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:00.198740 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-rhtch"] Oct 30 20:15:00.232485 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:00.232457 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llhk5\" (UniqueName: \"kubernetes.io/projected/33291efb-da2a-441d-88ba-40cf29977cb7-kube-api-access-llhk5\") pod \"maas-api-key-cleanup-29889855-rhtch\" (UID: \"33291efb-da2a-441d-88ba-40cf29977cb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-rhtch" Oct 30 20:15:00.232602 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:00.232584 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33291efb-da2a-441d-88ba-40cf29977cb7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889855-rhtch\" (UID: \"33291efb-da2a-441d-88ba-40cf29977cb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-rhtch" Oct 30 20:15:00.333604 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:00.333571 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33291efb-da2a-441d-88ba-40cf29977cb7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889855-rhtch\" (UID: \"33291efb-da2a-441d-88ba-40cf29977cb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-rhtch" Oct 30 20:15:00.333778 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:00.333676 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llhk5\" (UniqueName: \"kubernetes.io/projected/33291efb-da2a-441d-88ba-40cf29977cb7-kube-api-access-llhk5\") pod \"maas-api-key-cleanup-29889855-rhtch\" (UID: \"33291efb-da2a-441d-88ba-40cf29977cb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-rhtch" Oct 30 20:15:00.334314 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:00.334290 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33291efb-da2a-441d-88ba-40cf29977cb7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889855-rhtch\" (UID: \"33291efb-da2a-441d-88ba-40cf29977cb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-rhtch" Oct 30 20:15:00.360667 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:00.360635 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llhk5\" (UniqueName: \"kubernetes.io/projected/33291efb-da2a-441d-88ba-40cf29977cb7-kube-api-access-llhk5\") pod \"maas-api-key-cleanup-29889855-rhtch\" (UID: \"33291efb-da2a-441d-88ba-40cf29977cb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-rhtch" Oct 30 20:15:00.467894 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:00.467804 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-rhtch" Oct 30 20:15:00.774876 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:00.774843 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-rhtch"] Oct 30 20:15:00.776224 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:15:00.776198 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33291efb_da2a_441d_88ba_40cf29977cb7.slice/crio-b2c9e933e4f6018e24eaa82f08567557cd5056defb9449dd941a01e000626b13 WatchSource:0}: Error finding container b2c9e933e4f6018e24eaa82f08567557cd5056defb9449dd941a01e000626b13: Status 404 returned error can't find the container with id b2c9e933e4f6018e24eaa82f08567557cd5056defb9449dd941a01e000626b13 Oct 30 20:15:01.039585 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:01.039496 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fb6dw"] Oct 30 20:15:01.052039 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:01.051994 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fb6dw"] Oct 30 20:15:01.331737 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:01.331635 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-rhtch" event={"ID":"33291efb-da2a-441d-88ba-40cf29977cb7","Type":"ContainerStarted","Data":"f92f95162c9bba5acc40a736a070fbc8d226ff21e248e7320660b8b9d567d1e3"} Oct 30 20:15:01.331737 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:01.331701 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-rhtch" event={"ID":"33291efb-da2a-441d-88ba-40cf29977cb7","Type":"ContainerStarted","Data":"b2c9e933e4f6018e24eaa82f08567557cd5056defb9449dd941a01e000626b13"} Oct 30 20:15:01.378275 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:01.378224 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-rhtch" podStartSLOduration=1.3782116580000001 podStartE2EDuration="1.378211658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:01.375827157 +0000 UTC m=+1662.788540941" watchObservedRunningTime="2026-10-30 20:15:01.378211658 +0000 UTC m=+1662.790925433" Oct 30 20:15:02.339465 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:02.339429 5088 generic.go:356] "Generic (PLEG): container finished" podID="33291efb-da2a-441d-88ba-40cf29977cb7" containerID="f92f95162c9bba5acc40a736a070fbc8d226ff21e248e7320660b8b9d567d1e3" exitCode=60 Oct 30 20:15:02.339877 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:02.339497 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-rhtch" event={"ID":"33291efb-da2a-441d-88ba-40cf29977cb7","Type":"ContainerDied","Data":"f92f95162c9bba5acc40a736a070fbc8d226ff21e248e7320660b8b9d567d1e3"} Oct 30 20:15:02.339877 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:02.339780 5088 scope.go:122] "RemoveContainer" containerID="f92f95162c9bba5acc40a736a070fbc8d226ff21e248e7320660b8b9d567d1e3" Oct 30 20:15:02.738627 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:02.738590 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1be427d5-7034-4669-bc89-4986e9fe3ceb" path="/var/lib/kubelet/pods/1be427d5-7034-4669-bc89-4986e9fe3ceb/volumes" Oct 30 20:15:03.347065 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:03.347028 5088 generic.go:356] "Generic (PLEG): container finished" podID="33291efb-da2a-441d-88ba-40cf29977cb7" containerID="058eec83ffa3474a6ebb50471c37be8d5f72074b202dd6242146f7888f24e311" exitCode=60 Oct 30 20:15:03.347500 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:03.347093 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-rhtch" event={"ID":"33291efb-da2a-441d-88ba-40cf29977cb7","Type":"ContainerDied","Data":"058eec83ffa3474a6ebb50471c37be8d5f72074b202dd6242146f7888f24e311"} Oct 30 20:15:03.347500 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:03.347133 5088 scope.go:122] "RemoveContainer" containerID="f92f95162c9bba5acc40a736a070fbc8d226ff21e248e7320660b8b9d567d1e3" Oct 30 20:15:03.347500 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:03.347378 5088 scope.go:122] "RemoveContainer" containerID="058eec83ffa3474a6ebb50471c37be8d5f72074b202dd6242146f7888f24e311" Oct 30 20:15:03.347694 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:15:03.347673 5088 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889855-rhtch_odh-ai-gateway-infra(33291efb-da2a-441d-88ba-40cf29977cb7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-rhtch" podUID="33291efb-da2a-441d-88ba-40cf29977cb7" Oct 30 20:15:04.354564 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:04.354535 5088 scope.go:122] "RemoveContainer" containerID="058eec83ffa3474a6ebb50471c37be8d5f72074b202dd6242146f7888f24e311" Oct 30 20:15:04.354972 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:15:04.354829 5088 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889855-rhtch_odh-ai-gateway-infra(33291efb-da2a-441d-88ba-40cf29977cb7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-rhtch" podUID="33291efb-da2a-441d-88ba-40cf29977cb7" Oct 30 20:15:12.756449 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:12.756416 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:15:15.732401 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:15.732368 5088 scope.go:122] "RemoveContainer" containerID="058eec83ffa3474a6ebb50471c37be8d5f72074b202dd6242146f7888f24e311" Oct 30 20:15:16.425224 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:16.425127 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-rhtch" event={"ID":"33291efb-da2a-441d-88ba-40cf29977cb7","Type":"ContainerStarted","Data":"6a74a3cb945102c55048a94a082d9ec01b0399d635211bf29f7f59f9807bc229"} Oct 30 20:15:17.433935 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:17.433902 5088 generic.go:356] "Generic (PLEG): container finished" podID="33291efb-da2a-441d-88ba-40cf29977cb7" containerID="6a74a3cb945102c55048a94a082d9ec01b0399d635211bf29f7f59f9807bc229" exitCode=60 Oct 30 20:15:17.434373 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:17.433975 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-rhtch" event={"ID":"33291efb-da2a-441d-88ba-40cf29977cb7","Type":"ContainerDied","Data":"6a74a3cb945102c55048a94a082d9ec01b0399d635211bf29f7f59f9807bc229"} Oct 30 20:15:17.434373 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:17.434040 5088 scope.go:122] "RemoveContainer" containerID="058eec83ffa3474a6ebb50471c37be8d5f72074b202dd6242146f7888f24e311" Oct 30 20:15:17.434373 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:17.434361 5088 scope.go:122] "RemoveContainer" containerID="6a74a3cb945102c55048a94a082d9ec01b0399d635211bf29f7f59f9807bc229" Oct 30 20:15:17.434677 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:15:17.434652 5088 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889855-rhtch_odh-ai-gateway-infra(33291efb-da2a-441d-88ba-40cf29977cb7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-rhtch" podUID="33291efb-da2a-441d-88ba-40cf29977cb7" Oct 30 20:15:17.497363 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:17.497324 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-rhtch"] Oct 30 20:15:18.589609 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:18.589579 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-rhtch" Oct 30 20:15:18.621694 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:18.621666 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33291efb-da2a-441d-88ba-40cf29977cb7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33291efb-da2a-441d-88ba-40cf29977cb7-maas-api-service-ca\") pod \"33291efb-da2a-441d-88ba-40cf29977cb7\" (UID: \"33291efb-da2a-441d-88ba-40cf29977cb7\") " Oct 30 20:15:18.621856 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:18.621725 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33291efb-da2a-441d-88ba-40cf29977cb7-kube-api-access-llhk5\" (UniqueName: \"kubernetes.io/projected/33291efb-da2a-441d-88ba-40cf29977cb7-kube-api-access-llhk5\") pod \"33291efb-da2a-441d-88ba-40cf29977cb7\" (UID: \"33291efb-da2a-441d-88ba-40cf29977cb7\") " Oct 30 20:15:18.622022 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:18.622002 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33291efb-da2a-441d-88ba-40cf29977cb7-maas-api-service-ca" pod "33291efb-da2a-441d-88ba-40cf29977cb7" (UID: "33291efb-da2a-441d-88ba-40cf29977cb7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:18.623894 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:18.623867 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33291efb-da2a-441d-88ba-40cf29977cb7-kube-api-access-llhk5" pod "33291efb-da2a-441d-88ba-40cf29977cb7" (UID: "33291efb-da2a-441d-88ba-40cf29977cb7"). InnerVolumeSpecName "kube-api-access-llhk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:18.723197 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:18.723112 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33291efb-da2a-441d-88ba-40cf29977cb7-maas-api-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:15:18.723197 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:18.723146 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llhk5\" (UniqueName: \"kubernetes.io/projected/33291efb-da2a-441d-88ba-40cf29977cb7-kube-api-access-llhk5\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:15:18.858672 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:18.858628 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7073189-wxjzb"] Oct 30 20:15:18.859112 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:18.859099 5088 cpu_manager.go:414] "containerMap: removing container" podUID="33291efb-da2a-441d-88ba-40cf29977cb7" containerName="cleanup" Oct 30 20:15:18.859159 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:18.859114 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33291efb-da2a-441d-88ba-40cf29977cb7" containerName="cleanup" Oct 30 20:15:18.859159 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:18.859124 5088 cpu_manager.go:414] "containerMap: removing container" podUID="33291efb-da2a-441d-88ba-40cf29977cb7" containerName="cleanup" Oct 30 20:15:18.859159 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:18.859129 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33291efb-da2a-441d-88ba-40cf29977cb7" containerName="cleanup" Oct 30 20:15:18.859253 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:18.859219 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="33291efb-da2a-441d-88ba-40cf29977cb7" containerName="cleanup" Oct 30 20:15:18.859253 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:18.859236 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="33291efb-da2a-441d-88ba-40cf29977cb7" containerName="cleanup" Oct 30 20:15:18.862487 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:18.862471 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7073189-wxjzb" Oct 30 20:15:18.880804 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:18.880780 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:15:18.882759 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:18.882735 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7073189-wxjzb"] Oct 30 20:15:18.924858 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:18.924825 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6zc2\" (UniqueName: \"kubernetes.io/projected/421a34a5-84d2-499a-9a30-e3d391c3ab45-kube-api-access-w6zc2\") pod \"maas-api-revoke-sub-c7073189-wxjzb\" (UID: \"421a34a5-84d2-499a-9a30-e3d391c3ab45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7073189-wxjzb" Oct 30 20:15:18.925046 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:18.924962 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/421a34a5-84d2-499a-9a30-e3d391c3ab45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7073189-wxjzb\" (UID: \"421a34a5-84d2-499a-9a30-e3d391c3ab45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7073189-wxjzb" Oct 30 20:15:18.925046 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:18.925008 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/421a34a5-84d2-499a-9a30-e3d391c3ab45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7073189-wxjzb\" (UID: \"421a34a5-84d2-499a-9a30-e3d391c3ab45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7073189-wxjzb" Oct 30 20:15:19.026269 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:19.026233 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6zc2\" (UniqueName: \"kubernetes.io/projected/421a34a5-84d2-499a-9a30-e3d391c3ab45-kube-api-access-w6zc2\") pod \"maas-api-revoke-sub-c7073189-wxjzb\" (UID: \"421a34a5-84d2-499a-9a30-e3d391c3ab45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7073189-wxjzb" Oct 30 20:15:19.026471 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:19.026352 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/421a34a5-84d2-499a-9a30-e3d391c3ab45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7073189-wxjzb\" (UID: \"421a34a5-84d2-499a-9a30-e3d391c3ab45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7073189-wxjzb" Oct 30 20:15:19.026471 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:19.026412 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/421a34a5-84d2-499a-9a30-e3d391c3ab45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7073189-wxjzb\" (UID: \"421a34a5-84d2-499a-9a30-e3d391c3ab45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7073189-wxjzb" Oct 30 20:15:19.027255 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:19.027230 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/421a34a5-84d2-499a-9a30-e3d391c3ab45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7073189-wxjzb\" (UID: \"421a34a5-84d2-499a-9a30-e3d391c3ab45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7073189-wxjzb" Oct 30 20:15:19.042847 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:19.042818 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:15:19.049191 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:19.049160 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/421a34a5-84d2-499a-9a30-e3d391c3ab45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7073189-wxjzb\" (UID: \"421a34a5-84d2-499a-9a30-e3d391c3ab45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7073189-wxjzb" Oct 30 20:15:19.054460 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:19.054429 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6zc2\" (UniqueName: \"kubernetes.io/projected/421a34a5-84d2-499a-9a30-e3d391c3ab45-kube-api-access-w6zc2\") pod \"maas-api-revoke-sub-c7073189-wxjzb\" (UID: \"421a34a5-84d2-499a-9a30-e3d391c3ab45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7073189-wxjzb" Oct 30 20:15:19.176999 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:19.176953 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7073189-wxjzb" Oct 30 20:15:19.450245 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:19.450130 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-rhtch" Oct 30 20:15:19.450245 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:19.450142 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-rhtch" event={"ID":"33291efb-da2a-441d-88ba-40cf29977cb7","Type":"ContainerDied","Data":"b2c9e933e4f6018e24eaa82f08567557cd5056defb9449dd941a01e000626b13"} Oct 30 20:15:19.450505 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:19.450287 5088 scope.go:122] "RemoveContainer" containerID="6a74a3cb945102c55048a94a082d9ec01b0399d635211bf29f7f59f9807bc229" Oct 30 20:15:19.489670 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:19.489572 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7073189-wxjzb"] Oct 30 20:15:19.491320 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:15:19.491286 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod421a34a5_84d2_499a_9a30_e3d391c3ab45.slice/crio-e066071050ad4d9441cefc020ccaddb4cd0cd567c2c258cb24b1b14043c18650 WatchSource:0}: Error finding container e066071050ad4d9441cefc020ccaddb4cd0cd567c2c258cb24b1b14043c18650: Status 404 returned error can't find the container with id e066071050ad4d9441cefc020ccaddb4cd0cd567c2c258cb24b1b14043c18650 Oct 30 20:15:19.514288 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:19.514239 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-rhtch"] Oct 30 20:15:19.521696 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:19.521639 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-rhtch"] Oct 30 20:15:20.459198 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:20.459157 5088 generic.go:356] "Generic (PLEG): container finished" podID="421a34a5-84d2-499a-9a30-e3d391c3ab45" containerID="74bf49655ac0e2012136c0e50202ce8066c1c5562ef1f5c2afdb6256f52b1de3" exitCode=0 Oct 30 20:15:20.459607 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:20.459237 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7073189-wxjzb" event={"ID":"421a34a5-84d2-499a-9a30-e3d391c3ab45","Type":"ContainerDied","Data":"74bf49655ac0e2012136c0e50202ce8066c1c5562ef1f5c2afdb6256f52b1de3"} Oct 30 20:15:20.459607 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:20.459300 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7073189-wxjzb" event={"ID":"421a34a5-84d2-499a-9a30-e3d391c3ab45","Type":"ContainerStarted","Data":"e066071050ad4d9441cefc020ccaddb4cd0cd567c2c258cb24b1b14043c18650"} Oct 30 20:15:20.739382 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:20.739347 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33291efb-da2a-441d-88ba-40cf29977cb7" path="/var/lib/kubelet/pods/33291efb-da2a-441d-88ba-40cf29977cb7/volumes" Oct 30 20:15:21.608623 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:21.608595 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7073189-wxjzb" Oct 30 20:15:21.655082 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:21.655055 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/421a34a5-84d2-499a-9a30-e3d391c3ab45-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/421a34a5-84d2-499a-9a30-e3d391c3ab45-maas-api-xks-ca\") pod \"421a34a5-84d2-499a-9a30-e3d391c3ab45\" (UID: \"421a34a5-84d2-499a-9a30-e3d391c3ab45\") " Oct 30 20:15:21.655266 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:21.655107 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/421a34a5-84d2-499a-9a30-e3d391c3ab45-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/421a34a5-84d2-499a-9a30-e3d391c3ab45-maas-api-openshift-ca\") pod \"421a34a5-84d2-499a-9a30-e3d391c3ab45\" (UID: \"421a34a5-84d2-499a-9a30-e3d391c3ab45\") " Oct 30 20:15:21.655266 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:21.655173 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/421a34a5-84d2-499a-9a30-e3d391c3ab45-kube-api-access-w6zc2\" (UniqueName: \"kubernetes.io/projected/421a34a5-84d2-499a-9a30-e3d391c3ab45-kube-api-access-w6zc2\") pod \"421a34a5-84d2-499a-9a30-e3d391c3ab45\" (UID: \"421a34a5-84d2-499a-9a30-e3d391c3ab45\") " Oct 30 20:15:21.655567 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:21.655540 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/421a34a5-84d2-499a-9a30-e3d391c3ab45-maas-api-openshift-ca" pod "421a34a5-84d2-499a-9a30-e3d391c3ab45" (UID: "421a34a5-84d2-499a-9a30-e3d391c3ab45"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:21.657547 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:21.657521 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/421a34a5-84d2-499a-9a30-e3d391c3ab45-maas-api-xks-ca" pod "421a34a5-84d2-499a-9a30-e3d391c3ab45" (UID: "421a34a5-84d2-499a-9a30-e3d391c3ab45"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:15:21.657676 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:21.657653 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/421a34a5-84d2-499a-9a30-e3d391c3ab45-kube-api-access-w6zc2" pod "421a34a5-84d2-499a-9a30-e3d391c3ab45" (UID: "421a34a5-84d2-499a-9a30-e3d391c3ab45"). InnerVolumeSpecName "kube-api-access-w6zc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:21.756523 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:21.756491 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/421a34a5-84d2-499a-9a30-e3d391c3ab45-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:15:21.756523 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:21.756520 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/421a34a5-84d2-499a-9a30-e3d391c3ab45-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:15:21.756523 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:21.756530 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6zc2\" (UniqueName: \"kubernetes.io/projected/421a34a5-84d2-499a-9a30-e3d391c3ab45-kube-api-access-w6zc2\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:15:22.437405 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:15:22.437368 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:15:22.477525 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:22.477487 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7073189-wxjzb" event={"ID":"421a34a5-84d2-499a-9a30-e3d391c3ab45","Type":"ContainerDied","Data":"e066071050ad4d9441cefc020ccaddb4cd0cd567c2c258cb24b1b14043c18650"} Oct 30 20:15:22.477525 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:22.477510 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7073189-wxjzb" Oct 30 20:15:22.477835 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:22.477568 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e066071050ad4d9441cefc020ccaddb4cd0cd567c2c258cb24b1b14043c18650" Oct 30 20:15:27.697660 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:27.697571 5088 scope.go:122] "RemoveContainer" containerID="f2793a1a24f84756d52de31e4f03449dfd4ea04b92b1177378505ee6b83e6dd0" Oct 30 20:15:27.710284 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:27.710258 5088 scope.go:122] "RemoveContainer" containerID="2f012f570ac80688faf6c9aa785fdab95bdac1c70cf4e79fdc31d380b24e0130" Oct 30 20:15:27.721894 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:27.721853 5088 scope.go:122] "RemoveContainer" containerID="f98887d568494e01a9837fcaa2ce0d347dbc14bbea23527a601f5d8f8403c522" Oct 30 20:15:27.733933 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:27.733897 5088 scope.go:122] "RemoveContainer" containerID="bdce559f15a9ddd71a156d312b953b301abbec6193968167c3875a8224e1244b" Oct 30 20:15:30.156045 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:30.156008 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a989da-v8v79"] Oct 30 20:15:30.156498 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:30.156461 5088 cpu_manager.go:414] "containerMap: removing container" podUID="421a34a5-84d2-499a-9a30-e3d391c3ab45" containerName="revoke-keys" Oct 30 20:15:30.156498 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:30.156473 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="421a34a5-84d2-499a-9a30-e3d391c3ab45" containerName="revoke-keys" Oct 30 20:15:30.156586 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:30.156510 5088 cpu_manager.go:414] "containerMap: removing container" podUID="33291efb-da2a-441d-88ba-40cf29977cb7" containerName="cleanup" Oct 30 20:15:30.156586 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:30.156515 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33291efb-da2a-441d-88ba-40cf29977cb7" containerName="cleanup" Oct 30 20:15:30.156586 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:30.156580 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="421a34a5-84d2-499a-9a30-e3d391c3ab45" containerName="revoke-keys" Oct 30 20:15:30.156710 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:30.156597 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="33291efb-da2a-441d-88ba-40cf29977cb7" containerName="cleanup" Oct 30 20:15:30.161498 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:30.161475 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a989da-v8v79" Oct 30 20:15:30.180202 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:30.180177 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a989da-v8v79"] Oct 30 20:15:30.190571 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:30.190539 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:15:30.190762 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:30.190539 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:15:30.190762 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:30.190539 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:15:30.191743 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:30.191722 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:15:30.252322 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:30.252284 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21eb743f-41cd-425c-94e6-06109aa776b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a989da-v8v79\" (UID: \"21eb743f-41cd-425c-94e6-06109aa776b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a989da-v8v79" Oct 30 20:15:30.252524 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:30.252428 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21eb743f-41cd-425c-94e6-06109aa776b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a989da-v8v79\" (UID: \"21eb743f-41cd-425c-94e6-06109aa776b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a989da-v8v79" Oct 30 20:15:30.252524 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:30.252456 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2c8w\" (UniqueName: \"kubernetes.io/projected/21eb743f-41cd-425c-94e6-06109aa776b4-kube-api-access-d2c8w\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a989da-v8v79\" (UID: \"21eb743f-41cd-425c-94e6-06109aa776b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a989da-v8v79" Oct 30 20:15:30.354156 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:30.354114 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21eb743f-41cd-425c-94e6-06109aa776b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a989da-v8v79\" (UID: \"21eb743f-41cd-425c-94e6-06109aa776b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a989da-v8v79" Oct 30 20:15:30.354383 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:30.354203 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21eb743f-41cd-425c-94e6-06109aa776b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a989da-v8v79\" (UID: \"21eb743f-41cd-425c-94e6-06109aa776b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a989da-v8v79" Oct 30 20:15:30.354383 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:30.354220 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2c8w\" (UniqueName: \"kubernetes.io/projected/21eb743f-41cd-425c-94e6-06109aa776b4-kube-api-access-d2c8w\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a989da-v8v79\" (UID: \"21eb743f-41cd-425c-94e6-06109aa776b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a989da-v8v79" Oct 30 20:15:30.354931 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:30.354905 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21eb743f-41cd-425c-94e6-06109aa776b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a989da-v8v79\" (UID: \"21eb743f-41cd-425c-94e6-06109aa776b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a989da-v8v79" Oct 30 20:15:30.356622 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:30.356593 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21eb743f-41cd-425c-94e6-06109aa776b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a989da-v8v79\" (UID: \"21eb743f-41cd-425c-94e6-06109aa776b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a989da-v8v79" Oct 30 20:15:30.380699 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:30.380665 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2c8w\" (UniqueName: \"kubernetes.io/projected/21eb743f-41cd-425c-94e6-06109aa776b4-kube-api-access-d2c8w\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a989da-v8v79\" (UID: \"21eb743f-41cd-425c-94e6-06109aa776b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a989da-v8v79" Oct 30 20:15:30.432752 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:30.432671 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:15:30.474430 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:30.474391 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a989da-v8v79" Oct 30 20:15:30.786644 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:30.786601 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a989da-v8v79"] Oct 30 20:15:30.788409 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:15:30.788375 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21eb743f_41cd_425c_94e6_06109aa776b4.slice/crio-356c09c60910bf1068bccd6173ca4e02df62c81a3c453b399884a45055649ea3 WatchSource:0}: Error finding container 356c09c60910bf1068bccd6173ca4e02df62c81a3c453b399884a45055649ea3: Status 404 returned error can't find the container with id 356c09c60910bf1068bccd6173ca4e02df62c81a3c453b399884a45055649ea3 Oct 30 20:15:31.551117 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:31.551081 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a989da-v8v79" event={"ID":"21eb743f-41cd-425c-94e6-06109aa776b4","Type":"ContainerStarted","Data":"bdf1a02543f77160dae0b8fb18ac193f74c2f106328899a16c1e1f98f8bce719"} Oct 30 20:15:31.551598 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:31.551159 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a989da-v8v79" event={"ID":"21eb743f-41cd-425c-94e6-06109aa776b4","Type":"ContainerStarted","Data":"356c09c60910bf1068bccd6173ca4e02df62c81a3c453b399884a45055649ea3"} Oct 30 20:15:31.600743 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:31.600683 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a989da-v8v79" podStartSLOduration=1.600668958 podStartE2EDuration="1.600668958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:31.598736763 +0000 UTC m=+1693.011450542" watchObservedRunningTime="2026-10-30 20:15:31.600668958 +0000 UTC m=+1693.013382729" Oct 30 20:15:32.559775 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:32.559720 5088 generic.go:356] "Generic (PLEG): container finished" podID="21eb743f-41cd-425c-94e6-06109aa776b4" containerID="bdf1a02543f77160dae0b8fb18ac193f74c2f106328899a16c1e1f98f8bce719" exitCode=0 Oct 30 20:15:32.560295 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:32.559824 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a989da-v8v79" event={"ID":"21eb743f-41cd-425c-94e6-06109aa776b4","Type":"ContainerDied","Data":"bdf1a02543f77160dae0b8fb18ac193f74c2f106328899a16c1e1f98f8bce719"} Oct 30 20:15:33.721852 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:33.721821 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a989da-v8v79" Oct 30 20:15:33.893606 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:33.893525 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21eb743f-41cd-425c-94e6-06109aa776b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21eb743f-41cd-425c-94e6-06109aa776b4-maas-api-openshift-ca\") pod \"21eb743f-41cd-425c-94e6-06109aa776b4\" (UID: \"21eb743f-41cd-425c-94e6-06109aa776b4\") " Oct 30 20:15:33.893783 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:33.893607 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21eb743f-41cd-425c-94e6-06109aa776b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21eb743f-41cd-425c-94e6-06109aa776b4-maas-api-xks-ca\") pod \"21eb743f-41cd-425c-94e6-06109aa776b4\" (UID: \"21eb743f-41cd-425c-94e6-06109aa776b4\") " Oct 30 20:15:33.893783 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:33.893715 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21eb743f-41cd-425c-94e6-06109aa776b4-kube-api-access-d2c8w\" (UniqueName: \"kubernetes.io/projected/21eb743f-41cd-425c-94e6-06109aa776b4-kube-api-access-d2c8w\") pod \"21eb743f-41cd-425c-94e6-06109aa776b4\" (UID: \"21eb743f-41cd-425c-94e6-06109aa776b4\") " Oct 30 20:15:33.893941 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:33.893918 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21eb743f-41cd-425c-94e6-06109aa776b4-maas-api-openshift-ca" pod "21eb743f-41cd-425c-94e6-06109aa776b4" (UID: "21eb743f-41cd-425c-94e6-06109aa776b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:33.894339 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:33.894142 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21eb743f-41cd-425c-94e6-06109aa776b4-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:15:33.896273 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:33.896251 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21eb743f-41cd-425c-94e6-06109aa776b4-maas-api-xks-ca" pod "21eb743f-41cd-425c-94e6-06109aa776b4" (UID: "21eb743f-41cd-425c-94e6-06109aa776b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:15:33.896371 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:33.896258 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21eb743f-41cd-425c-94e6-06109aa776b4-kube-api-access-d2c8w" pod "21eb743f-41cd-425c-94e6-06109aa776b4" (UID: "21eb743f-41cd-425c-94e6-06109aa776b4"). InnerVolumeSpecName "kube-api-access-d2c8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:33.994782 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:33.994739 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21eb743f-41cd-425c-94e6-06109aa776b4-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:15:33.994782 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:33.994774 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2c8w\" (UniqueName: \"kubernetes.io/projected/21eb743f-41cd-425c-94e6-06109aa776b4-kube-api-access-d2c8w\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:15:34.573151 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:34.573109 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a989da-v8v79" event={"ID":"21eb743f-41cd-425c-94e6-06109aa776b4","Type":"ContainerDied","Data":"356c09c60910bf1068bccd6173ca4e02df62c81a3c453b399884a45055649ea3"} Oct 30 20:15:34.573151 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:34.573134 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a989da-v8v79" Oct 30 20:15:34.573377 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:34.573172 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="356c09c60910bf1068bccd6173ca4e02df62c81a3c453b399884a45055649ea3" Oct 30 20:15:40.638757 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:40.638715 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a989da-67c48bfc7lvnq6"] Oct 30 20:15:40.741945 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:40.741906 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a989da-6658db8qkp"] Oct 30 20:15:42.397139 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:42.397103 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a989da-v8v79"] Oct 30 20:15:42.409901 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:42.409868 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a989da-v8v79"] Oct 30 20:15:42.739196 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:42.739153 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21eb743f-41cd-425c-94e6-06109aa776b4" path="/var/lib/kubelet/pods/21eb743f-41cd-425c-94e6-06109aa776b4/volumes" Oct 30 20:15:44.200893 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:44.200859 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:15:45.639503 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:45.639463 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a989da-67c48bfc7lvnq6" podUID="1fc1f63b-1880-484a-85c9-8b94119e81c6" containerName="payload-processing" containerID="cri-o://831ab5225d82bce2ab4703da906d266aa4d153b9cf355420d8c59fc068185629" gracePeriod=25 Oct 30 20:15:45.743140 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:45.743073 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a989da-6658db8qkp" podUID="5416142f-771f-438e-910b-56163ea335ae" containerName="payload-pre-processing" containerID="cri-o://bc6c68ce0e1c96259c155924fad5cd98b2535a24e1e82576fd792ae8a3130970" gracePeriod=25 Oct 30 20:15:50.045139 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:50.045101 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sjpgp"] Oct 30 20:15:50.060473 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:50.060441 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sjpgp"] Oct 30 20:15:50.739603 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:50.739571 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b887acf-189b-4980-92c2-6deed24c9e29" path="/var/lib/kubelet/pods/5b887acf-189b-4980-92c2-6deed24c9e29/volumes" Oct 30 20:15:51.491681 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:51.491639 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zklw6"] Oct 30 20:15:51.504340 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:51.504306 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zklw6"] Oct 30 20:15:51.535146 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:51.535113 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c8l"] Oct 30 20:15:51.535663 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:51.535635 5088 cpu_manager.go:414] "containerMap: removing container" podUID="21eb743f-41cd-425c-94e6-06109aa776b4" containerName="revoke-keys" Oct 30 20:15:51.535663 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:51.535653 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21eb743f-41cd-425c-94e6-06109aa776b4" containerName="revoke-keys" Oct 30 20:15:51.535837 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:51.535751 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="21eb743f-41cd-425c-94e6-06109aa776b4" containerName="revoke-keys" Oct 30 20:15:51.540266 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:51.540246 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c8l" Oct 30 20:15:51.558158 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:51.558117 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:15:51.558158 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:51.558142 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:15:51.558158 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:51.558156 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:15:51.558457 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:51.558210 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:15:51.560992 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:51.560968 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c8l"] Oct 30 20:15:51.589520 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:51.589488 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/262a8036-5981-483d-bd53-7d11f4dbfb68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-b2c8l\" (UID: \"262a8036-5981-483d-bd53-7d11f4dbfb68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c8l" Oct 30 20:15:51.589753 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:51.589551 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4xjf\" (UniqueName: \"kubernetes.io/projected/262a8036-5981-483d-bd53-7d11f4dbfb68-kube-api-access-f4xjf\") pod \"maas-api-revoke-sub-3d4156f1-b2c8l\" (UID: \"262a8036-5981-483d-bd53-7d11f4dbfb68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c8l" Oct 30 20:15:51.589753 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:51.589710 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/262a8036-5981-483d-bd53-7d11f4dbfb68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-b2c8l\" (UID: \"262a8036-5981-483d-bd53-7d11f4dbfb68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c8l" Oct 30 20:15:51.690432 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:51.690396 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/262a8036-5981-483d-bd53-7d11f4dbfb68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-b2c8l\" (UID: \"262a8036-5981-483d-bd53-7d11f4dbfb68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c8l" Oct 30 20:15:51.690662 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:51.690452 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4xjf\" (UniqueName: \"kubernetes.io/projected/262a8036-5981-483d-bd53-7d11f4dbfb68-kube-api-access-f4xjf\") pod \"maas-api-revoke-sub-3d4156f1-b2c8l\" (UID: \"262a8036-5981-483d-bd53-7d11f4dbfb68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c8l" Oct 30 20:15:51.690662 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:51.690482 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/262a8036-5981-483d-bd53-7d11f4dbfb68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-b2c8l\" (UID: \"262a8036-5981-483d-bd53-7d11f4dbfb68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c8l" Oct 30 20:15:51.691109 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:51.691086 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/262a8036-5981-483d-bd53-7d11f4dbfb68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-b2c8l\" (UID: \"262a8036-5981-483d-bd53-7d11f4dbfb68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c8l" Oct 30 20:15:51.692921 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:51.692899 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/262a8036-5981-483d-bd53-7d11f4dbfb68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-b2c8l\" (UID: \"262a8036-5981-483d-bd53-7d11f4dbfb68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c8l" Oct 30 20:15:51.718024 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:51.717990 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4xjf\" (UniqueName: \"kubernetes.io/projected/262a8036-5981-483d-bd53-7d11f4dbfb68-kube-api-access-f4xjf\") pod \"maas-api-revoke-sub-3d4156f1-b2c8l\" (UID: \"262a8036-5981-483d-bd53-7d11f4dbfb68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c8l" Oct 30 20:15:51.853512 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:51.853422 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c8l" Oct 30 20:15:52.163137 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:52.163092 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c8l"] Oct 30 20:15:52.163807 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:15:52.163775 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod262a8036_5981_483d_bd53_7d11f4dbfb68.slice/crio-877c9637fa73b5e178afe4a8395e3fd9f96f39d355e22bc4ce52b6dcd6217c52 WatchSource:0}: Error finding container 877c9637fa73b5e178afe4a8395e3fd9f96f39d355e22bc4ce52b6dcd6217c52: Status 404 returned error can't find the container with id 877c9637fa73b5e178afe4a8395e3fd9f96f39d355e22bc4ce52b6dcd6217c52 Oct 30 20:15:52.205453 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:52.205415 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a989da-6658db8qkp" podUID="5416142f-771f-438e-910b-56163ea335ae" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:15:52.205453 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:52.205434 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a989da-67c48bfc7lvnq6" podUID="1fc1f63b-1880-484a-85c9-8b94119e81c6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:15:52.438845 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:15:52.438809 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:15:52.688391 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:52.688295 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c8l" event={"ID":"262a8036-5981-483d-bd53-7d11f4dbfb68","Type":"ContainerStarted","Data":"fbad9fa17beec065baa10e57bcb0f0cd73c0a91a574d5d09b1c97a7e6ce55317"} Oct 30 20:15:52.688831 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:52.688397 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c8l" event={"ID":"262a8036-5981-483d-bd53-7d11f4dbfb68","Type":"ContainerStarted","Data":"877c9637fa73b5e178afe4a8395e3fd9f96f39d355e22bc4ce52b6dcd6217c52"} Oct 30 20:15:52.735168 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:52.735108 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c8l" podStartSLOduration=1.735094795 podStartE2EDuration="1.735094795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:52.733786544 +0000 UTC m=+1714.146500359" watchObservedRunningTime="2026-10-30 20:15:52.735094795 +0000 UTC m=+1714.147808622" Oct 30 20:15:52.738697 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:52.738672 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73d4afb1-deda-4dfb-b9f0-8958e8a6b913" path="/var/lib/kubelet/pods/73d4afb1-deda-4dfb-b9f0-8958e8a6b913/volumes" Oct 30 20:15:53.695267 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:53.695228 5088 generic.go:356] "Generic (PLEG): container finished" podID="262a8036-5981-483d-bd53-7d11f4dbfb68" containerID="fbad9fa17beec065baa10e57bcb0f0cd73c0a91a574d5d09b1c97a7e6ce55317" exitCode=0 Oct 30 20:15:53.695743 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:53.695320 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c8l" event={"ID":"262a8036-5981-483d-bd53-7d11f4dbfb68","Type":"ContainerDied","Data":"fbad9fa17beec065baa10e57bcb0f0cd73c0a91a574d5d09b1c97a7e6ce55317"} Oct 30 20:15:54.846765 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:54.846740 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c8l" Oct 30 20:15:54.925531 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:54.925481 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/262a8036-5981-483d-bd53-7d11f4dbfb68-kube-api-access-f4xjf\" (UniqueName: \"kubernetes.io/projected/262a8036-5981-483d-bd53-7d11f4dbfb68-kube-api-access-f4xjf\") pod \"262a8036-5981-483d-bd53-7d11f4dbfb68\" (UID: \"262a8036-5981-483d-bd53-7d11f4dbfb68\") " Oct 30 20:15:54.925778 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:54.925604 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/262a8036-5981-483d-bd53-7d11f4dbfb68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/262a8036-5981-483d-bd53-7d11f4dbfb68-maas-api-openshift-ca\") pod \"262a8036-5981-483d-bd53-7d11f4dbfb68\" (UID: \"262a8036-5981-483d-bd53-7d11f4dbfb68\") " Oct 30 20:15:54.925778 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:54.925702 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/262a8036-5981-483d-bd53-7d11f4dbfb68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/262a8036-5981-483d-bd53-7d11f4dbfb68-maas-api-xks-ca\") pod \"262a8036-5981-483d-bd53-7d11f4dbfb68\" (UID: \"262a8036-5981-483d-bd53-7d11f4dbfb68\") " Oct 30 20:15:54.925995 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:54.925959 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/262a8036-5981-483d-bd53-7d11f4dbfb68-maas-api-openshift-ca" pod "262a8036-5981-483d-bd53-7d11f4dbfb68" (UID: "262a8036-5981-483d-bd53-7d11f4dbfb68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:54.926125 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:54.926104 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/262a8036-5981-483d-bd53-7d11f4dbfb68-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:15:54.927840 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:54.927813 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/262a8036-5981-483d-bd53-7d11f4dbfb68-kube-api-access-f4xjf" pod "262a8036-5981-483d-bd53-7d11f4dbfb68" (UID: "262a8036-5981-483d-bd53-7d11f4dbfb68"). InnerVolumeSpecName "kube-api-access-f4xjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:54.927978 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:54.927860 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/262a8036-5981-483d-bd53-7d11f4dbfb68-maas-api-xks-ca" pod "262a8036-5981-483d-bd53-7d11f4dbfb68" (UID: "262a8036-5981-483d-bd53-7d11f4dbfb68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:15:55.027413 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:55.027381 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/262a8036-5981-483d-bd53-7d11f4dbfb68-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:15:55.027413 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:55.027409 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4xjf\" (UniqueName: \"kubernetes.io/projected/262a8036-5981-483d-bd53-7d11f4dbfb68-kube-api-access-f4xjf\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:15:55.708196 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:55.708168 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c8l" Oct 30 20:15:55.708394 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:55.708208 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c8l" event={"ID":"262a8036-5981-483d-bd53-7d11f4dbfb68","Type":"ContainerDied","Data":"877c9637fa73b5e178afe4a8395e3fd9f96f39d355e22bc4ce52b6dcd6217c52"} Oct 30 20:15:55.708394 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:15:55.708263 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="877c9637fa73b5e178afe4a8395e3fd9f96f39d355e22bc4ce52b6dcd6217c52" Oct 30 20:16:02.205822 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:02.205782 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a989da-67c48bfc7lvnq6" podUID="1fc1f63b-1880-484a-85c9-8b94119e81c6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:16:02.206375 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:02.205788 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a989da-6658db8qkp" podUID="5416142f-771f-438e-910b-56163ea335ae" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:16:03.631035 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:03.630998 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:16:05.856951 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:05.856901 5088 generic.go:356] "Generic (PLEG): container finished" podID="1fc1f63b-1880-484a-85c9-8b94119e81c6" containerID="831ab5225d82bce2ab4703da906d266aa4d153b9cf355420d8c59fc068185629" exitCode=0 Oct 30 20:16:05.877296 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:05.876687 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a989da-67c48bfc7lvnq6" event={"ID":"1fc1f63b-1880-484a-85c9-8b94119e81c6","Type":"ContainerDied","Data":"831ab5225d82bce2ab4703da906d266aa4d153b9cf355420d8c59fc068185629"} Oct 30 20:16:06.238815 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:06.238727 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a989da-67c48bfc7lvnq6" Oct 30 20:16:06.293943 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:06.293913 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a989da-6658db8qkp" Oct 30 20:16:06.319953 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:06.319925 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fc1f63b-1880-484a-85c9-8b94119e81c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fc1f63b-1880-484a-85c9-8b94119e81c6-plugins-config-volume\") pod \"1fc1f63b-1880-484a-85c9-8b94119e81c6\" (UID: \"1fc1f63b-1880-484a-85c9-8b94119e81c6\") " Oct 30 20:16:06.320110 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:06.319994 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fc1f63b-1880-484a-85c9-8b94119e81c6-kube-api-access-6q6s7\" (UniqueName: \"kubernetes.io/projected/1fc1f63b-1880-484a-85c9-8b94119e81c6-kube-api-access-6q6s7\") pod \"1fc1f63b-1880-484a-85c9-8b94119e81c6\" (UID: \"1fc1f63b-1880-484a-85c9-8b94119e81c6\") " Oct 30 20:16:06.320371 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:06.320347 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fc1f63b-1880-484a-85c9-8b94119e81c6-plugins-config-volume" pod "1fc1f63b-1880-484a-85c9-8b94119e81c6" (UID: "1fc1f63b-1880-484a-85c9-8b94119e81c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:16:06.322168 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:06.322143 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fc1f63b-1880-484a-85c9-8b94119e81c6-kube-api-access-6q6s7" pod "1fc1f63b-1880-484a-85c9-8b94119e81c6" (UID: "1fc1f63b-1880-484a-85c9-8b94119e81c6"). InnerVolumeSpecName "kube-api-access-6q6s7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:06.421385 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:06.421341 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5416142f-771f-438e-910b-56163ea335ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5416142f-771f-438e-910b-56163ea335ae-plugins-config-volume\") pod \"5416142f-771f-438e-910b-56163ea335ae\" (UID: \"5416142f-771f-438e-910b-56163ea335ae\") " Oct 30 20:16:06.421606 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:06.421429 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5416142f-771f-438e-910b-56163ea335ae-kube-api-access-57kn8\" (UniqueName: \"kubernetes.io/projected/5416142f-771f-438e-910b-56163ea335ae-kube-api-access-57kn8\") pod \"5416142f-771f-438e-910b-56163ea335ae\" (UID: \"5416142f-771f-438e-910b-56163ea335ae\") " Oct 30 20:16:06.421795 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:06.421771 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5416142f-771f-438e-910b-56163ea335ae-plugins-config-volume" pod "5416142f-771f-438e-910b-56163ea335ae" (UID: "5416142f-771f-438e-910b-56163ea335ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:16:06.421846 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:06.421816 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q6s7\" (UniqueName: \"kubernetes.io/projected/1fc1f63b-1880-484a-85c9-8b94119e81c6-kube-api-access-6q6s7\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:16:06.421846 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:06.421829 5088 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fc1f63b-1880-484a-85c9-8b94119e81c6-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:16:06.423512 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:06.423491 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5416142f-771f-438e-910b-56163ea335ae-kube-api-access-57kn8" pod "5416142f-771f-438e-910b-56163ea335ae" (UID: "5416142f-771f-438e-910b-56163ea335ae"). InnerVolumeSpecName "kube-api-access-57kn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:06.522438 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:06.522404 5088 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5416142f-771f-438e-910b-56163ea335ae-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:16:06.522438 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:06.522434 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57kn8\" (UniqueName: \"kubernetes.io/projected/5416142f-771f-438e-910b-56163ea335ae-kube-api-access-57kn8\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:16:06.884785 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:06.884688 5088 generic.go:356] "Generic (PLEG): container finished" podID="5416142f-771f-438e-910b-56163ea335ae" containerID="bc6c68ce0e1c96259c155924fad5cd98b2535a24e1e82576fd792ae8a3130970" exitCode=0 Oct 30 20:16:06.884785 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:06.884742 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a989da-6658db8qkp" Oct 30 20:16:06.885331 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:06.884731 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a989da-6658db8qkp" event={"ID":"5416142f-771f-438e-910b-56163ea335ae","Type":"ContainerDied","Data":"bc6c68ce0e1c96259c155924fad5cd98b2535a24e1e82576fd792ae8a3130970"} Oct 30 20:16:06.885331 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:06.884888 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a989da-6658db8qkp" event={"ID":"5416142f-771f-438e-910b-56163ea335ae","Type":"ContainerDied","Data":"d8a6880142b16b17181d8a1f1832287954668eb9c6897a7b1a022ed766d303f2"} Oct 30 20:16:06.885331 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:06.884934 5088 scope.go:122] "RemoveContainer" containerID="bc6c68ce0e1c96259c155924fad5cd98b2535a24e1e82576fd792ae8a3130970" Oct 30 20:16:06.886444 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:06.886401 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a989da-67c48bfc7lvnq6" Oct 30 20:16:06.886517 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:06.886442 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a989da-67c48bfc7lvnq6" event={"ID":"1fc1f63b-1880-484a-85c9-8b94119e81c6","Type":"ContainerDied","Data":"1bc2c7b8db82b66fb48cf5a5861f7450438776c68828f77aeb309e416f2a1325"} Oct 30 20:16:06.897688 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:06.896996 5088 scope.go:122] "RemoveContainer" containerID="bc6c68ce0e1c96259c155924fad5cd98b2535a24e1e82576fd792ae8a3130970" Oct 30 20:16:06.897787 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:16:06.897761 5088 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc6c68ce0e1c96259c155924fad5cd98b2535a24e1e82576fd792ae8a3130970\": container with ID starting with bc6c68ce0e1c96259c155924fad5cd98b2535a24e1e82576fd792ae8a3130970 not found: ID does not exist" containerID="bc6c68ce0e1c96259c155924fad5cd98b2535a24e1e82576fd792ae8a3130970" Oct 30 20:16:06.897838 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:06.897795 5088 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc6c68ce0e1c96259c155924fad5cd98b2535a24e1e82576fd792ae8a3130970"} err="failed to get container status \"bc6c68ce0e1c96259c155924fad5cd98b2535a24e1e82576fd792ae8a3130970\": rpc error: code = NotFound desc = could not find container \"bc6c68ce0e1c96259c155924fad5cd98b2535a24e1e82576fd792ae8a3130970\": container with ID starting with bc6c68ce0e1c96259c155924fad5cd98b2535a24e1e82576fd792ae8a3130970 not found: ID does not exist" Oct 30 20:16:06.897838 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:06.897815 5088 scope.go:122] "RemoveContainer" containerID="831ab5225d82bce2ab4703da906d266aa4d153b9cf355420d8c59fc068185629" Oct 30 20:16:06.944172 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:06.944135 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a989da-67c48bfc7lvnq6"] Oct 30 20:16:06.957824 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:06.957798 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a989da-67c48bfc7lvnq6"] Oct 30 20:16:06.988673 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:06.988643 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a989da-6658db8qkp"] Oct 30 20:16:06.995276 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:06.995247 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a989da-6658db8qkp"] Oct 30 20:16:08.738873 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:08.738843 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fc1f63b-1880-484a-85c9-8b94119e81c6" path="/var/lib/kubelet/pods/1fc1f63b-1880-484a-85c9-8b94119e81c6/volumes" Oct 30 20:16:08.739326 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:08.739209 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5416142f-771f-438e-910b-56163ea335ae" path="/var/lib/kubelet/pods/5416142f-771f-438e-910b-56163ea335ae/volumes" Oct 30 20:16:09.060196 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:09.060107 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:16:10.035386 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:10.035351 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z9pbl"] Oct 30 20:16:10.049253 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:10.049221 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-z9pbl"] Oct 30 20:16:10.739247 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:10.739213 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f32c9d53-2197-4ded-8237-305cdd5528a9" path="/var/lib/kubelet/pods/f32c9d53-2197-4ded-8237-305cdd5528a9/volumes" Oct 30 20:16:21.042700 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:21.042663 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rqscv"] Oct 30 20:16:21.056079 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:21.056018 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rqscv"] Oct 30 20:16:22.437935 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:16:22.437895 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:16:22.738934 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:22.738896 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d" path="/var/lib/kubelet/pods/d48ebadd-aacd-4e1d-9bc1-1ad174b5f09d/volumes" Oct 30 20:16:27.795917 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:27.795884 5088 scope.go:122] "RemoveContainer" containerID="c302df3e92694e010371046019929f6c93910cedebc8f040354bdc056abace48" Oct 30 20:16:49.035168 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:49.035128 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6z5jn"] Oct 30 20:16:49.048732 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:49.048700 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6z5jn"] Oct 30 20:16:50.738229 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:16:50.738195 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f01cb5ca-b494-4248-a645-4b0f0edccae9" path="/var/lib/kubelet/pods/f01cb5ca-b494-4248-a645-4b0f0edccae9/volumes" Oct 30 20:16:52.439130 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:16:52.439086 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:17:00.037655 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:00.037603 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-j4mt8"] Oct 30 20:17:00.050263 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:00.050221 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-j4mt8"] Oct 30 20:17:00.738948 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:00.738915 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6a9f523-bdde-48a1-9b10-4771a344df4a" path="/var/lib/kubelet/pods/f6a9f523-bdde-48a1-9b10-4771a344df4a/volumes" Oct 30 20:17:18.960042 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:18.960015 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qdj4h_da850825-59ab-480f-9047-afd7d2bdc19f/console-operator/1.log" Oct 30 20:17:18.964654 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:18.964630 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qdj4h_da850825-59ab-480f-9047-afd7d2bdc19f/console-operator/1.log" Oct 30 20:17:18.964780 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:18.964757 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xpjpn_688d079a-932d-4f63-97da-5cd6222a03e1/ovn-acl-logging/0.log" Oct 30 20:17:18.969650 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:18.969628 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xpjpn_688d079a-932d-4f63-97da-5cd6222a03e1/ovn-acl-logging/0.log" Oct 30 20:17:22.038728 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:22.038691 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6fr7c"] Oct 30 20:17:22.052457 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:22.052424 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6fr7c"] Oct 30 20:17:22.438122 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:17:22.438035 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:17:22.738948 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:22.738915 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62ceedbd-7e7e-4e87-99f0-88e6485f65f2" path="/var/lib/kubelet/pods/62ceedbd-7e7e-4e87-99f0-88e6485f65f2/volumes" Oct 30 20:17:27.827636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:27.827591 5088 scope.go:122] "RemoveContainer" containerID="613230f6ff2137f88200daff11ec519423d55d9a968dabbcdfbc8dd763853cfb" Oct 30 20:17:27.836746 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:27.836720 5088 scope.go:122] "RemoveContainer" containerID="010705871fc21f2262e38b7113638d2c3e0def8585d3e51e88400c932cfa7b38" Oct 30 20:17:27.845323 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:27.845302 5088 scope.go:122] "RemoveContainer" containerID="aef673440fb9c4fc235657af681a500ce1247a4ba055dff59744064f1a18950c" Oct 30 20:17:38.225759 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:38.225724 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:17:42.166416 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:42.166380 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flq2j"] Oct 30 20:17:42.166944 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:42.166843 5088 cpu_manager.go:414] "containerMap: removing container" podUID="5416142f-771f-438e-910b-56163ea335ae" containerName="payload-pre-processing" Oct 30 20:17:42.166944 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:42.166857 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5416142f-771f-438e-910b-56163ea335ae" containerName="payload-pre-processing" Oct 30 20:17:42.166944 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:42.166884 5088 cpu_manager.go:414] "containerMap: removing container" podUID="262a8036-5981-483d-bd53-7d11f4dbfb68" containerName="revoke-keys" Oct 30 20:17:42.166944 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:42.166889 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="262a8036-5981-483d-bd53-7d11f4dbfb68" containerName="revoke-keys" Oct 30 20:17:42.166944 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:42.166902 5088 cpu_manager.go:414] "containerMap: removing container" podUID="1fc1f63b-1880-484a-85c9-8b94119e81c6" containerName="payload-processing" Oct 30 20:17:42.166944 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:42.166907 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fc1f63b-1880-484a-85c9-8b94119e81c6" containerName="payload-processing" Oct 30 20:17:42.167145 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:42.166988 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="5416142f-771f-438e-910b-56163ea335ae" containerName="payload-pre-processing" Oct 30 20:17:42.167145 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:42.167002 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="262a8036-5981-483d-bd53-7d11f4dbfb68" containerName="revoke-keys" Oct 30 20:17:42.167145 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:42.167036 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fc1f63b-1880-484a-85c9-8b94119e81c6" containerName="payload-processing" Oct 30 20:17:42.170187 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:42.170170 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flq2j" Oct 30 20:17:42.192061 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:42.192021 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flq2j"] Oct 30 20:17:42.202888 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:42.202864 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:17:42.202888 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:42.202862 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:17:42.229037 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:42.228991 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:17:42.229037 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:42.229010 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:17:42.285270 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:42.285236 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5hlm\" (UniqueName: \"kubernetes.io/projected/d6beaf64-915d-4053-83a4-3362d28bbb91-kube-api-access-f5hlm\") pod \"maas-api-revoke-sub-d576e68b-flq2j\" (UID: \"d6beaf64-915d-4053-83a4-3362d28bbb91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flq2j" Oct 30 20:17:42.285452 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:42.285291 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6beaf64-915d-4053-83a4-3362d28bbb91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-flq2j\" (UID: \"d6beaf64-915d-4053-83a4-3362d28bbb91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flq2j" Oct 30 20:17:42.285508 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:42.285459 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6beaf64-915d-4053-83a4-3362d28bbb91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-flq2j\" (UID: \"d6beaf64-915d-4053-83a4-3362d28bbb91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flq2j" Oct 30 20:17:42.386329 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:42.386288 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5hlm\" (UniqueName: \"kubernetes.io/projected/d6beaf64-915d-4053-83a4-3362d28bbb91-kube-api-access-f5hlm\") pod \"maas-api-revoke-sub-d576e68b-flq2j\" (UID: \"d6beaf64-915d-4053-83a4-3362d28bbb91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flq2j" Oct 30 20:17:42.386538 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:42.386361 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6beaf64-915d-4053-83a4-3362d28bbb91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-flq2j\" (UID: \"d6beaf64-915d-4053-83a4-3362d28bbb91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flq2j" Oct 30 20:17:42.386538 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:42.386431 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6beaf64-915d-4053-83a4-3362d28bbb91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-flq2j\" (UID: \"d6beaf64-915d-4053-83a4-3362d28bbb91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flq2j" Oct 30 20:17:42.387119 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:42.387092 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6beaf64-915d-4053-83a4-3362d28bbb91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-flq2j\" (UID: \"d6beaf64-915d-4053-83a4-3362d28bbb91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flq2j" Oct 30 20:17:42.388851 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:42.388831 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6beaf64-915d-4053-83a4-3362d28bbb91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-flq2j\" (UID: \"d6beaf64-915d-4053-83a4-3362d28bbb91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flq2j" Oct 30 20:17:42.414323 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:42.414286 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5hlm\" (UniqueName: \"kubernetes.io/projected/d6beaf64-915d-4053-83a4-3362d28bbb91-kube-api-access-f5hlm\") pod \"maas-api-revoke-sub-d576e68b-flq2j\" (UID: \"d6beaf64-915d-4053-83a4-3362d28bbb91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flq2j" Oct 30 20:17:42.482751 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:42.482719 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flq2j" Oct 30 20:17:42.783679 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:42.783644 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flq2j"] Oct 30 20:17:42.786449 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:17:42.786423 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6beaf64_915d_4053_83a4_3362d28bbb91.slice/crio-8e3c8143d1984ca11a589fb2b678ef09edbb2e8386fc27f7bbf71d7fb617b48e WatchSource:0}: Error finding container 8e3c8143d1984ca11a589fb2b678ef09edbb2e8386fc27f7bbf71d7fb617b48e: Status 404 returned error can't find the container with id 8e3c8143d1984ca11a589fb2b678ef09edbb2e8386fc27f7bbf71d7fb617b48e Oct 30 20:17:43.034521 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:43.034451 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p4wm7"] Oct 30 20:17:43.038875 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:43.038852 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p4wm7"] Oct 30 20:17:43.454804 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:43.454717 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flq2j" event={"ID":"d6beaf64-915d-4053-83a4-3362d28bbb91","Type":"ContainerStarted","Data":"a74f410379ca73b7360a3a2af9831ead469f2a37f4c52379ba2434f67e6d6c70"} Oct 30 20:17:43.454804 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:43.454771 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flq2j" event={"ID":"d6beaf64-915d-4053-83a4-3362d28bbb91","Type":"ContainerStarted","Data":"8e3c8143d1984ca11a589fb2b678ef09edbb2e8386fc27f7bbf71d7fb617b48e"} Oct 30 20:17:43.500544 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:43.500493 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flq2j" podStartSLOduration=1.500480145 podStartE2EDuration="1.500480145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:17:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:17:43.499085213 +0000 UTC m=+1824.911799016" watchObservedRunningTime="2026-10-30 20:17:43.500480145 +0000 UTC m=+1824.913193928" Oct 30 20:17:44.465300 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:44.465261 5088 generic.go:356] "Generic (PLEG): container finished" podID="d6beaf64-915d-4053-83a4-3362d28bbb91" containerID="a74f410379ca73b7360a3a2af9831ead469f2a37f4c52379ba2434f67e6d6c70" exitCode=0 Oct 30 20:17:44.465743 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:44.465343 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flq2j" event={"ID":"d6beaf64-915d-4053-83a4-3362d28bbb91","Type":"ContainerDied","Data":"a74f410379ca73b7360a3a2af9831ead469f2a37f4c52379ba2434f67e6d6c70"} Oct 30 20:17:44.738535 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:44.738495 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e46d7707-4ed5-44ae-90ac-73116de6403e" path="/var/lib/kubelet/pods/e46d7707-4ed5-44ae-90ac-73116de6403e/volumes" Oct 30 20:17:45.607527 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:45.607497 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flq2j" Oct 30 20:17:45.718321 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:45.718278 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6beaf64-915d-4053-83a4-3362d28bbb91-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6beaf64-915d-4053-83a4-3362d28bbb91-maas-api-openshift-ca\") pod \"d6beaf64-915d-4053-83a4-3362d28bbb91\" (UID: \"d6beaf64-915d-4053-83a4-3362d28bbb91\") " Oct 30 20:17:45.718321 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:45.718335 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6beaf64-915d-4053-83a4-3362d28bbb91-kube-api-access-f5hlm\" (UniqueName: \"kubernetes.io/projected/d6beaf64-915d-4053-83a4-3362d28bbb91-kube-api-access-f5hlm\") pod \"d6beaf64-915d-4053-83a4-3362d28bbb91\" (UID: \"d6beaf64-915d-4053-83a4-3362d28bbb91\") " Oct 30 20:17:45.718580 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:45.718409 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6beaf64-915d-4053-83a4-3362d28bbb91-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6beaf64-915d-4053-83a4-3362d28bbb91-maas-api-xks-ca\") pod \"d6beaf64-915d-4053-83a4-3362d28bbb91\" (UID: \"d6beaf64-915d-4053-83a4-3362d28bbb91\") " Oct 30 20:17:45.718730 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:45.718701 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6beaf64-915d-4053-83a4-3362d28bbb91-maas-api-openshift-ca" pod "d6beaf64-915d-4053-83a4-3362d28bbb91" (UID: "d6beaf64-915d-4053-83a4-3362d28bbb91"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:45.720510 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:45.720487 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6beaf64-915d-4053-83a4-3362d28bbb91-maas-api-xks-ca" pod "d6beaf64-915d-4053-83a4-3362d28bbb91" (UID: "d6beaf64-915d-4053-83a4-3362d28bbb91"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:17:45.720593 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:45.720524 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6beaf64-915d-4053-83a4-3362d28bbb91-kube-api-access-f5hlm" pod "d6beaf64-915d-4053-83a4-3362d28bbb91" (UID: "d6beaf64-915d-4053-83a4-3362d28bbb91"). InnerVolumeSpecName "kube-api-access-f5hlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:45.819229 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:45.819192 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6beaf64-915d-4053-83a4-3362d28bbb91-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:17:45.819229 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:45.819225 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6beaf64-915d-4053-83a4-3362d28bbb91-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:17:45.819229 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:45.819235 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5hlm\" (UniqueName: \"kubernetes.io/projected/d6beaf64-915d-4053-83a4-3362d28bbb91-kube-api-access-f5hlm\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:17:46.478484 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:46.478455 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flq2j" Oct 30 20:17:46.478698 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:46.478495 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flq2j" event={"ID":"d6beaf64-915d-4053-83a4-3362d28bbb91","Type":"ContainerDied","Data":"8e3c8143d1984ca11a589fb2b678ef09edbb2e8386fc27f7bbf71d7fb617b48e"} Oct 30 20:17:46.478698 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:46.478551 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e3c8143d1984ca11a589fb2b678ef09edbb2e8386fc27f7bbf71d7fb617b48e" Oct 30 20:17:52.437523 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:17:52.437481 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:17:53.352950 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:53.347701 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:17:59.056904 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:59.056855 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lp2zb"] Oct 30 20:17:59.057473 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:59.057320 5088 cpu_manager.go:414] "containerMap: removing container" podUID="d6beaf64-915d-4053-83a4-3362d28bbb91" containerName="revoke-keys" Oct 30 20:17:59.057473 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:59.057332 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6beaf64-915d-4053-83a4-3362d28bbb91" containerName="revoke-keys" Oct 30 20:17:59.057473 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:59.057446 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6beaf64-915d-4053-83a4-3362d28bbb91" containerName="revoke-keys" Oct 30 20:17:59.061007 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:59.060980 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lp2zb" Oct 30 20:17:59.081125 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:59.081094 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lp2zb"] Oct 30 20:17:59.091813 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:59.091781 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:17:59.091813 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:59.091805 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:17:59.091813 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:59.091805 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:17:59.092170 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:59.091874 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:17:59.101541 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:59.101509 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6fdn\" (UniqueName: \"kubernetes.io/projected/7de7652b-0108-4745-90a4-5a31085b0409-kube-api-access-x6fdn\") pod \"maas-api-revoke-sub-d2bc93bc-lp2zb\" (UID: \"7de7652b-0108-4745-90a4-5a31085b0409\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lp2zb" Oct 30 20:17:59.101740 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:59.101549 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7de7652b-0108-4745-90a4-5a31085b0409-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-lp2zb\" (UID: \"7de7652b-0108-4745-90a4-5a31085b0409\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lp2zb" Oct 30 20:17:59.101829 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:59.101737 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7de7652b-0108-4745-90a4-5a31085b0409-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-lp2zb\" (UID: \"7de7652b-0108-4745-90a4-5a31085b0409\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lp2zb" Oct 30 20:17:59.202478 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:59.202435 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7de7652b-0108-4745-90a4-5a31085b0409-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-lp2zb\" (UID: \"7de7652b-0108-4745-90a4-5a31085b0409\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lp2zb" Oct 30 20:17:59.202727 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:59.202491 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6fdn\" (UniqueName: \"kubernetes.io/projected/7de7652b-0108-4745-90a4-5a31085b0409-kube-api-access-x6fdn\") pod \"maas-api-revoke-sub-d2bc93bc-lp2zb\" (UID: \"7de7652b-0108-4745-90a4-5a31085b0409\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lp2zb" Oct 30 20:17:59.202727 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:59.202525 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7de7652b-0108-4745-90a4-5a31085b0409-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-lp2zb\" (UID: \"7de7652b-0108-4745-90a4-5a31085b0409\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lp2zb" Oct 30 20:17:59.203218 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:59.203191 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7de7652b-0108-4745-90a4-5a31085b0409-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-lp2zb\" (UID: \"7de7652b-0108-4745-90a4-5a31085b0409\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lp2zb" Oct 30 20:17:59.204953 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:59.204933 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7de7652b-0108-4745-90a4-5a31085b0409-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-lp2zb\" (UID: \"7de7652b-0108-4745-90a4-5a31085b0409\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lp2zb" Oct 30 20:17:59.229345 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:59.229313 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6fdn\" (UniqueName: \"kubernetes.io/projected/7de7652b-0108-4745-90a4-5a31085b0409-kube-api-access-x6fdn\") pod \"maas-api-revoke-sub-d2bc93bc-lp2zb\" (UID: \"7de7652b-0108-4745-90a4-5a31085b0409\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lp2zb" Oct 30 20:17:59.374385 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:59.374290 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lp2zb" Oct 30 20:17:59.675578 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:17:59.675551 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lp2zb"] Oct 30 20:17:59.677528 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:17:59.677499 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7de7652b_0108_4745_90a4_5a31085b0409.slice/crio-1244425bf118a5caaf647242adf804a12e2f2f69ee08b367b3aa52fad936f4bf WatchSource:0}: Error finding container 1244425bf118a5caaf647242adf804a12e2f2f69ee08b367b3aa52fad936f4bf: Status 404 returned error can't find the container with id 1244425bf118a5caaf647242adf804a12e2f2f69ee08b367b3aa52fad936f4bf Oct 30 20:18:00.588574 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:00.588540 5088 generic.go:356] "Generic (PLEG): container finished" podID="7de7652b-0108-4745-90a4-5a31085b0409" containerID="e6d8133d0fefb47337affcaa164ed1f9e0cb4f018eb1d27f62c3e02de306fe76" exitCode=0 Oct 30 20:18:00.588970 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:00.588640 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lp2zb" event={"ID":"7de7652b-0108-4745-90a4-5a31085b0409","Type":"ContainerDied","Data":"e6d8133d0fefb47337affcaa164ed1f9e0cb4f018eb1d27f62c3e02de306fe76"} Oct 30 20:18:00.588970 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:00.588694 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lp2zb" event={"ID":"7de7652b-0108-4745-90a4-5a31085b0409","Type":"ContainerStarted","Data":"1244425bf118a5caaf647242adf804a12e2f2f69ee08b367b3aa52fad936f4bf"} Oct 30 20:18:01.732592 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:01.732555 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lp2zb" Oct 30 20:18:01.826571 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:01.826526 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7de7652b-0108-4745-90a4-5a31085b0409-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7de7652b-0108-4745-90a4-5a31085b0409-maas-api-openshift-ca\") pod \"7de7652b-0108-4745-90a4-5a31085b0409\" (UID: \"7de7652b-0108-4745-90a4-5a31085b0409\") " Oct 30 20:18:01.826571 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:01.826578 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7de7652b-0108-4745-90a4-5a31085b0409-kube-api-access-x6fdn\" (UniqueName: \"kubernetes.io/projected/7de7652b-0108-4745-90a4-5a31085b0409-kube-api-access-x6fdn\") pod \"7de7652b-0108-4745-90a4-5a31085b0409\" (UID: \"7de7652b-0108-4745-90a4-5a31085b0409\") " Oct 30 20:18:01.826821 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:01.826672 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7de7652b-0108-4745-90a4-5a31085b0409-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7de7652b-0108-4745-90a4-5a31085b0409-maas-api-xks-ca\") pod \"7de7652b-0108-4745-90a4-5a31085b0409\" (UID: \"7de7652b-0108-4745-90a4-5a31085b0409\") " Oct 30 20:18:01.827011 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:01.826984 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7de7652b-0108-4745-90a4-5a31085b0409-maas-api-openshift-ca" pod "7de7652b-0108-4745-90a4-5a31085b0409" (UID: "7de7652b-0108-4745-90a4-5a31085b0409"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:18:01.827184 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:01.827168 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7de7652b-0108-4745-90a4-5a31085b0409-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:18:01.828887 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:01.828869 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7de7652b-0108-4745-90a4-5a31085b0409-maas-api-xks-ca" pod "7de7652b-0108-4745-90a4-5a31085b0409" (UID: "7de7652b-0108-4745-90a4-5a31085b0409"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:18:01.829034 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:01.829007 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7de7652b-0108-4745-90a4-5a31085b0409-kube-api-access-x6fdn" pod "7de7652b-0108-4745-90a4-5a31085b0409" (UID: "7de7652b-0108-4745-90a4-5a31085b0409"). InnerVolumeSpecName "kube-api-access-x6fdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:01.928445 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:01.928362 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6fdn\" (UniqueName: \"kubernetes.io/projected/7de7652b-0108-4745-90a4-5a31085b0409-kube-api-access-x6fdn\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:18:01.928445 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:01.928392 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7de7652b-0108-4745-90a4-5a31085b0409-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:18:02.600732 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:02.600702 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lp2zb" Oct 30 20:18:02.600916 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:02.600701 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lp2zb" event={"ID":"7de7652b-0108-4745-90a4-5a31085b0409","Type":"ContainerDied","Data":"1244425bf118a5caaf647242adf804a12e2f2f69ee08b367b3aa52fad936f4bf"} Oct 30 20:18:02.600916 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:02.600827 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1244425bf118a5caaf647242adf804a12e2f2f69ee08b367b3aa52fad936f4bf" Oct 30 20:18:04.036730 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:04.036684 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nqnxl"] Oct 30 20:18:04.050084 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:04.050051 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nqnxl"] Oct 30 20:18:04.738361 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:04.738324 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fbcac15-3eac-4838-b1db-ddd8a11e9df1" path="/var/lib/kubelet/pods/0fbcac15-3eac-4838-b1db-ddd8a11e9df1/volumes" Oct 30 20:18:16.901584 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:16.901543 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5d6bba-84fb4fb64rcj95"] Oct 30 20:18:16.902803 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:16.902776 5088 cpu_manager.go:414] "containerMap: removing container" podUID="7de7652b-0108-4745-90a4-5a31085b0409" containerName="revoke-keys" Oct 30 20:18:16.902984 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:16.902969 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7de7652b-0108-4745-90a4-5a31085b0409" containerName="revoke-keys" Oct 30 20:18:16.903430 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:16.903413 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="7de7652b-0108-4745-90a4-5a31085b0409" containerName="revoke-keys" Oct 30 20:18:16.909036 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:16.908990 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d6bba-84fb4fb64rcj95" Oct 30 20:18:16.925752 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:16.925725 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5d6bba-84fb4fb64rcj95"] Oct 30 20:18:16.941807 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:16.941783 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-5d6bba\"" Oct 30 20:18:16.942030 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:16.942012 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-bb58bc79-dockercfg-k9b8j\"" Oct 30 20:18:16.983375 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:16.983345 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdt8x\" (UniqueName: \"kubernetes.io/projected/f31414c5-bc04-42de-a73a-2bf16f64610f-kube-api-access-tdt8x\") pod \"payload-processing-e2e-worker-single-main-5d6bba-84fb4fb64rcj95\" (UID: \"f31414c5-bc04-42de-a73a-2bf16f64610f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d6bba-84fb4fb64rcj95" Oct 30 20:18:16.983540 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:16.983408 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f31414c5-bc04-42de-a73a-2bf16f64610f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-5d6bba-84fb4fb64rcj95\" (UID: \"f31414c5-bc04-42de-a73a-2bf16f64610f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d6bba-84fb4fb64rcj95" Oct 30 20:18:17.084909 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:17.084875 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f31414c5-bc04-42de-a73a-2bf16f64610f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-5d6bba-84fb4fb64rcj95\" (UID: \"f31414c5-bc04-42de-a73a-2bf16f64610f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d6bba-84fb4fb64rcj95" Oct 30 20:18:17.085104 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:17.084985 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdt8x\" (UniqueName: \"kubernetes.io/projected/f31414c5-bc04-42de-a73a-2bf16f64610f-kube-api-access-tdt8x\") pod \"payload-processing-e2e-worker-single-main-5d6bba-84fb4fb64rcj95\" (UID: \"f31414c5-bc04-42de-a73a-2bf16f64610f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d6bba-84fb4fb64rcj95" Oct 30 20:18:17.085695 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:17.085674 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f31414c5-bc04-42de-a73a-2bf16f64610f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-5d6bba-84fb4fb64rcj95\" (UID: \"f31414c5-bc04-42de-a73a-2bf16f64610f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d6bba-84fb4fb64rcj95" Oct 30 20:18:17.113322 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:17.113285 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdt8x\" (UniqueName: \"kubernetes.io/projected/f31414c5-bc04-42de-a73a-2bf16f64610f-kube-api-access-tdt8x\") pod \"payload-processing-e2e-worker-single-main-5d6bba-84fb4fb64rcj95\" (UID: \"f31414c5-bc04-42de-a73a-2bf16f64610f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d6bba-84fb4fb64rcj95" Oct 30 20:18:17.225129 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:17.225039 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d6bba-84fb4fb64rcj95" Oct 30 20:18:17.543382 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:17.543354 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5d6bba-84fb4fb64rcj95"] Oct 30 20:18:17.545365 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:18:17.545329 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf31414c5_bc04_42de_a73a_2bf16f64610f.slice/crio-217198b991add201247711bf27f7a472ca08ccc4b53b038859f2c92fd0b12a51 WatchSource:0}: Error finding container 217198b991add201247711bf27f7a472ca08ccc4b53b038859f2c92fd0b12a51: Status 404 returned error can't find the container with id 217198b991add201247711bf27f7a472ca08ccc4b53b038859f2c92fd0b12a51 Oct 30 20:18:17.688337 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:17.688296 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d6bba-84fb4fb64rcj95" event={"ID":"f31414c5-bc04-42de-a73a-2bf16f64610f","Type":"ContainerStarted","Data":"997a0df2f7c280912559225436eabad6c785cae31b768df881cd68464d6b4744"} Oct 30 20:18:17.688540 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:17.688362 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d6bba-84fb4fb64rcj95" event={"ID":"f31414c5-bc04-42de-a73a-2bf16f64610f","Type":"ContainerStarted","Data":"217198b991add201247711bf27f7a472ca08ccc4b53b038859f2c92fd0b12a51"} Oct 30 20:18:17.688540 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:17.688474 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d6bba-84fb4fb64rcj95" Oct 30 20:18:17.733457 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:17.733397 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d6bba-84fb4fb64rcj95" podStartSLOduration=1.733384193 podStartE2EDuration="1.733384193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:17.732706062 +0000 UTC m=+1859.145419861" watchObservedRunningTime="2026-10-30 20:18:17.733384193 +0000 UTC m=+1859.146097972" Oct 30 20:18:20.567497 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:20.567459 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:18:21.035807 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:21.035768 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-29tcl"] Oct 30 20:18:21.049253 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:21.049222 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-29tcl"] Oct 30 20:18:22.438916 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:18:22.438880 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:18:22.738878 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:22.738842 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c71e4333-f91b-4cad-864c-a5859918847a" path="/var/lib/kubelet/pods/c71e4333-f91b-4cad-864c-a5859918847a/volumes" Oct 30 20:18:27.880567 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:27.880536 5088 scope.go:122] "RemoveContainer" containerID="7f2e875dab008bb0c67acd8f27918547aa6dbd49275345ff73cb7c5969a6ff97" Oct 30 20:18:27.889503 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:27.889479 5088 scope.go:122] "RemoveContainer" containerID="a3707ef386fbf3e61e087256f81c7811e90e357c4c4ed9a6ebca88e3155c1c5e" Oct 30 20:18:27.900199 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:27.900175 5088 scope.go:122] "RemoveContainer" containerID="fce48059b335127c85c3e0dcfd88a2cdf5dfdcd0b500ebb4b4f31f4466a8a185" Oct 30 20:18:28.705521 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:28.705485 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d6bba-84fb4fb64rcj95" Oct 30 20:18:34.869740 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:34.869699 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:18:49.731486 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:49.731450 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:18:51.790220 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:51.790180 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8b95186-ch52x"] Oct 30 20:18:51.794502 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:51.794480 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b95186-ch52x" Oct 30 20:18:51.812302 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:51.812270 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:18:51.825253 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:51.825204 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:18:51.825253 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:51.825235 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:18:51.825457 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:51.825286 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:18:51.828040 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:51.828017 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8b95186-ch52x"] Oct 30 20:18:51.874145 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:51.874111 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw6sq\" (UniqueName: \"kubernetes.io/projected/6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720-kube-api-access-fw6sq\") pod \"maas-api-revoke-sub-b8b95186-ch52x\" (UID: \"6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b95186-ch52x" Oct 30 20:18:51.874348 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:51.874210 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8b95186-ch52x\" (UID: \"6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b95186-ch52x" Oct 30 20:18:51.874348 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:51.874239 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8b95186-ch52x\" (UID: \"6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b95186-ch52x" Oct 30 20:18:51.975723 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:51.975687 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw6sq\" (UniqueName: \"kubernetes.io/projected/6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720-kube-api-access-fw6sq\") pod \"maas-api-revoke-sub-b8b95186-ch52x\" (UID: \"6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b95186-ch52x" Oct 30 20:18:51.975897 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:51.975737 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8b95186-ch52x\" (UID: \"6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b95186-ch52x" Oct 30 20:18:51.975946 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:51.975922 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8b95186-ch52x\" (UID: \"6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b95186-ch52x" Oct 30 20:18:51.976431 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:51.976414 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8b95186-ch52x\" (UID: \"6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b95186-ch52x" Oct 30 20:18:51.978203 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:51.978186 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8b95186-ch52x\" (UID: \"6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b95186-ch52x" Oct 30 20:18:52.002037 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:52.002006 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw6sq\" (UniqueName: \"kubernetes.io/projected/6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720-kube-api-access-fw6sq\") pod \"maas-api-revoke-sub-b8b95186-ch52x\" (UID: \"6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b95186-ch52x" Oct 30 20:18:52.108262 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:52.108182 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b95186-ch52x" Oct 30 20:18:52.410712 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:52.410680 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8b95186-ch52x"] Oct 30 20:18:52.414555 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:18:52.414517 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bb9b70f_ddb8_439a_9ce0_7c2c4ee80720.slice/crio-a2263c44dd17d72cd0eb8c47a7854c20ca710f51b141ce2aa62dc2be92adc16f WatchSource:0}: Error finding container a2263c44dd17d72cd0eb8c47a7854c20ca710f51b141ce2aa62dc2be92adc16f: Status 404 returned error can't find the container with id a2263c44dd17d72cd0eb8c47a7854c20ca710f51b141ce2aa62dc2be92adc16f Oct 30 20:18:52.439796 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:18:52.439760 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:18:52.884721 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:52.884675 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b95186-ch52x" event={"ID":"6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720","Type":"ContainerStarted","Data":"f1dfc2d87b830c6eecd5bd5c13ca5d670ca33b6bc51424d4389fad0b93e9ffd8"} Oct 30 20:18:52.885190 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:52.884738 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b95186-ch52x" event={"ID":"6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720","Type":"ContainerStarted","Data":"a2263c44dd17d72cd0eb8c47a7854c20ca710f51b141ce2aa62dc2be92adc16f"} Oct 30 20:18:52.932999 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:52.932944 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b95186-ch52x" podStartSLOduration=1.932925876 podStartE2EDuration="1.932925876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:52.930115128 +0000 UTC m=+1894.342828919" watchObservedRunningTime="2026-10-30 20:18:52.932925876 +0000 UTC m=+1894.345639651" Oct 30 20:18:53.891216 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:53.891177 5088 generic.go:356] "Generic (PLEG): container finished" podID="6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720" containerID="f1dfc2d87b830c6eecd5bd5c13ca5d670ca33b6bc51424d4389fad0b93e9ffd8" exitCode=0 Oct 30 20:18:53.891706 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:53.891250 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b95186-ch52x" event={"ID":"6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720","Type":"ContainerDied","Data":"f1dfc2d87b830c6eecd5bd5c13ca5d670ca33b6bc51424d4389fad0b93e9ffd8"} Oct 30 20:18:55.043397 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:55.043372 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b95186-ch52x" Oct 30 20:18:55.211867 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:55.211779 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720-maas-api-openshift-ca\") pod \"6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720\" (UID: \"6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720\") " Oct 30 20:18:55.212034 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:55.211885 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720-kube-api-access-fw6sq\" (UniqueName: \"kubernetes.io/projected/6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720-kube-api-access-fw6sq\") pod \"6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720\" (UID: \"6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720\") " Oct 30 20:18:55.212034 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:55.211957 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720-maas-api-xks-ca\") pod \"6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720\" (UID: \"6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720\") " Oct 30 20:18:55.212253 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:55.212222 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720-maas-api-openshift-ca" pod "6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720" (UID: "6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:18:55.214102 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:55.214073 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720-kube-api-access-fw6sq" pod "6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720" (UID: "6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720"). InnerVolumeSpecName "kube-api-access-fw6sq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:55.214102 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:55.214084 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720-maas-api-xks-ca" pod "6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720" (UID: "6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:18:55.313631 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:55.313576 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:18:55.313631 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:55.313609 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw6sq\" (UniqueName: \"kubernetes.io/projected/6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720-kube-api-access-fw6sq\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:18:55.313631 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:55.313631 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:18:55.904822 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:55.904779 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b95186-ch52x" event={"ID":"6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720","Type":"ContainerDied","Data":"a2263c44dd17d72cd0eb8c47a7854c20ca710f51b141ce2aa62dc2be92adc16f"} Oct 30 20:18:55.904822 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:55.904806 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b95186-ch52x" Oct 30 20:18:55.905051 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:18:55.904845 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2263c44dd17d72cd0eb8c47a7854c20ca710f51b141ce2aa62dc2be92adc16f" Oct 30 20:19:02.859771 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:02.859731 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d6bba-hjz42"] Oct 30 20:19:02.860226 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:02.860214 5088 cpu_manager.go:414] "containerMap: removing container" podUID="6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720" containerName="revoke-keys" Oct 30 20:19:02.860276 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:02.860228 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720" containerName="revoke-keys" Oct 30 20:19:02.860324 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:02.860298 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720" containerName="revoke-keys" Oct 30 20:19:02.863750 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:02.863733 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d6bba-hjz42" Oct 30 20:19:02.885667 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:02.885637 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d6bba-hjz42"] Oct 30 20:19:02.896645 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:02.895738 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:19:02.896645 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:02.895738 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:19:02.896645 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:02.895777 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:19:02.896645 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:02.895884 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:19:02.996079 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:02.996036 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61db385d-60af-43d0-8628-66e5d990663a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5d6bba-hjz42\" (UID: \"61db385d-60af-43d0-8628-66e5d990663a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d6bba-hjz42" Oct 30 20:19:02.996262 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:02.996096 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qt7s\" (UniqueName: \"kubernetes.io/projected/61db385d-60af-43d0-8628-66e5d990663a-kube-api-access-8qt7s\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5d6bba-hjz42\" (UID: \"61db385d-60af-43d0-8628-66e5d990663a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d6bba-hjz42" Oct 30 20:19:02.996262 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:02.996202 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61db385d-60af-43d0-8628-66e5d990663a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5d6bba-hjz42\" (UID: \"61db385d-60af-43d0-8628-66e5d990663a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d6bba-hjz42" Oct 30 20:19:03.096955 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:03.096915 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61db385d-60af-43d0-8628-66e5d990663a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5d6bba-hjz42\" (UID: \"61db385d-60af-43d0-8628-66e5d990663a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d6bba-hjz42" Oct 30 20:19:03.097178 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:03.096967 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qt7s\" (UniqueName: \"kubernetes.io/projected/61db385d-60af-43d0-8628-66e5d990663a-kube-api-access-8qt7s\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5d6bba-hjz42\" (UID: \"61db385d-60af-43d0-8628-66e5d990663a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d6bba-hjz42" Oct 30 20:19:03.097178 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:03.097010 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61db385d-60af-43d0-8628-66e5d990663a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5d6bba-hjz42\" (UID: \"61db385d-60af-43d0-8628-66e5d990663a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d6bba-hjz42" Oct 30 20:19:03.097701 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:03.097678 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61db385d-60af-43d0-8628-66e5d990663a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5d6bba-hjz42\" (UID: \"61db385d-60af-43d0-8628-66e5d990663a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d6bba-hjz42" Oct 30 20:19:03.099469 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:03.099445 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61db385d-60af-43d0-8628-66e5d990663a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5d6bba-hjz42\" (UID: \"61db385d-60af-43d0-8628-66e5d990663a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d6bba-hjz42" Oct 30 20:19:03.124387 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:03.124321 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qt7s\" (UniqueName: \"kubernetes.io/projected/61db385d-60af-43d0-8628-66e5d990663a-kube-api-access-8qt7s\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5d6bba-hjz42\" (UID: \"61db385d-60af-43d0-8628-66e5d990663a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d6bba-hjz42" Oct 30 20:19:03.176789 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:03.176759 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d6bba-hjz42" Oct 30 20:19:03.225406 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:03.225374 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:19:03.494583 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:03.494531 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d6bba-hjz42"] Oct 30 20:19:03.958438 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:03.958406 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d6bba-hjz42" event={"ID":"61db385d-60af-43d0-8628-66e5d990663a","Type":"ContainerStarted","Data":"8c75180b7eab96bc04ba2d4434cefa3cdf614c9878ce91bd27f7848491fbea5c"} Oct 30 20:19:03.958893 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:03.958461 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d6bba-hjz42" event={"ID":"61db385d-60af-43d0-8628-66e5d990663a","Type":"ContainerStarted","Data":"606bbddf739653c95260b4f2871b9acc70a3ceb81e2cfd47799f653504161e9d"} Oct 30 20:19:04.011540 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:04.011483 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d6bba-hjz42" podStartSLOduration=2.011469368 podStartE2EDuration="2.011469368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:04.009748911 +0000 UTC m=+1905.422462747" watchObservedRunningTime="2026-10-30 20:19:04.011469368 +0000 UTC m=+1905.424183188" Oct 30 20:19:04.965199 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:04.965163 5088 generic.go:356] "Generic (PLEG): container finished" podID="61db385d-60af-43d0-8628-66e5d990663a" containerID="8c75180b7eab96bc04ba2d4434cefa3cdf614c9878ce91bd27f7848491fbea5c" exitCode=0 Oct 30 20:19:04.965635 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:04.965246 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d6bba-hjz42" event={"ID":"61db385d-60af-43d0-8628-66e5d990663a","Type":"ContainerDied","Data":"8c75180b7eab96bc04ba2d4434cefa3cdf614c9878ce91bd27f7848491fbea5c"} Oct 30 20:19:06.106076 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:06.106053 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d6bba-hjz42" Oct 30 20:19:06.130196 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:06.130162 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61db385d-60af-43d0-8628-66e5d990663a-kube-api-access-8qt7s\" (UniqueName: \"kubernetes.io/projected/61db385d-60af-43d0-8628-66e5d990663a-kube-api-access-8qt7s\") pod \"61db385d-60af-43d0-8628-66e5d990663a\" (UID: \"61db385d-60af-43d0-8628-66e5d990663a\") " Oct 30 20:19:06.130359 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:06.130221 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61db385d-60af-43d0-8628-66e5d990663a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61db385d-60af-43d0-8628-66e5d990663a-maas-api-openshift-ca\") pod \"61db385d-60af-43d0-8628-66e5d990663a\" (UID: \"61db385d-60af-43d0-8628-66e5d990663a\") " Oct 30 20:19:06.130359 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:06.130288 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61db385d-60af-43d0-8628-66e5d990663a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61db385d-60af-43d0-8628-66e5d990663a-maas-api-xks-ca\") pod \"61db385d-60af-43d0-8628-66e5d990663a\" (UID: \"61db385d-60af-43d0-8628-66e5d990663a\") " Oct 30 20:19:06.130602 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:06.130581 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61db385d-60af-43d0-8628-66e5d990663a-maas-api-openshift-ca" pod "61db385d-60af-43d0-8628-66e5d990663a" (UID: "61db385d-60af-43d0-8628-66e5d990663a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:19:06.132416 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:06.132398 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61db385d-60af-43d0-8628-66e5d990663a-kube-api-access-8qt7s" pod "61db385d-60af-43d0-8628-66e5d990663a" (UID: "61db385d-60af-43d0-8628-66e5d990663a"). InnerVolumeSpecName "kube-api-access-8qt7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:19:06.132468 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:06.132411 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61db385d-60af-43d0-8628-66e5d990663a-maas-api-xks-ca" pod "61db385d-60af-43d0-8628-66e5d990663a" (UID: "61db385d-60af-43d0-8628-66e5d990663a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:19:06.231410 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:06.231318 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61db385d-60af-43d0-8628-66e5d990663a-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:19:06.231410 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:06.231351 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61db385d-60af-43d0-8628-66e5d990663a-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:19:06.231410 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:06.231367 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qt7s\" (UniqueName: \"kubernetes.io/projected/61db385d-60af-43d0-8628-66e5d990663a-kube-api-access-8qt7s\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:19:06.978757 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:06.978724 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d6bba-hjz42" Oct 30 20:19:06.978757 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:06.978746 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d6bba-hjz42" event={"ID":"61db385d-60af-43d0-8628-66e5d990663a","Type":"ContainerDied","Data":"606bbddf739653c95260b4f2871b9acc70a3ceb81e2cfd47799f653504161e9d"} Oct 30 20:19:06.978980 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:06.978790 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="606bbddf739653c95260b4f2871b9acc70a3ceb81e2cfd47799f653504161e9d" Oct 30 20:19:13.402482 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:13.402437 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5d6bba-84fb4fb64rcj95"] Oct 30 20:19:15.292869 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:15.292804 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d6bba-hjz42"] Oct 30 20:19:15.305394 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:15.305362 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5d6bba-hjz42"] Oct 30 20:19:16.739418 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:16.739380 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61db385d-60af-43d0-8628-66e5d990663a" path="/var/lib/kubelet/pods/61db385d-60af-43d0-8628-66e5d990663a/volumes" Oct 30 20:19:16.996115 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:16.996030 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:19:18.403549 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:18.403486 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d6bba-84fb4fb64rcj95" podUID="f31414c5-bc04-42de-a73a-2bf16f64610f" containerName="payload-processing" containerID="cri-o://997a0df2f7c280912559225436eabad6c785cae31b768df881cd68464d6b4744" gracePeriod=25 Oct 30 20:19:18.695942 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:18.695850 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d6bba-84fb4fb64rcj95" podUID="f31414c5-bc04-42de-a73a-2bf16f64610f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:19:22.448601 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:19:22.448529 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:19:27.930706 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:27.930672 5088 scope.go:122] "RemoveContainer" containerID="f2649853951573801a308505fa3af0008c16ff2a07bccd78437e8d549e45e5c9" Oct 30 20:19:27.940071 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:27.940049 5088 scope.go:122] "RemoveContainer" containerID="e9948dadfed2fb320f63aad60b5e068574ffb6adfe45cb22b4533bf77d9ccd2b" Oct 30 20:19:27.948410 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:27.948389 5088 scope.go:122] "RemoveContainer" containerID="3877d83d30d5a9021cd8b7ffad1cb3919c6890ee875f441fa16e8ee09a66a495" Oct 30 20:19:28.696377 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:28.696338 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d6bba-84fb4fb64rcj95" podUID="f31414c5-bc04-42de-a73a-2bf16f64610f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:19:28.869152 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:28.869117 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:19:38.654523 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:38.654496 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d6bba-84fb4fb64rcj95" Oct 30 20:19:38.824653 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:38.824592 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f31414c5-bc04-42de-a73a-2bf16f64610f-kube-api-access-tdt8x\" (UniqueName: \"kubernetes.io/projected/f31414c5-bc04-42de-a73a-2bf16f64610f-kube-api-access-tdt8x\") pod \"f31414c5-bc04-42de-a73a-2bf16f64610f\" (UID: \"f31414c5-bc04-42de-a73a-2bf16f64610f\") " Oct 30 20:19:38.824883 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:38.824669 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f31414c5-bc04-42de-a73a-2bf16f64610f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f31414c5-bc04-42de-a73a-2bf16f64610f-plugins-config-volume\") pod \"f31414c5-bc04-42de-a73a-2bf16f64610f\" (UID: \"f31414c5-bc04-42de-a73a-2bf16f64610f\") " Oct 30 20:19:38.825068 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:38.825045 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f31414c5-bc04-42de-a73a-2bf16f64610f-plugins-config-volume" pod "f31414c5-bc04-42de-a73a-2bf16f64610f" (UID: "f31414c5-bc04-42de-a73a-2bf16f64610f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:19:38.826753 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:38.826728 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f31414c5-bc04-42de-a73a-2bf16f64610f-kube-api-access-tdt8x" pod "f31414c5-bc04-42de-a73a-2bf16f64610f" (UID: "f31414c5-bc04-42de-a73a-2bf16f64610f"). InnerVolumeSpecName "kube-api-access-tdt8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:19:38.925897 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:38.925858 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdt8x\" (UniqueName: \"kubernetes.io/projected/f31414c5-bc04-42de-a73a-2bf16f64610f-kube-api-access-tdt8x\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:19:38.925897 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:38.925889 5088 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f31414c5-bc04-42de-a73a-2bf16f64610f-plugins-config-volume\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:19:39.223177 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:39.223091 5088 generic.go:356] "Generic (PLEG): container finished" podID="f31414c5-bc04-42de-a73a-2bf16f64610f" containerID="997a0df2f7c280912559225436eabad6c785cae31b768df881cd68464d6b4744" exitCode=0 Oct 30 20:19:39.223177 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:39.223145 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d6bba-84fb4fb64rcj95" Oct 30 20:19:39.223368 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:39.223174 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d6bba-84fb4fb64rcj95" event={"ID":"f31414c5-bc04-42de-a73a-2bf16f64610f","Type":"ContainerDied","Data":"997a0df2f7c280912559225436eabad6c785cae31b768df881cd68464d6b4744"} Oct 30 20:19:39.223368 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:39.223230 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5d6bba-84fb4fb64rcj95" event={"ID":"f31414c5-bc04-42de-a73a-2bf16f64610f","Type":"ContainerDied","Data":"217198b991add201247711bf27f7a472ca08ccc4b53b038859f2c92fd0b12a51"} Oct 30 20:19:39.223368 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:39.223252 5088 scope.go:122] "RemoveContainer" containerID="997a0df2f7c280912559225436eabad6c785cae31b768df881cd68464d6b4744" Oct 30 20:19:39.232405 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:39.232382 5088 scope.go:122] "RemoveContainer" containerID="997a0df2f7c280912559225436eabad6c785cae31b768df881cd68464d6b4744" Oct 30 20:19:39.232737 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:19:39.232716 5088 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"997a0df2f7c280912559225436eabad6c785cae31b768df881cd68464d6b4744\": container with ID starting with 997a0df2f7c280912559225436eabad6c785cae31b768df881cd68464d6b4744 not found: ID does not exist" containerID="997a0df2f7c280912559225436eabad6c785cae31b768df881cd68464d6b4744" Oct 30 20:19:39.232822 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:39.232746 5088 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"997a0df2f7c280912559225436eabad6c785cae31b768df881cd68464d6b4744"} err="failed to get container status \"997a0df2f7c280912559225436eabad6c785cae31b768df881cd68464d6b4744\": rpc error: code = NotFound desc = could not find container \"997a0df2f7c280912559225436eabad6c785cae31b768df881cd68464d6b4744\": container with ID starting with 997a0df2f7c280912559225436eabad6c785cae31b768df881cd68464d6b4744 not found: ID does not exist" Oct 30 20:19:39.284793 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:39.284753 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5d6bba-84fb4fb64rcj95"] Oct 30 20:19:39.289964 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:39.289934 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5d6bba-84fb4fb64rcj95"] Oct 30 20:19:40.738743 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:40.738708 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f31414c5-bc04-42de-a73a-2bf16f64610f" path="/var/lib/kubelet/pods/f31414c5-bc04-42de-a73a-2bf16f64610f/volumes" Oct 30 20:19:45.730738 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:45.730701 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:19:49.325782 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:49.325736 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-c2tb9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:19:49.326449 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:19:49.326395 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-c2tb9" podUID="ce0a1543-79bc-440d-ad18-c045ac745786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:19:52.436856 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:19:52.436822 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:20:07.171353 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:07.171319 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:20:08.287535 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:08.287481 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l722w"] Oct 30 20:20:08.288230 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:08.288203 5088 cpu_manager.go:414] "containerMap: removing container" podUID="f31414c5-bc04-42de-a73a-2bf16f64610f" containerName="payload-processing" Oct 30 20:20:08.288334 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:08.288230 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f31414c5-bc04-42de-a73a-2bf16f64610f" containerName="payload-processing" Oct 30 20:20:08.288334 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:08.288287 5088 cpu_manager.go:414] "containerMap: removing container" podUID="61db385d-60af-43d0-8628-66e5d990663a" containerName="revoke-keys" Oct 30 20:20:08.288334 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:08.288297 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61db385d-60af-43d0-8628-66e5d990663a" containerName="revoke-keys" Oct 30 20:20:08.288514 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:08.288416 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="61db385d-60af-43d0-8628-66e5d990663a" containerName="revoke-keys" Oct 30 20:20:08.288514 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:08.288436 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="f31414c5-bc04-42de-a73a-2bf16f64610f" containerName="payload-processing" Oct 30 20:20:08.292137 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:08.292111 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l722w" Oct 30 20:20:08.314179 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:08.314128 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l722w"] Oct 30 20:20:08.322312 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:08.322266 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:20:08.322730 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:08.322263 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:20:08.322908 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:08.322519 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:20:08.326718 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:08.326679 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:20:08.406704 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:08.406662 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abe392f6-e542-47eb-bcea-56650ab0e0f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-l722w\" (UID: \"abe392f6-e542-47eb-bcea-56650ab0e0f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l722w" Oct 30 20:20:08.406994 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:08.406972 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abe392f6-e542-47eb-bcea-56650ab0e0f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-l722w\" (UID: \"abe392f6-e542-47eb-bcea-56650ab0e0f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l722w" Oct 30 20:20:08.407193 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:08.407176 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cwq4\" (UniqueName: \"kubernetes.io/projected/abe392f6-e542-47eb-bcea-56650ab0e0f5-kube-api-access-9cwq4\") pod \"maas-api-revoke-sub-452b6d7b-l722w\" (UID: \"abe392f6-e542-47eb-bcea-56650ab0e0f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l722w" Oct 30 20:20:08.508988 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:08.508927 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abe392f6-e542-47eb-bcea-56650ab0e0f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-l722w\" (UID: \"abe392f6-e542-47eb-bcea-56650ab0e0f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l722w" Oct 30 20:20:08.509235 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:08.509013 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abe392f6-e542-47eb-bcea-56650ab0e0f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-l722w\" (UID: \"abe392f6-e542-47eb-bcea-56650ab0e0f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l722w" Oct 30 20:20:08.509235 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:08.509109 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cwq4\" (UniqueName: \"kubernetes.io/projected/abe392f6-e542-47eb-bcea-56650ab0e0f5-kube-api-access-9cwq4\") pod \"maas-api-revoke-sub-452b6d7b-l722w\" (UID: \"abe392f6-e542-47eb-bcea-56650ab0e0f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l722w" Oct 30 20:20:08.510360 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:08.510327 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abe392f6-e542-47eb-bcea-56650ab0e0f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-l722w\" (UID: \"abe392f6-e542-47eb-bcea-56650ab0e0f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l722w" Oct 30 20:20:08.512387 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:08.512334 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abe392f6-e542-47eb-bcea-56650ab0e0f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-l722w\" (UID: \"abe392f6-e542-47eb-bcea-56650ab0e0f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l722w" Oct 30 20:20:08.537275 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:08.537231 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cwq4\" (UniqueName: \"kubernetes.io/projected/abe392f6-e542-47eb-bcea-56650ab0e0f5-kube-api-access-9cwq4\") pod \"maas-api-revoke-sub-452b6d7b-l722w\" (UID: \"abe392f6-e542-47eb-bcea-56650ab0e0f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l722w" Oct 30 20:20:08.616896 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:08.616794 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l722w" Oct 30 20:20:08.997839 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:08.997802 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l722w"] Oct 30 20:20:09.437094 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:09.437053 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l722w" event={"ID":"abe392f6-e542-47eb-bcea-56650ab0e0f5","Type":"ContainerStarted","Data":"cfef8ef7410a137ec1d0426fb5f54f903598c4cbe43ae7d7c60a5d60fc0112e8"} Oct 30 20:20:10.473568 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:10.473469 5088 generic.go:356] "Generic (PLEG): container finished" podID="abe392f6-e542-47eb-bcea-56650ab0e0f5" containerID="8af49fee6ffe062a91be82d07bcd8860dcf4c0ecd549cdee5ec82b153c059819" exitCode=0 Oct 30 20:20:10.474371 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:10.474342 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l722w" event={"ID":"abe392f6-e542-47eb-bcea-56650ab0e0f5","Type":"ContainerDied","Data":"8af49fee6ffe062a91be82d07bcd8860dcf4c0ecd549cdee5ec82b153c059819"} Oct 30 20:20:11.624690 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:11.624664 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l722w" Oct 30 20:20:11.695886 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:11.695842 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abe392f6-e542-47eb-bcea-56650ab0e0f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abe392f6-e542-47eb-bcea-56650ab0e0f5-maas-api-xks-ca\") pod \"abe392f6-e542-47eb-bcea-56650ab0e0f5\" (UID: \"abe392f6-e542-47eb-bcea-56650ab0e0f5\") " Oct 30 20:20:11.696090 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:11.695903 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abe392f6-e542-47eb-bcea-56650ab0e0f5-kube-api-access-9cwq4\" (UniqueName: \"kubernetes.io/projected/abe392f6-e542-47eb-bcea-56650ab0e0f5-kube-api-access-9cwq4\") pod \"abe392f6-e542-47eb-bcea-56650ab0e0f5\" (UID: \"abe392f6-e542-47eb-bcea-56650ab0e0f5\") " Oct 30 20:20:11.696090 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:11.696019 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abe392f6-e542-47eb-bcea-56650ab0e0f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abe392f6-e542-47eb-bcea-56650ab0e0f5-maas-api-openshift-ca\") pod \"abe392f6-e542-47eb-bcea-56650ab0e0f5\" (UID: \"abe392f6-e542-47eb-bcea-56650ab0e0f5\") " Oct 30 20:20:11.696459 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:11.696424 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abe392f6-e542-47eb-bcea-56650ab0e0f5-maas-api-openshift-ca" pod "abe392f6-e542-47eb-bcea-56650ab0e0f5" (UID: "abe392f6-e542-47eb-bcea-56650ab0e0f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:20:11.698233 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:11.698200 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abe392f6-e542-47eb-bcea-56650ab0e0f5-maas-api-xks-ca" pod "abe392f6-e542-47eb-bcea-56650ab0e0f5" (UID: "abe392f6-e542-47eb-bcea-56650ab0e0f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:20:11.698363 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:11.698241 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abe392f6-e542-47eb-bcea-56650ab0e0f5-kube-api-access-9cwq4" pod "abe392f6-e542-47eb-bcea-56650ab0e0f5" (UID: "abe392f6-e542-47eb-bcea-56650ab0e0f5"). InnerVolumeSpecName "kube-api-access-9cwq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:20:11.797853 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:11.797816 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abe392f6-e542-47eb-bcea-56650ab0e0f5-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:20:11.797853 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:11.797846 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cwq4\" (UniqueName: \"kubernetes.io/projected/abe392f6-e542-47eb-bcea-56650ab0e0f5-kube-api-access-9cwq4\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:20:11.797853 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:11.797858 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abe392f6-e542-47eb-bcea-56650ab0e0f5-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:20:12.488526 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:12.488488 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l722w" event={"ID":"abe392f6-e542-47eb-bcea-56650ab0e0f5","Type":"ContainerDied","Data":"cfef8ef7410a137ec1d0426fb5f54f903598c4cbe43ae7d7c60a5d60fc0112e8"} Oct 30 20:20:12.488526 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:12.488521 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l722w" Oct 30 20:20:12.488870 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:12.488548 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfef8ef7410a137ec1d0426fb5f54f903598c4cbe43ae7d7c60a5d60fc0112e8" Oct 30 20:20:21.335936 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:21.335898 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:20:22.039028 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:22.038994 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7073189-wxjzb"] Oct 30 20:20:22.052832 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:22.052366 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7073189-wxjzb"] Oct 30 20:20:22.449858 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:20:22.449757 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:20:22.743033 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:22.742981 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="421a34a5-84d2-499a-9a30-e3d391c3ab45" path="/var/lib/kubelet/pods/421a34a5-84d2-499a-9a30-e3d391c3ab45/volumes" Oct 30 20:20:24.538783 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:24.538733 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lnxpq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:20:24.539789 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:24.539430 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" podUID="f7ff5ff7-5866-473f-9055-e8c74f9380bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:20:24.570969 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:24.570697 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-ms5th container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:20:24.570969 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:24.570790 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" podUID="4045e4a8-b80c-4b45-96b7-e9e4203ebe5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:20:26.698806 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:26.698761 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7qhzl"] Oct 30 20:20:26.699506 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:26.699480 5088 cpu_manager.go:414] "containerMap: removing container" podUID="abe392f6-e542-47eb-bcea-56650ab0e0f5" containerName="revoke-keys" Oct 30 20:20:26.699596 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:26.699508 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abe392f6-e542-47eb-bcea-56650ab0e0f5" containerName="revoke-keys" Oct 30 20:20:26.699719 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:26.699701 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="abe392f6-e542-47eb-bcea-56650ab0e0f5" containerName="revoke-keys" Oct 30 20:20:26.703781 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:26.703759 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7qhzl" Oct 30 20:20:26.730857 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:26.730827 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:20:26.735821 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:26.735798 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:20:26.735985 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:26.735836 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:20:26.736092 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:26.735893 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:20:26.752590 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:26.752547 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7qhzl"] Oct 30 20:20:26.828808 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:26.828763 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhv6v\" (UniqueName: \"kubernetes.io/projected/52cacfd0-e445-486b-9e86-1ccace51ff16-kube-api-access-fhv6v\") pod \"maas-api-revoke-sub-3e31a506-7qhzl\" (UID: \"52cacfd0-e445-486b-9e86-1ccace51ff16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7qhzl" Oct 30 20:20:26.829011 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:26.828873 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52cacfd0-e445-486b-9e86-1ccace51ff16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-7qhzl\" (UID: \"52cacfd0-e445-486b-9e86-1ccace51ff16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7qhzl" Oct 30 20:20:26.829011 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:26.828985 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52cacfd0-e445-486b-9e86-1ccace51ff16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-7qhzl\" (UID: \"52cacfd0-e445-486b-9e86-1ccace51ff16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7qhzl" Oct 30 20:20:26.929976 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:26.929933 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52cacfd0-e445-486b-9e86-1ccace51ff16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-7qhzl\" (UID: \"52cacfd0-e445-486b-9e86-1ccace51ff16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7qhzl" Oct 30 20:20:26.930206 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:26.930031 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52cacfd0-e445-486b-9e86-1ccace51ff16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-7qhzl\" (UID: \"52cacfd0-e445-486b-9e86-1ccace51ff16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7qhzl" Oct 30 20:20:26.930206 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:26.930165 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhv6v\" (UniqueName: \"kubernetes.io/projected/52cacfd0-e445-486b-9e86-1ccace51ff16-kube-api-access-fhv6v\") pod \"maas-api-revoke-sub-3e31a506-7qhzl\" (UID: \"52cacfd0-e445-486b-9e86-1ccace51ff16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7qhzl" Oct 30 20:20:26.931005 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:26.930975 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52cacfd0-e445-486b-9e86-1ccace51ff16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-7qhzl\" (UID: \"52cacfd0-e445-486b-9e86-1ccace51ff16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7qhzl" Oct 30 20:20:26.933488 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:26.933453 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52cacfd0-e445-486b-9e86-1ccace51ff16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-7qhzl\" (UID: \"52cacfd0-e445-486b-9e86-1ccace51ff16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7qhzl" Oct 30 20:20:26.959018 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:26.958936 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhv6v\" (UniqueName: \"kubernetes.io/projected/52cacfd0-e445-486b-9e86-1ccace51ff16-kube-api-access-fhv6v\") pod \"maas-api-revoke-sub-3e31a506-7qhzl\" (UID: \"52cacfd0-e445-486b-9e86-1ccace51ff16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7qhzl" Oct 30 20:20:27.022876 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:27.022834 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7qhzl" Oct 30 20:20:27.396636 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:27.396400 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7qhzl"] Oct 30 20:20:27.623473 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:27.623415 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7qhzl" event={"ID":"52cacfd0-e445-486b-9e86-1ccace51ff16","Type":"ContainerStarted","Data":"a830c8c5f24701ec788f7c6c482ed50b7d0c8dd74cbac1171e13273fa1ab668e"} Oct 30 20:20:28.032006 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:28.031972 5088 scope.go:122] "RemoveContainer" containerID="398fb7cfc4e31f8a363dc1d552c0ab7097fd7b3d43225b784fcdc30d40535840" Oct 30 20:20:28.202430 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:28.202006 5088 scope.go:122] "RemoveContainer" containerID="55f6becc4110005ba46b39f027f66437a9d5f58a3b1577cf3944209150e73a09" Oct 30 20:20:28.280790 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:28.280397 5088 scope.go:122] "RemoveContainer" containerID="3e5a5f548c077e1f54c95c455d71db3b59337570cbd571b5e8dcf04d38c0791c" Oct 30 20:20:28.658120 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:28.658010 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7qhzl" event={"ID":"52cacfd0-e445-486b-9e86-1ccace51ff16","Type":"ContainerStarted","Data":"3ed2921870eb7e10f39f3b6cc3b2e04107b4966970da891dbe819658154311be"} Oct 30 20:20:28.720859 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:28.720787 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7qhzl" podStartSLOduration=2.720754383 podStartE2EDuration="2.720754383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:20:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:20:28.720166407 +0000 UTC m=+1990.132880187" watchObservedRunningTime="2026-10-30 20:20:28.720754383 +0000 UTC m=+1990.133468163" Oct 30 20:20:29.420897 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:29.420867 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:20:29.664323 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:29.664286 5088 generic.go:356] "Generic (PLEG): container finished" podID="52cacfd0-e445-486b-9e86-1ccace51ff16" containerID="3ed2921870eb7e10f39f3b6cc3b2e04107b4966970da891dbe819658154311be" exitCode=0 Oct 30 20:20:29.664496 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:29.664368 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7qhzl" event={"ID":"52cacfd0-e445-486b-9e86-1ccace51ff16","Type":"ContainerDied","Data":"3ed2921870eb7e10f39f3b6cc3b2e04107b4966970da891dbe819658154311be"} Oct 30 20:20:30.813219 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:30.813196 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7qhzl" Oct 30 20:20:30.825407 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:30.825381 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52cacfd0-e445-486b-9e86-1ccace51ff16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52cacfd0-e445-486b-9e86-1ccace51ff16-maas-api-openshift-ca\") pod \"52cacfd0-e445-486b-9e86-1ccace51ff16\" (UID: \"52cacfd0-e445-486b-9e86-1ccace51ff16\") " Oct 30 20:20:30.825530 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:30.825460 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52cacfd0-e445-486b-9e86-1ccace51ff16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52cacfd0-e445-486b-9e86-1ccace51ff16-maas-api-xks-ca\") pod \"52cacfd0-e445-486b-9e86-1ccace51ff16\" (UID: \"52cacfd0-e445-486b-9e86-1ccace51ff16\") " Oct 30 20:20:30.825530 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:30.825522 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52cacfd0-e445-486b-9e86-1ccace51ff16-kube-api-access-fhv6v\" (UniqueName: \"kubernetes.io/projected/52cacfd0-e445-486b-9e86-1ccace51ff16-kube-api-access-fhv6v\") pod \"52cacfd0-e445-486b-9e86-1ccace51ff16\" (UID: \"52cacfd0-e445-486b-9e86-1ccace51ff16\") " Oct 30 20:20:30.825804 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:30.825777 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52cacfd0-e445-486b-9e86-1ccace51ff16-maas-api-openshift-ca" pod "52cacfd0-e445-486b-9e86-1ccace51ff16" (UID: "52cacfd0-e445-486b-9e86-1ccace51ff16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:20:30.826009 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:30.825988 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52cacfd0-e445-486b-9e86-1ccace51ff16-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:20:30.827473 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:30.827451 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52cacfd0-e445-486b-9e86-1ccace51ff16-maas-api-xks-ca" pod "52cacfd0-e445-486b-9e86-1ccace51ff16" (UID: "52cacfd0-e445-486b-9e86-1ccace51ff16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:20:30.827720 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:30.827702 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52cacfd0-e445-486b-9e86-1ccace51ff16-kube-api-access-fhv6v" pod "52cacfd0-e445-486b-9e86-1ccace51ff16" (UID: "52cacfd0-e445-486b-9e86-1ccace51ff16"). InnerVolumeSpecName "kube-api-access-fhv6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:20:30.926890 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:30.926818 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhv6v\" (UniqueName: \"kubernetes.io/projected/52cacfd0-e445-486b-9e86-1ccace51ff16-kube-api-access-fhv6v\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:20:30.926890 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:30.926842 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52cacfd0-e445-486b-9e86-1ccace51ff16-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:20:31.676213 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:31.676182 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7qhzl" Oct 30 20:20:31.676397 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:31.676210 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7qhzl" event={"ID":"52cacfd0-e445-486b-9e86-1ccace51ff16","Type":"ContainerDied","Data":"a830c8c5f24701ec788f7c6c482ed50b7d0c8dd74cbac1171e13273fa1ab668e"} Oct 30 20:20:31.676397 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:31.676253 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a830c8c5f24701ec788f7c6c482ed50b7d0c8dd74cbac1171e13273fa1ab668e" Oct 30 20:20:37.941140 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:37.941098 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:20:39.543767 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:39.543727 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lnxpq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:20:39.544459 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:39.544423 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lnxpq" podUID="f7ff5ff7-5866-473f-9055-e8c74f9380bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:20:39.580637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:39.579726 5088 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-ms5th container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:20:39.580637 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:39.579817 5088 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-ms5th" podUID="4045e4a8-b80c-4b45-96b7-e9e4203ebe5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:20:42.565925 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:42.565891 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:20:49.315137 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:49.315104 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dfa9a98-ssmmz"] Oct 30 20:20:49.315576 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:49.315561 5088 cpu_manager.go:414] "containerMap: removing container" podUID="52cacfd0-e445-486b-9e86-1ccace51ff16" containerName="revoke-keys" Oct 30 20:20:49.315633 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:49.315577 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52cacfd0-e445-486b-9e86-1ccace51ff16" containerName="revoke-keys" Oct 30 20:20:49.315707 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:49.315697 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="52cacfd0-e445-486b-9e86-1ccace51ff16" containerName="revoke-keys" Oct 30 20:20:49.318958 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:49.318936 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dfa9a98-ssmmz" Oct 30 20:20:49.337258 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:49.337232 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:20:49.350355 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:49.350316 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:20:49.350516 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:49.350420 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-scs2w\"" Oct 30 20:20:49.350516 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:49.350500 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:20:49.352967 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:49.352940 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dfa9a98-ssmmz"] Oct 30 20:20:49.407839 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:49.407809 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncsk6\" (UniqueName: \"kubernetes.io/projected/7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a-kube-api-access-ncsk6\") pod \"maas-api-revoke-sub-7dfa9a98-ssmmz\" (UID: \"7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dfa9a98-ssmmz" Oct 30 20:20:49.407989 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:49.407962 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7dfa9a98-ssmmz\" (UID: \"7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dfa9a98-ssmmz" Oct 30 20:20:49.408032 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:49.407999 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7dfa9a98-ssmmz\" (UID: \"7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dfa9a98-ssmmz" Oct 30 20:20:49.508678 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:49.508648 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7dfa9a98-ssmmz\" (UID: \"7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dfa9a98-ssmmz" Oct 30 20:20:49.508840 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:49.508684 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7dfa9a98-ssmmz\" (UID: \"7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dfa9a98-ssmmz" Oct 30 20:20:49.508840 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:49.508740 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncsk6\" (UniqueName: \"kubernetes.io/projected/7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a-kube-api-access-ncsk6\") pod \"maas-api-revoke-sub-7dfa9a98-ssmmz\" (UID: \"7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dfa9a98-ssmmz" Oct 30 20:20:49.509463 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:49.509442 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7dfa9a98-ssmmz\" (UID: \"7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dfa9a98-ssmmz" Oct 30 20:20:49.511042 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:49.511025 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7dfa9a98-ssmmz\" (UID: \"7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dfa9a98-ssmmz" Oct 30 20:20:49.536282 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:49.536258 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncsk6\" (UniqueName: \"kubernetes.io/projected/7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a-kube-api-access-ncsk6\") pod \"maas-api-revoke-sub-7dfa9a98-ssmmz\" (UID: \"7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dfa9a98-ssmmz" Oct 30 20:20:49.632893 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:49.632805 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dfa9a98-ssmmz" Oct 30 20:20:49.936068 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:49.936040 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dfa9a98-ssmmz"] Oct 30 20:20:49.937253 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:20:49.937227 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7759fcd3_db9f_4da4_a2d9_0a6b4b14f11a.slice/crio-9688bfd046b2ad7a58fd722477b28e2cb7a23deaf555294dc437660c6811835b WatchSource:0}: Error finding container 9688bfd046b2ad7a58fd722477b28e2cb7a23deaf555294dc437660c6811835b: Status 404 returned error can't find the container with id 9688bfd046b2ad7a58fd722477b28e2cb7a23deaf555294dc437660c6811835b Oct 30 20:20:50.842824 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:50.842792 5088 generic.go:356] "Generic (PLEG): container finished" podID="7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a" containerID="e524f63d89bd1bb5ee43c23aec3f7018fa668dff824a032be229d666ad2499c9" exitCode=0 Oct 30 20:20:50.843195 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:50.842831 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dfa9a98-ssmmz" event={"ID":"7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a","Type":"ContainerDied","Data":"e524f63d89bd1bb5ee43c23aec3f7018fa668dff824a032be229d666ad2499c9"} Oct 30 20:20:50.843195 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:50.842891 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dfa9a98-ssmmz" event={"ID":"7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a","Type":"ContainerStarted","Data":"9688bfd046b2ad7a58fd722477b28e2cb7a23deaf555294dc437660c6811835b"} Oct 30 20:20:51.994567 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:51.994542 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dfa9a98-ssmmz" Oct 30 20:20:52.032823 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:52.032800 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a-kube-api-access-ncsk6\" (UniqueName: \"kubernetes.io/projected/7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a-kube-api-access-ncsk6\") pod \"7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a\" (UID: \"7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a\") " Oct 30 20:20:52.032959 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:52.032893 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a-maas-api-xks-ca\") pod \"7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a\" (UID: \"7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a\") " Oct 30 20:20:52.033031 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:52.032960 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a-maas-api-openshift-ca\") pod \"7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a\" (UID: \"7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a\") " Oct 30 20:20:52.033334 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:52.033310 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a-maas-api-openshift-ca" pod "7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a" (UID: "7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:20:52.034865 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:52.034834 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a-kube-api-access-ncsk6" pod "7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a" (UID: "7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a"). InnerVolumeSpecName "kube-api-access-ncsk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:20:52.034953 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:52.034873 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a-maas-api-xks-ca" pod "7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a" (UID: "7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:20:52.134563 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:52.134486 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a-maas-api-xks-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:20:52.134563 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:52.134515 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a-maas-api-openshift-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:20:52.134563 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:52.134525 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncsk6\" (UniqueName: \"kubernetes.io/projected/7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a-kube-api-access-ncsk6\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:20:52.437689 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:20:52.437582 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:20:52.855305 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:52.855265 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dfa9a98-ssmmz" event={"ID":"7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a","Type":"ContainerDied","Data":"9688bfd046b2ad7a58fd722477b28e2cb7a23deaf555294dc437660c6811835b"} Oct 30 20:20:52.855305 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:52.855292 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dfa9a98-ssmmz" Oct 30 20:20:52.855516 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:52.855317 5088 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9688bfd046b2ad7a58fd722477b28e2cb7a23deaf555294dc437660c6811835b" Oct 30 20:20:55.045989 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:55.045956 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c8l"] Oct 30 20:20:55.059248 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:55.059219 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c8l"] Oct 30 20:20:56.739011 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:20:56.738977 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="262a8036-5981-483d-bd53-7d11f4dbfb68" path="/var/lib/kubelet/pods/262a8036-5981-483d-bd53-7d11f4dbfb68/volumes" Oct 30 20:21:00.461490 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:00.461455 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:21:01.902039 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:01.901950 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-8t8mg"] Oct 30 20:21:01.902444 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:01.902414 5088 cpu_manager.go:414] "containerMap: removing container" podUID="7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a" containerName="revoke-keys" Oct 30 20:21:01.902444 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:01.902425 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a" containerName="revoke-keys" Oct 30 20:21:01.902569 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:01.902516 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a" containerName="revoke-keys" Oct 30 20:21:01.906923 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:01.906901 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-8t8mg" Oct 30 20:21:01.945922 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:01.945883 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:21:01.945922 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:01.945908 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:21:01.946169 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:01.945928 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 20:21:01.946169 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:01.946012 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7wqnw\"" Oct 30 20:21:01.951058 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:01.951023 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 20:21:01.954985 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:01.954956 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-8t8mg"] Oct 30 20:21:02.040880 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:02.040837 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khmrv\" (UniqueName: \"kubernetes.io/projected/dc123ad4-ca52-4982-9167-f57f94427806-kube-api-access-khmrv\") pod \"maas-api-5c48877c75-8t8mg\" (UID: \"dc123ad4-ca52-4982-9167-f57f94427806\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-8t8mg" Oct 30 20:21:02.041109 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:02.040917 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc123ad4-ca52-4982-9167-f57f94427806-openshift-service-ca\") pod \"maas-api-5c48877c75-8t8mg\" (UID: \"dc123ad4-ca52-4982-9167-f57f94427806\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-8t8mg" Oct 30 20:21:02.041109 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:02.040946 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc123ad4-ca52-4982-9167-f57f94427806-maas-api-tls\") pod \"maas-api-5c48877c75-8t8mg\" (UID: \"dc123ad4-ca52-4982-9167-f57f94427806\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-8t8mg" Oct 30 20:21:02.041109 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:02.040970 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc123ad4-ca52-4982-9167-f57f94427806-metrics-certs\") pod \"maas-api-5c48877c75-8t8mg\" (UID: \"dc123ad4-ca52-4982-9167-f57f94427806\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-8t8mg" Oct 30 20:21:02.142775 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:02.142737 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khmrv\" (UniqueName: \"kubernetes.io/projected/dc123ad4-ca52-4982-9167-f57f94427806-kube-api-access-khmrv\") pod \"maas-api-5c48877c75-8t8mg\" (UID: \"dc123ad4-ca52-4982-9167-f57f94427806\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-8t8mg" Oct 30 20:21:02.143000 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:02.142792 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc123ad4-ca52-4982-9167-f57f94427806-openshift-service-ca\") pod \"maas-api-5c48877c75-8t8mg\" (UID: \"dc123ad4-ca52-4982-9167-f57f94427806\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-8t8mg" Oct 30 20:21:02.143000 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:02.142813 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc123ad4-ca52-4982-9167-f57f94427806-maas-api-tls\") pod \"maas-api-5c48877c75-8t8mg\" (UID: \"dc123ad4-ca52-4982-9167-f57f94427806\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-8t8mg" Oct 30 20:21:02.143000 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:02.142827 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc123ad4-ca52-4982-9167-f57f94427806-metrics-certs\") pod \"maas-api-5c48877c75-8t8mg\" (UID: \"dc123ad4-ca52-4982-9167-f57f94427806\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-8t8mg" Oct 30 20:21:02.143571 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:02.143547 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc123ad4-ca52-4982-9167-f57f94427806-openshift-service-ca\") pod \"maas-api-5c48877c75-8t8mg\" (UID: \"dc123ad4-ca52-4982-9167-f57f94427806\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-8t8mg" Oct 30 20:21:02.145395 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:02.145370 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc123ad4-ca52-4982-9167-f57f94427806-metrics-certs\") pod \"maas-api-5c48877c75-8t8mg\" (UID: \"dc123ad4-ca52-4982-9167-f57f94427806\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-8t8mg" Oct 30 20:21:02.145484 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:02.145454 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc123ad4-ca52-4982-9167-f57f94427806-maas-api-tls\") pod \"maas-api-5c48877c75-8t8mg\" (UID: \"dc123ad4-ca52-4982-9167-f57f94427806\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-8t8mg" Oct 30 20:21:02.169377 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:02.169320 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khmrv\" (UniqueName: \"kubernetes.io/projected/dc123ad4-ca52-4982-9167-f57f94427806-kube-api-access-khmrv\") pod \"maas-api-5c48877c75-8t8mg\" (UID: \"dc123ad4-ca52-4982-9167-f57f94427806\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-8t8mg" Oct 30 20:21:02.220848 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:02.220809 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-8t8mg" Oct 30 20:21:02.522953 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:02.522924 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-8t8mg"] Oct 30 20:21:02.526483 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:21:02.526456 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc123ad4_ca52_4982_9167_f57f94427806.slice/crio-84f7ada9118950d3df3334ab685b644fc4091f9c0330c673043b56c1b72ea9a5 WatchSource:0}: Error finding container 84f7ada9118950d3df3334ab685b644fc4091f9c0330c673043b56c1b72ea9a5: Status 404 returned error can't find the container with id 84f7ada9118950d3df3334ab685b644fc4091f9c0330c673043b56c1b72ea9a5 Oct 30 20:21:02.527864 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:02.527846 5088 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:21:02.916158 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:02.916121 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-8t8mg" event={"ID":"dc123ad4-ca52-4982-9167-f57f94427806","Type":"ContainerStarted","Data":"66c4754a8305ed34b0a40765f4010a8b5177452f7ff76208aa3c73da86636120"} Oct 30 20:21:02.916605 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:02.916183 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-8t8mg" Oct 30 20:21:02.916605 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:02.916199 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-8t8mg" event={"ID":"dc123ad4-ca52-4982-9167-f57f94427806","Type":"ContainerStarted","Data":"84f7ada9118950d3df3334ab685b644fc4091f9c0330c673043b56c1b72ea9a5"} Oct 30 20:21:02.964989 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:02.964937 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-8t8mg" podStartSLOduration=1.6783452159999999 podStartE2EDuration="1.964922135s" totalImagesPullingTime="286.576919ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:21:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:02.964055574 +0000 UTC m=+2024.376769353" watchObservedRunningTime="2026-10-30 20:21:02.964922135 +0000 UTC m=+2024.377635909" Oct 30 20:21:08.926773 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:08.926745 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-8t8mg" Oct 30 20:21:09.450433 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:09.450398 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-kz95z"] Oct 30 20:21:09.454320 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:09.454296 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-kz95z" Oct 30 20:21:09.476012 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:09.475981 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-kz95z"] Oct 30 20:21:09.617448 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:09.617413 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a25e58c3-e841-48d0-8342-9b43228dfa82-openshift-service-ca\") pod \"maas-api-595cc648c-kz95z\" (UID: \"a25e58c3-e841-48d0-8342-9b43228dfa82\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-kz95z" Oct 30 20:21:09.617664 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:09.617583 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a25e58c3-e841-48d0-8342-9b43228dfa82-metrics-certs\") pod \"maas-api-595cc648c-kz95z\" (UID: \"a25e58c3-e841-48d0-8342-9b43228dfa82\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-kz95z" Oct 30 20:21:09.617727 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:09.617694 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhgcd\" (UniqueName: \"kubernetes.io/projected/a25e58c3-e841-48d0-8342-9b43228dfa82-kube-api-access-dhgcd\") pod \"maas-api-595cc648c-kz95z\" (UID: \"a25e58c3-e841-48d0-8342-9b43228dfa82\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-kz95z" Oct 30 20:21:09.617727 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:09.617721 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a25e58c3-e841-48d0-8342-9b43228dfa82-maas-api-tls\") pod \"maas-api-595cc648c-kz95z\" (UID: \"a25e58c3-e841-48d0-8342-9b43228dfa82\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-kz95z" Oct 30 20:21:09.718501 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:09.718398 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a25e58c3-e841-48d0-8342-9b43228dfa82-openshift-service-ca\") pod \"maas-api-595cc648c-kz95z\" (UID: \"a25e58c3-e841-48d0-8342-9b43228dfa82\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-kz95z" Oct 30 20:21:09.718712 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:09.718541 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a25e58c3-e841-48d0-8342-9b43228dfa82-metrics-certs\") pod \"maas-api-595cc648c-kz95z\" (UID: \"a25e58c3-e841-48d0-8342-9b43228dfa82\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-kz95z" Oct 30 20:21:09.718712 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:09.718596 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhgcd\" (UniqueName: \"kubernetes.io/projected/a25e58c3-e841-48d0-8342-9b43228dfa82-kube-api-access-dhgcd\") pod \"maas-api-595cc648c-kz95z\" (UID: \"a25e58c3-e841-48d0-8342-9b43228dfa82\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-kz95z" Oct 30 20:21:09.718712 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:09.718645 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a25e58c3-e841-48d0-8342-9b43228dfa82-maas-api-tls\") pod \"maas-api-595cc648c-kz95z\" (UID: \"a25e58c3-e841-48d0-8342-9b43228dfa82\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-kz95z" Oct 30 20:21:09.719133 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:09.719109 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a25e58c3-e841-48d0-8342-9b43228dfa82-openshift-service-ca\") pod \"maas-api-595cc648c-kz95z\" (UID: \"a25e58c3-e841-48d0-8342-9b43228dfa82\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-kz95z" Oct 30 20:21:09.721014 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:09.720988 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a25e58c3-e841-48d0-8342-9b43228dfa82-maas-api-tls\") pod \"maas-api-595cc648c-kz95z\" (UID: \"a25e58c3-e841-48d0-8342-9b43228dfa82\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-kz95z" Oct 30 20:21:09.721197 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:09.721177 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a25e58c3-e841-48d0-8342-9b43228dfa82-metrics-certs\") pod \"maas-api-595cc648c-kz95z\" (UID: \"a25e58c3-e841-48d0-8342-9b43228dfa82\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-kz95z" Oct 30 20:21:09.745670 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:09.745602 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhgcd\" (UniqueName: \"kubernetes.io/projected/a25e58c3-e841-48d0-8342-9b43228dfa82-kube-api-access-dhgcd\") pod \"maas-api-595cc648c-kz95z\" (UID: \"a25e58c3-e841-48d0-8342-9b43228dfa82\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-kz95z" Oct 30 20:21:09.770076 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:09.770039 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-kz95z" Oct 30 20:21:10.075295 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:10.075267 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-kz95z"] Oct 30 20:21:10.076938 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:21:10.076909 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda25e58c3_e841_48d0_8342_9b43228dfa82.slice/crio-b378cb0c110bc9386f74ca445bd8acc1dfcfb24d3355820aa86777711b5f07ca WatchSource:0}: Error finding container b378cb0c110bc9386f74ca445bd8acc1dfcfb24d3355820aa86777711b5f07ca: Status 404 returned error can't find the container with id b378cb0c110bc9386f74ca445bd8acc1dfcfb24d3355820aa86777711b5f07ca Oct 30 20:21:10.457575 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:10.457546 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:21:10.986050 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:10.986001 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-kz95z" event={"ID":"a25e58c3-e841-48d0-8342-9b43228dfa82","Type":"ContainerStarted","Data":"f19b139dc8b65ab6401f81419cc9145d13fed51bbd98fceabcf8e6d9de4f7185"} Oct 30 20:21:10.986266 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:10.986110 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-kz95z" event={"ID":"a25e58c3-e841-48d0-8342-9b43228dfa82","Type":"ContainerStarted","Data":"b378cb0c110bc9386f74ca445bd8acc1dfcfb24d3355820aa86777711b5f07ca"} Oct 30 20:21:10.986432 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:10.986413 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-kz95z" Oct 30 20:21:11.045952 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:11.045877 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-kz95z" podStartSLOduration=1.737390624 podStartE2EDuration="2.045857195s" totalImagesPullingTime="308.466571ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:21:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:11.042237158 +0000 UTC m=+2032.454950938" watchObservedRunningTime="2026-10-30 20:21:11.045857195 +0000 UTC m=+2032.458570977" Oct 30 20:21:17.045527 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:17.045495 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-kz95z" Oct 30 20:21:17.138674 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:17.138637 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-8t8mg"] Oct 30 20:21:17.138905 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:17.138858 5088 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-8t8mg" podUID="dc123ad4-ca52-4982-9167-f57f94427806" containerName="maas-api" containerID="cri-o://66c4754a8305ed34b0a40765f4010a8b5177452f7ff76208aa3c73da86636120" gracePeriod=30 Oct 30 20:21:17.402326 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:17.402304 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-8t8mg" Oct 30 20:21:17.458891 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:17.458864 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc123ad4-ca52-4982-9167-f57f94427806-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc123ad4-ca52-4982-9167-f57f94427806-openshift-service-ca\") pod \"dc123ad4-ca52-4982-9167-f57f94427806\" (UID: \"dc123ad4-ca52-4982-9167-f57f94427806\") " Oct 30 20:21:17.459045 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:17.458920 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc123ad4-ca52-4982-9167-f57f94427806-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc123ad4-ca52-4982-9167-f57f94427806-metrics-certs\") pod \"dc123ad4-ca52-4982-9167-f57f94427806\" (UID: \"dc123ad4-ca52-4982-9167-f57f94427806\") " Oct 30 20:21:17.459045 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:17.458967 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc123ad4-ca52-4982-9167-f57f94427806-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc123ad4-ca52-4982-9167-f57f94427806-maas-api-tls\") pod \"dc123ad4-ca52-4982-9167-f57f94427806\" (UID: \"dc123ad4-ca52-4982-9167-f57f94427806\") " Oct 30 20:21:17.459045 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:17.459043 5088 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc123ad4-ca52-4982-9167-f57f94427806-kube-api-access-khmrv\" (UniqueName: \"kubernetes.io/projected/dc123ad4-ca52-4982-9167-f57f94427806-kube-api-access-khmrv\") pod \"dc123ad4-ca52-4982-9167-f57f94427806\" (UID: \"dc123ad4-ca52-4982-9167-f57f94427806\") " Oct 30 20:21:17.459303 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:17.459273 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc123ad4-ca52-4982-9167-f57f94427806-openshift-service-ca" pod "dc123ad4-ca52-4982-9167-f57f94427806" (UID: "dc123ad4-ca52-4982-9167-f57f94427806"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:21:17.461497 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:17.461471 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc123ad4-ca52-4982-9167-f57f94427806-kube-api-access-khmrv" pod "dc123ad4-ca52-4982-9167-f57f94427806" (UID: "dc123ad4-ca52-4982-9167-f57f94427806"). InnerVolumeSpecName "kube-api-access-khmrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:21:17.461848 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:17.461820 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc123ad4-ca52-4982-9167-f57f94427806-metrics-certs" pod "dc123ad4-ca52-4982-9167-f57f94427806" (UID: "dc123ad4-ca52-4982-9167-f57f94427806"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:21:17.462262 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:17.462235 5088 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc123ad4-ca52-4982-9167-f57f94427806-maas-api-tls" pod "dc123ad4-ca52-4982-9167-f57f94427806" (UID: "dc123ad4-ca52-4982-9167-f57f94427806"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:21:17.560147 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:17.560061 5088 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc123ad4-ca52-4982-9167-f57f94427806-openshift-service-ca\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:21:17.560147 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:17.560093 5088 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc123ad4-ca52-4982-9167-f57f94427806-metrics-certs\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:21:17.560147 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:17.560102 5088 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc123ad4-ca52-4982-9167-f57f94427806-maas-api-tls\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:21:17.560147 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:17.560114 5088 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khmrv\" (UniqueName: \"kubernetes.io/projected/dc123ad4-ca52-4982-9167-f57f94427806-kube-api-access-khmrv\") on node \"ip-10-0-132-173.ec2.internal\" DevicePath \"\"" Oct 30 20:21:18.104138 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:18.104104 5088 generic.go:356] "Generic (PLEG): container finished" podID="dc123ad4-ca52-4982-9167-f57f94427806" containerID="66c4754a8305ed34b0a40765f4010a8b5177452f7ff76208aa3c73da86636120" exitCode=0 Oct 30 20:21:18.104672 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:18.104148 5088 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-8t8mg" Oct 30 20:21:18.104672 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:18.104192 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-8t8mg" event={"ID":"dc123ad4-ca52-4982-9167-f57f94427806","Type":"ContainerDied","Data":"66c4754a8305ed34b0a40765f4010a8b5177452f7ff76208aa3c73da86636120"} Oct 30 20:21:18.104672 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:18.104256 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-8t8mg" event={"ID":"dc123ad4-ca52-4982-9167-f57f94427806","Type":"ContainerDied","Data":"84f7ada9118950d3df3334ab685b644fc4091f9c0330c673043b56c1b72ea9a5"} Oct 30 20:21:18.104672 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:18.104287 5088 scope.go:122] "RemoveContainer" containerID="66c4754a8305ed34b0a40765f4010a8b5177452f7ff76208aa3c73da86636120" Oct 30 20:21:18.114597 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:18.114579 5088 scope.go:122] "RemoveContainer" containerID="66c4754a8305ed34b0a40765f4010a8b5177452f7ff76208aa3c73da86636120" Oct 30 20:21:18.114904 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:21:18.114886 5088 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66c4754a8305ed34b0a40765f4010a8b5177452f7ff76208aa3c73da86636120\": container with ID starting with 66c4754a8305ed34b0a40765f4010a8b5177452f7ff76208aa3c73da86636120 not found: ID does not exist" containerID="66c4754a8305ed34b0a40765f4010a8b5177452f7ff76208aa3c73da86636120" Oct 30 20:21:18.114953 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:18.114915 5088 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66c4754a8305ed34b0a40765f4010a8b5177452f7ff76208aa3c73da86636120"} err="failed to get container status \"66c4754a8305ed34b0a40765f4010a8b5177452f7ff76208aa3c73da86636120\": rpc error: code = NotFound desc = could not find container \"66c4754a8305ed34b0a40765f4010a8b5177452f7ff76208aa3c73da86636120\": container with ID starting with 66c4754a8305ed34b0a40765f4010a8b5177452f7ff76208aa3c73da86636120 not found: ID does not exist" Oct 30 20:21:18.171381 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:18.171354 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-8t8mg"] Oct 30 20:21:18.185050 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:18.185020 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-8t8mg"] Oct 30 20:21:18.739930 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:18.739895 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc123ad4-ca52-4982-9167-f57f94427806" path="/var/lib/kubelet/pods/dc123ad4-ca52-4982-9167-f57f94427806/volumes" Oct 30 20:21:18.930506 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:18.930476 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:21:22.436726 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:21:22.436690 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:21:28.460093 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:28.460060 5088 scope.go:122] "RemoveContainer" containerID="74bf49655ac0e2012136c0e50202ce8066c1c5562ef1f5c2afdb6256f52b1de3" Oct 30 20:21:32.738966 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:21:32.738933 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:21:52.436685 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:21:52.436646 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:22:00.555454 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:00.555423 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pdkrz"] Oct 30 20:22:13.226884 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:13.226847 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-774477d6c-wsdl5_4132b74d-b894-4c19-9333-201adb0ef1ad/manager/0.log" Oct 30 20:22:14.709604 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:14.709573 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f6c7b487f-g84tm_c4416f2d-d31c-4a91-b910-66ef37e203b5/copy-manifests/0.log" Oct 30 20:22:14.740559 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:14.740528 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f6c7b487f-g84tm_c4416f2d-d31c-4a91-b910-66ef37e203b5/manager/0.log" Oct 30 20:22:16.225664 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:16.225608 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-kz95z_a25e58c3-e841-48d0-8342-9b43228dfa82/maas-api/0.log" Oct 30 20:22:16.629992 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:16.629966 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-7qhzl_52cacfd0-e445-486b-9e86-1ccace51ff16/revoke-keys/0.log" Oct 30 20:22:16.810653 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:16.810602 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-l722w_abe392f6-e542-47eb-bcea-56650ab0e0f5/revoke-keys/0.log" Oct 30 20:22:17.209030 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:17.208994 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-7dfa9a98-ssmmz_7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a/revoke-keys/0.log" Oct 30 20:22:17.413822 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:17.413794 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b8b95186-ch52x_6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720/revoke-keys/0.log" Oct 30 20:22:17.609286 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:17.609226 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-lp2zb_7de7652b-0108-4745-90a4-5a31085b0409/revoke-keys/0.log" Oct 30 20:22:17.806537 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:17.806508 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-flq2j_d6beaf64-915d-4053-83a4-3362d28bbb91/revoke-keys/0.log" Oct 30 20:22:19.000840 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:19.000809 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qdj4h_da850825-59ab-480f-9047-afd7d2bdc19f/console-operator/1.log" Oct 30 20:22:19.005787 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:19.005760 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xpjpn_688d079a-932d-4f63-97da-5cd6222a03e1/ovn-acl-logging/0.log" Oct 30 20:22:19.007392 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:19.007370 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qdj4h_da850825-59ab-480f-9047-afd7d2bdc19f/console-operator/1.log" Oct 30 20:22:19.012021 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:19.012001 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xpjpn_688d079a-932d-4f63-97da-5cd6222a03e1/ovn-acl-logging/0.log" Oct 30 20:22:20.042512 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:20.042481 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-rq4t2_5a7ba990-22c5-4549-80a9-3a877255b7dd/manager/0.log" Oct 30 20:22:20.268847 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:20.268813 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-fvxph_335ef769-6b60-4648-bff4-ed5d0d1bbb2e/manager/0.log" Oct 30 20:22:20.917500 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:20.917469 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-pdkrz_a2a6ff3a-e8bf-4256-b860-89065d5116f7/limitador/0.log" Oct 30 20:22:21.132034 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:21.131999 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-m67x6_99574888-155b-41ad-a913-04105d65b7a0/manager/0.log" Oct 30 20:22:21.938633 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:21.938578 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6_825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1/istio-proxy/0.log" Oct 30 20:22:22.147275 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:22.147246 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-l9g8p_ed843517-0e6d-44b9-a6f9-4c6e686bad29/discovery/0.log" Oct 30 20:22:22.438372 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:22:22.438331 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:22:22.570008 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:22.569974 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f5c96996f-wsql6_e4e0fd61-e6e4-4d87-a607-ad397cc39697/kube-auth-proxy/0.log" Oct 30 20:22:22.771093 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:22.771056 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-bfxfk_4b460a5d-51f8-4694-8dd6-316f788bf8fc/istio-proxy/1.log" Oct 30 20:22:22.982377 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:22.982344 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-c2tb9_ce0a1543-79bc-440d-ad18-c045ac745786/istio-proxy/0.log" Oct 30 20:22:23.823246 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:23.823208 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-lnxpq_f7ff5ff7-5866-473f-9055-e8c74f9380bc/istio-proxy/0.log" Oct 30 20:22:24.039482 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:24.039447 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-ms5th_4045e4a8-b80c-4b45-96b7-e9e4203ebe5e/istio-proxy/0.log" Oct 30 20:22:25.286724 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:25.286693 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-79458f7dbc-vbb28_41534930-917f-44ec-bccd-86d6c6fc7c70/router/0.log" Oct 30 20:22:25.752783 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:25.752751 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-bfxfk_4b460a5d-51f8-4694-8dd6-316f788bf8fc/istio-proxy/0.log" Oct 30 20:22:26.311542 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:26.311504 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9_1fa4e381-a575-4bf9-a65d-33bdc3093587/main/0.log" Oct 30 20:22:26.724114 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:26.724018 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-wtlm7_419ffdb0-c223-4426-872a-50e3073812e4/main/0.log" Oct 30 20:22:26.937580 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:26.937547 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-f4snn_5ec64166-d2fc-47f6-96d1-48ea0ee1b70c/main/0.log" Oct 30 20:22:27.517941 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:27.517910 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2_ee73a969-6d25-490c-b943-2c9771cfa738/main/0.log" Oct 30 20:22:28.488185 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:28.488144 5088 scope.go:122] "RemoveContainer" containerID="bdf1a02543f77160dae0b8fb18ac193f74c2f106328899a16c1e1f98f8bce719" Oct 30 20:22:28.498887 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:28.498862 5088 scope.go:122] "RemoveContainer" containerID="fbad9fa17beec065baa10e57bcb0f0cd73c0a91a574d5d09b1c97a7e6ce55317" Oct 30 20:22:39.354928 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:39.354834 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-774477d6c-wsdl5_4132b74d-b894-4c19-9333-201adb0ef1ad/manager/0.log" Oct 30 20:22:40.784639 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:40.784579 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f6c7b487f-g84tm_c4416f2d-d31c-4a91-b910-66ef37e203b5/manager/0.log" Oct 30 20:22:40.814229 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:40.814199 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f6c7b487f-g84tm_c4416f2d-d31c-4a91-b910-66ef37e203b5/copy-manifests/0.log" Oct 30 20:22:41.815717 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:41.815684 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-kz95z_a25e58c3-e841-48d0-8342-9b43228dfa82/maas-api/0.log" Oct 30 20:22:42.205270 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:42.205191 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-7qhzl_52cacfd0-e445-486b-9e86-1ccace51ff16/revoke-keys/0.log" Oct 30 20:22:42.399120 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:42.399088 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-l722w_abe392f6-e542-47eb-bcea-56650ab0e0f5/revoke-keys/0.log" Oct 30 20:22:42.782770 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:42.782742 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-7dfa9a98-ssmmz_7759fcd3-db9f-4da4-a2d9-0a6b4b14f11a/revoke-keys/0.log" Oct 30 20:22:42.982061 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:42.982030 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b8b95186-ch52x_6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720/revoke-keys/0.log" Oct 30 20:22:43.163018 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:43.162906 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-lp2zb_7de7652b-0108-4745-90a4-5a31085b0409/revoke-keys/0.log" Oct 30 20:22:43.357730 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:43.357701 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-flq2j_d6beaf64-915d-4053-83a4-3362d28bbb91/revoke-keys/0.log" Oct 30 20:22:45.113970 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:45.113934 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-rq4t2_5a7ba990-22c5-4549-80a9-3a877255b7dd/manager/0.log" Oct 30 20:22:45.324782 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:45.324751 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-fvxph_335ef769-6b60-4648-bff4-ed5d0d1bbb2e/manager/0.log" Oct 30 20:22:45.953466 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:45.953435 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-pdkrz_a2a6ff3a-e8bf-4256-b860-89065d5116f7/limitador/0.log" Oct 30 20:22:46.036264 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:46.036221 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flq2j"] Oct 30 20:22:46.049745 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:46.049711 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flq2j"] Oct 30 20:22:46.164867 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:46.164831 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-m67x6_99574888-155b-41ad-a913-04105d65b7a0/manager/0.log" Oct 30 20:22:46.738810 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:46.738768 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6beaf64-915d-4053-83a4-3362d28bbb91" path="/var/lib/kubelet/pods/d6beaf64-915d-4053-83a4-3362d28bbb91/volumes" Oct 30 20:22:46.743836 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:46.743805 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6_825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1/istio-proxy/0.log" Oct 30 20:22:46.934670 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:46.934631 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-l9g8p_ed843517-0e6d-44b9-a6f9-4c6e686bad29/discovery/0.log" Oct 30 20:22:47.346594 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:47.346563 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f5c96996f-wsql6_e4e0fd61-e6e4-4d87-a607-ad397cc39697/kube-auth-proxy/0.log" Oct 30 20:22:47.539354 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:47.539321 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-bfxfk_4b460a5d-51f8-4694-8dd6-316f788bf8fc/istio-proxy/1.log" Oct 30 20:22:47.733176 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:47.733145 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-c2tb9_ce0a1543-79bc-440d-ad18-c045ac745786/istio-proxy/0.log" Oct 30 20:22:48.586398 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:48.586364 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-lnxpq_f7ff5ff7-5866-473f-9055-e8c74f9380bc/istio-proxy/0.log" Oct 30 20:22:48.798045 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:48.798012 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-ms5th_4045e4a8-b80c-4b45-96b7-e9e4203ebe5e/istio-proxy/0.log" Oct 30 20:22:50.010154 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:50.010126 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-79458f7dbc-vbb28_41534930-917f-44ec-bccd-86d6c6fc7c70/router/0.log" Oct 30 20:22:50.567869 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:50.567838 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-svmf9_1fa4e381-a575-4bf9-a65d-33bdc3093587/main/0.log" Oct 30 20:22:50.984194 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:50.984163 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-wtlm7_419ffdb0-c223-4426-872a-50e3073812e4/main/0.log" Oct 30 20:22:51.190311 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:51.190271 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-f4snn_5ec64166-d2fc-47f6-96d1-48ea0ee1b70c/main/0.log" Oct 30 20:22:51.786966 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:22:51.786934 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-xhhj2_ee73a969-6d25-490c-b943-2c9771cfa738/main/0.log" Oct 30 20:22:52.439222 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:22:52.439186 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:23:02.046967 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:23:02.046919 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lp2zb"] Oct 30 20:23:02.059538 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:23:02.059496 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lp2zb"] Oct 30 20:23:02.738230 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:23:02.738193 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7de7652b-0108-4745-90a4-5a31085b0409" path="/var/lib/kubelet/pods/7de7652b-0108-4745-90a4-5a31085b0409/volumes" Oct 30 20:23:22.437719 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:23:22.437680 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:23:27.509010 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:23:27.508971 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-774477d6c-wsdl5_4132b74d-b894-4c19-9333-201adb0ef1ad/manager/0.log" Oct 30 20:23:27.862587 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:23:27.862486 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-774477d6c-wsdl5_4132b74d-b894-4c19-9333-201adb0ef1ad/manager/0.log" Oct 30 20:23:52.437823 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:23:52.437782 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:23:55.039481 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:23:55.039440 5088 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8b95186-ch52x"] Oct 30 20:23:55.052750 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:23:55.052717 5088 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8b95186-ch52x"] Oct 30 20:23:56.738599 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:23:56.738551 5088 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720" path="/var/lib/kubelet/pods/6bb9b70f-ddb8-439a-9ce0-7c2c4ee80720/volumes" Oct 30 20:24:12.918723 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:12.918691 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-774477d6c-wsdl5_4132b74d-b894-4c19-9333-201adb0ef1ad/manager/0.log" Oct 30 20:24:13.247216 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:13.247178 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-774477d6c-wsdl5_4132b74d-b894-4c19-9333-201adb0ef1ad/manager/0.log" Oct 30 20:24:20.697229 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:20.697183 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hz6jn_73bb0a1f-597a-4078-8a01-20d0ed24715b/global-pull-secret-syncer/0.log" Oct 30 20:24:21.121396 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:21.121367 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-c7n9t_908c2874-510c-43c4-a07c-c1771b924e8d/connection-checker/0.log" Oct 30 20:24:21.250168 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:21.250142 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-kflff_aab2acf3-4af5-4d2e-8336-05b01a2dca1e/connection-checker/0.log" Oct 30 20:24:21.386118 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:21.386049 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-r7djq_2962e3da-8439-4316-9c3a-1679842609b7/connection-checker/0.log" Oct 30 20:24:21.902912 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:21.902882 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-djkzp_13962162-194c-4e37-ac6b-b3bbc810cedb/konnectivity-agent/0.log" Oct 30 20:24:22.151937 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:22.151906 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-173.ec2.internal_5b1dc20b8a7bd35e96a3f207cff8defc/haproxy/0.log" Oct 30 20:24:22.436760 ip-10-0-132-173 kubenswrapper[5088]: E1030 20:24:22.436726 5088 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8txdr" containerName="extractor" Oct 30 20:24:27.623091 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:27.623060 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-rq4t2_5a7ba990-22c5-4549-80a9-3a877255b7dd/manager/0.log" Oct 30 20:24:27.742889 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:27.742863 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-fvxph_335ef769-6b60-4648-bff4-ed5d0d1bbb2e/manager/0.log" Oct 30 20:24:28.128555 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:28.128519 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-pdkrz_a2a6ff3a-e8bf-4256-b860-89065d5116f7/limitador/0.log" Oct 30 20:24:28.294548 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:28.294516 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-m67x6_99574888-155b-41ad-a913-04105d65b7a0/manager/0.log" Oct 30 20:24:28.539630 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:28.539575 5088 scope.go:122] "RemoveContainer" containerID="a74f410379ca73b7360a3a2af9831ead469f2a37f4c52379ba2434f67e6d6c70" Oct 30 20:24:28.549449 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:28.549428 5088 scope.go:122] "RemoveContainer" containerID="e6d8133d0fefb47337affcaa164ed1f9e0cb4f018eb1d27f62c3e02de306fe76" Oct 30 20:24:30.493051 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:30.493020 5088 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kc489/perf-node-gather-daemonset-mbnsq"] Oct 30 20:24:30.493537 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:30.493522 5088 cpu_manager.go:414] "containerMap: removing container" podUID="dc123ad4-ca52-4982-9167-f57f94427806" containerName="maas-api" Oct 30 20:24:30.493585 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:30.493539 5088 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc123ad4-ca52-4982-9167-f57f94427806" containerName="maas-api" Oct 30 20:24:30.493696 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:30.493685 5088 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc123ad4-ca52-4982-9167-f57f94427806" containerName="maas-api" Oct 30 20:24:30.498495 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:30.498478 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kc489/perf-node-gather-daemonset-mbnsq" Oct 30 20:24:30.518219 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:30.518185 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kc489/perf-node-gather-daemonset-mbnsq"] Oct 30 20:24:30.545181 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:30.545150 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/148798ad-dba6-49bd-b387-ee80625c5a45-sys\") pod \"perf-node-gather-daemonset-mbnsq\" (UID: \"148798ad-dba6-49bd-b387-ee80625c5a45\") " pod="openshift-must-gather-kc489/perf-node-gather-daemonset-mbnsq" Oct 30 20:24:30.545367 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:30.545208 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/148798ad-dba6-49bd-b387-ee80625c5a45-lib-modules\") pod \"perf-node-gather-daemonset-mbnsq\" (UID: \"148798ad-dba6-49bd-b387-ee80625c5a45\") " pod="openshift-must-gather-kc489/perf-node-gather-daemonset-mbnsq" Oct 30 20:24:30.545367 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:30.545347 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/148798ad-dba6-49bd-b387-ee80625c5a45-podres\") pod \"perf-node-gather-daemonset-mbnsq\" (UID: \"148798ad-dba6-49bd-b387-ee80625c5a45\") " pod="openshift-must-gather-kc489/perf-node-gather-daemonset-mbnsq" Oct 30 20:24:30.545467 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:30.545441 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/148798ad-dba6-49bd-b387-ee80625c5a45-proc\") pod \"perf-node-gather-daemonset-mbnsq\" (UID: \"148798ad-dba6-49bd-b387-ee80625c5a45\") " pod="openshift-must-gather-kc489/perf-node-gather-daemonset-mbnsq" Oct 30 20:24:30.545467 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:30.545464 5088 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7gt7\" (UniqueName: \"kubernetes.io/projected/148798ad-dba6-49bd-b387-ee80625c5a45-kube-api-access-m7gt7\") pod \"perf-node-gather-daemonset-mbnsq\" (UID: \"148798ad-dba6-49bd-b387-ee80625c5a45\") " pod="openshift-must-gather-kc489/perf-node-gather-daemonset-mbnsq" Oct 30 20:24:30.604875 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:30.604836 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kc489\"/\"kube-root-ca.crt\"" Oct 30 20:24:30.604875 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:30.604863 5088 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kc489\"/\"default-dockercfg-zvbwx\"" Oct 30 20:24:30.605105 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:30.604924 5088 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kc489\"/\"openshift-service-ca.crt\"" Oct 30 20:24:30.646705 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:30.646673 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/148798ad-dba6-49bd-b387-ee80625c5a45-sys\") pod \"perf-node-gather-daemonset-mbnsq\" (UID: \"148798ad-dba6-49bd-b387-ee80625c5a45\") " pod="openshift-must-gather-kc489/perf-node-gather-daemonset-mbnsq" Oct 30 20:24:30.646912 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:30.646729 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/148798ad-dba6-49bd-b387-ee80625c5a45-lib-modules\") pod \"perf-node-gather-daemonset-mbnsq\" (UID: \"148798ad-dba6-49bd-b387-ee80625c5a45\") " pod="openshift-must-gather-kc489/perf-node-gather-daemonset-mbnsq" Oct 30 20:24:30.646912 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:30.646754 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/148798ad-dba6-49bd-b387-ee80625c5a45-podres\") pod \"perf-node-gather-daemonset-mbnsq\" (UID: \"148798ad-dba6-49bd-b387-ee80625c5a45\") " pod="openshift-must-gather-kc489/perf-node-gather-daemonset-mbnsq" Oct 30 20:24:30.646912 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:30.646792 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/148798ad-dba6-49bd-b387-ee80625c5a45-proc\") pod \"perf-node-gather-daemonset-mbnsq\" (UID: \"148798ad-dba6-49bd-b387-ee80625c5a45\") " pod="openshift-must-gather-kc489/perf-node-gather-daemonset-mbnsq" Oct 30 20:24:30.646912 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:30.646812 5088 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7gt7\" (UniqueName: \"kubernetes.io/projected/148798ad-dba6-49bd-b387-ee80625c5a45-kube-api-access-m7gt7\") pod \"perf-node-gather-daemonset-mbnsq\" (UID: \"148798ad-dba6-49bd-b387-ee80625c5a45\") " pod="openshift-must-gather-kc489/perf-node-gather-daemonset-mbnsq" Oct 30 20:24:30.646912 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:30.646811 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/148798ad-dba6-49bd-b387-ee80625c5a45-sys\") pod \"perf-node-gather-daemonset-mbnsq\" (UID: \"148798ad-dba6-49bd-b387-ee80625c5a45\") " pod="openshift-must-gather-kc489/perf-node-gather-daemonset-mbnsq" Oct 30 20:24:30.646912 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:30.646875 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/148798ad-dba6-49bd-b387-ee80625c5a45-proc\") pod \"perf-node-gather-daemonset-mbnsq\" (UID: \"148798ad-dba6-49bd-b387-ee80625c5a45\") " pod="openshift-must-gather-kc489/perf-node-gather-daemonset-mbnsq" Oct 30 20:24:30.646912 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:30.646882 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/148798ad-dba6-49bd-b387-ee80625c5a45-podres\") pod \"perf-node-gather-daemonset-mbnsq\" (UID: \"148798ad-dba6-49bd-b387-ee80625c5a45\") " pod="openshift-must-gather-kc489/perf-node-gather-daemonset-mbnsq" Oct 30 20:24:30.647198 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:30.646919 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/148798ad-dba6-49bd-b387-ee80625c5a45-lib-modules\") pod \"perf-node-gather-daemonset-mbnsq\" (UID: \"148798ad-dba6-49bd-b387-ee80625c5a45\") " pod="openshift-must-gather-kc489/perf-node-gather-daemonset-mbnsq" Oct 30 20:24:30.673519 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:30.673490 5088 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7gt7\" (UniqueName: \"kubernetes.io/projected/148798ad-dba6-49bd-b387-ee80625c5a45-kube-api-access-m7gt7\") pod \"perf-node-gather-daemonset-mbnsq\" (UID: \"148798ad-dba6-49bd-b387-ee80625c5a45\") " pod="openshift-must-gather-kc489/perf-node-gather-daemonset-mbnsq" Oct 30 20:24:30.721471 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:30.721445 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-qdv4s_20f593c8-594e-4a64-9c5e-881311b269a3/cluster-monitoring-operator/0.log" Oct 30 20:24:30.813939 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:30.813862 5088 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kc489/perf-node-gather-daemonset-mbnsq" Oct 30 20:24:31.120923 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:31.120897 5088 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kc489/perf-node-gather-daemonset-mbnsq"] Oct 30 20:24:31.123646 ip-10-0-132-173 kubenswrapper[5088]: W1030 20:24:31.123606 5088 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod148798ad_dba6_49bd_b387_ee80625c5a45.slice/crio-cc03a657f1c4550c3d8b934c5774d875fa35d529c006eeaff1445e07ae5e3636 WatchSource:0}: Error finding container cc03a657f1c4550c3d8b934c5774d875fa35d529c006eeaff1445e07ae5e3636: Status 404 returned error can't find the container with id cc03a657f1c4550c3d8b934c5774d875fa35d529c006eeaff1445e07ae5e3636 Oct 30 20:24:31.227405 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:31.227373 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kc489/perf-node-gather-daemonset-mbnsq" event={"ID":"148798ad-dba6-49bd-b387-ee80625c5a45","Type":"ContainerStarted","Data":"068213fdbb0a95c8a7dd789909d91f8d0939ea266a1685bbaf26bc743deb1f50"} Oct 30 20:24:31.227557 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:31.227430 5088 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kc489/perf-node-gather-daemonset-mbnsq" event={"ID":"148798ad-dba6-49bd-b387-ee80625c5a45","Type":"ContainerStarted","Data":"cc03a657f1c4550c3d8b934c5774d875fa35d529c006eeaff1445e07ae5e3636"} Oct 30 20:24:31.227557 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:31.227457 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kc489/perf-node-gather-daemonset-mbnsq" Oct 30 20:24:31.272759 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:31.272710 5088 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-kc489/perf-node-gather-daemonset-mbnsq" podStartSLOduration=1.2726962419999999 podStartE2EDuration="1.272696242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:24:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:24:31.271888568 +0000 UTC m=+2232.684602349" watchObservedRunningTime="2026-10-30 20:24:31.272696242 +0000 UTC m=+2232.685410017" Oct 30 20:24:32.199135 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:32.199108 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nxvww_b86bf452-ef24-4715-b1d1-c4af3e689cfc/node-exporter/0.log" Oct 30 20:24:32.318824 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:32.318796 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nxvww_b86bf452-ef24-4715-b1d1-c4af3e689cfc/kube-rbac-proxy/0.log" Oct 30 20:24:32.433602 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:32.433573 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nxvww_b86bf452-ef24-4715-b1d1-c4af3e689cfc/init-textfile/0.log" Oct 30 20:24:34.042050 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:34.042021 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-nw8kx_7ef6ce3a-0191-4e2b-a483-3c1bde6e2444/prometheus-operator-admission-webhook/0.log" Oct 30 20:24:36.640066 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:36.640036 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-tv4xh_afb6fa1c-9722-42c4-a914-4941bcbcaaad/networking-console-plugin/0.log" Oct 30 20:24:37.244288 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:37.244255 5088 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kc489/perf-node-gather-daemonset-mbnsq" Oct 30 20:24:37.681901 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:37.681813 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qdj4h_da850825-59ab-480f-9047-afd7d2bdc19f/console-operator/1.log" Oct 30 20:24:37.689260 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:37.689229 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qdj4h_da850825-59ab-480f-9047-afd7d2bdc19f/console-operator/2.log" Oct 30 20:24:38.791151 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:38.791115 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-mbstf_8d6ab503-19e8-4585-b7c7-612f33db3e07/download-server/0.log" Oct 30 20:24:40.111730 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:40.111691 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-xlgtn_9b6dd02b-d4bd-4959-9a06-8941d6baf1a5/volume-data-source-validator/0.log" Oct 30 20:24:42.964115 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:42.964084 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ml5lp_b8e3ed9d-5156-4698-82c3-df72fa728f94/dns/0.log" Oct 30 20:24:43.074478 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:43.074446 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ml5lp_b8e3ed9d-5156-4698-82c3-df72fa728f94/kube-rbac-proxy/0.log" Oct 30 20:24:43.310320 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:43.310281 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l9fpp_08b5a0ca-cfe1-4366-a404-5ebbaeb7c149/dns-node-resolver/0.log" Oct 30 20:24:43.445140 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:43.445106 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6_825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1/istio-proxy/0.log" Oct 30 20:24:44.544759 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:44.544726 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-857cdcc958-b2kxp_7a81c307-a7a7-492b-bce4-cca4b59e1de4/registry/0.log" Oct 30 20:24:44.925996 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:44.925913 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hkvn6_2e03cb52-fd85-4e35-87b4-38833cbc7d74/node-ca/0.log" Oct 30 20:24:45.663795 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:45.663764 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-l9g8p_ed843517-0e6d-44b9-a6f9-4c6e686bad29/discovery/0.log" Oct 30 20:24:45.876652 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:45.876622 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f5c96996f-wsql6_e4e0fd61-e6e4-4d87-a607-ad397cc39697/kube-auth-proxy/0.log" Oct 30 20:24:45.947092 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:45.946995 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-bfxfk_4b460a5d-51f8-4694-8dd6-316f788bf8fc/istio-proxy/0.log" Oct 30 20:24:45.948896 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:45.948870 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-bfxfk_4b460a5d-51f8-4694-8dd6-316f788bf8fc/istio-proxy/1.log" Oct 30 20:24:45.994967 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:45.994927 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-c2tb9_ce0a1543-79bc-440d-ad18-c045ac745786/istio-proxy/0.log" Oct 30 20:24:46.270428 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:46.270393 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-lnxpq_f7ff5ff7-5866-473f-9055-e8c74f9380bc/istio-proxy/0.log" Oct 30 20:24:46.337922 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:46.337882 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-ms5th_4045e4a8-b80c-4b45-96b7-e9e4203ebe5e/istio-proxy/0.log" Oct 30 20:24:46.755977 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:46.755948 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-79458f7dbc-vbb28_41534930-917f-44ec-bccd-86d6c6fc7c70/router/0.log" Oct 30 20:24:46.929332 ip-10-0-132-173 kubenswrapper[5088]: I1030 20:24:46.929301 5088 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7cfc97f8f77q7l6_825ed6e8-c40b-423a-97f2-b9b2dcc6cdd1/istio-proxy/0.log"